AttributeError: 'Module' object has no attribute 'doc_node'

[JCZuurmond]

Astroid tries to get the doc_node on Module object while it does not exists. A safer implementation gets the attribute only when it exists, otherwise gets None.

InternalError: linting workflows(1234) task failed: 'Module' object has no attribute 'doc_node'
  File "/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.11/site-packages/databricks/labs/ucx/source_code/python/python_infer.py", line 78, in _unsafe_infer_internal
    all_inferred = node.inferred()
                   ^^^^^^^^^^^^^^^
  File "/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.11/site-packages/astroid/nodes/node_ng.py", line 584, in inferred
    return list(self.infer())
           ^^^^^^^^^^^^^^^^^^
  File "/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.11/site-packages/astroid/nodes/node_ng.py", line 168, in infer
    for i, result in enumerate(self._infer(context=context, **kwargs)):
  File "/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.11/site-packages/astroid/decorators.py", line 86, in inner
    yield next(generator)
          ^^^^^^^^^^^^^^^
  File "/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.11/site-packages/astroid/decorators.py", line 49, in wrapped
    for res in _func(node, context, **kwargs):
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.11/site-packages/astroid/nodes/node_classes.py", line 585, in _infer
    frame, stmts = self.lookup(self.name)
                   ^^^^^^^^^^^^^^^^^^^^^^
  File "/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.11/site-packages/astroid/nodes/_base_nodes.py", line 277, in lookup
    return self.scope().scope_lookup(self, name)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.11/site-packages/astroid/nodes/scoped_nodes/scoped_nodes.py", line 1706, in scope_lookup
    return frame._scope_lookup(node, name, offset)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.11/site-packages/astroid/nodes/scoped_nodes/mixin.py", line 91, in _scope_lookup
    return pscope.scope_lookup(node, name)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.11/site-packages/astroid/nodes/scoped_nodes/scoped_nodes.py", line 325, in scope_lookup
    return self, self.getattr(name)
                 ^^^^^^^^^^^^^^^^^^
  File "/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.11/site-packages/astroid/nodes/scoped_nodes/scoped_nodes.py", line 355, in getattr
    result = [self.special_attributes.lookup(name)]
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.11/site-packages/astroid/interpreter/objectmodel.py", line 132, in lookup
    return getattr(self, IMPL_PREFIX + name)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.11/site-packages/astroid/interpreter/objectmodel.py", line 203, in attr___doc__
    value=getattr(self._instance.doc_node, "value", None),
                  ^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'Module' object has no attribute 'doc_node'

Originally raise on databrickslabs/ucx#3659

[DanielNoord]

Do you know how the Module was constructed?

[JCZuurmond]

Hi @DanielNoord , I do not have the source code, maybe @pamons-databricks can provide that.

We use astroid.parse to create a Astroid node from source code str

[DanielNoord]

Is it possible that something is not calling Module.postinit?

>>> astroid.scoped_nodes.Module("a").doc_node
Traceback (most recent call last):
  File "<python-input-6>", line 1, in <module>
    astroid.scoped_nodes.Module("a").doc_node
AttributeError: 'Module' object has no attribute 'doc_node'

astroid/astroid/nodes/scoped_nodes/scoped_nodes.py

Lines 277 to 281 in 7318c43

	def postinit(
	self, body: list[node_classes.NodeNG], *, doc_node: Const | None = None
	):
	self.body = body
	self.doc_node = doc_node

I checked the flow of parse and that seems to call postinit just fine, so I don't really see how this could go wrong...

[JCZuurmond]

TBH, I am not 100% if we use the postinit correctly.

I am confident that all Python source code is converted to a node using parse.

But, I also see we use the postinit on what looks to be a custom node class. The postinit is not added to another custom node class. Could that mess things up?

Fyi: Both the custom node classes and the postinit are new to me

[DanielNoord]

Const doesn't have a postinit so that should be fine.

Are you making Modules anywhere?

[pamons-databricks]

The failing workflow has a single Python Script task which calls a python file in a workspace location
In the parameters box it passes a list of paths in a volume, ex ["/Volumes/path/to/csv_file.csv"]

The workflow has never been run either

[JCZuurmond]

Const doesn't have a postinit so that should be fine.

Are you making Modules anywhere?

We do that here to create "root" modules to attach trees to

[DanielNoord]

@JCZuurmond You should definitely call postinit there. I think there is a good chance that will fix your issue.