nonlocal variables are included in locals while global variables not

[yueyinqiu]

Hello! I'm trying locals to get the variables (or names) in a function. However I've found that global variables are excluded, while nonlocal variables are here:

import astroid.nodes

module = astroid.parse(
"""
x1 = 1                      # Line 2
def f1():                   # Line 3
    x2 = 2                  # Line 4
    def f2():               # Line 5
        global x1           # Line 6
        nonlocal x2         # Line 7
        x1 = 1              # Line 8
        x2 = 2              # Line 9
""")

f2 = module.locals["f1"][0].locals["f2"][0]
print(f2.locals)

# Output:
# {'x2': [<AssignName.x2 l.9 at 0x1ddffab04d0>]}

That's really weird to me. Is it or bug or intentionally designed? If this behavior is expected, what is the recommended solution to exclude those nonlocal variables? (and include global ones?)

I have found #250 , which however was moved from bitbucket in 2015, so I'm not sure whether it is outdated.

Thanks in advance.

[DanielNoord]

Seems like a bug to me! We should fix this and are open to pull requests :)

[yueyinqiu]

I would try on this. It seems not that hard, since there is global acting as a reference.

The only difference seems to be it requires to go through the tree and find what this nonlocal variable actually refers.

Hope I haven't missed anything.

[jacobtylerwalls]

Including x2 in locals makes sense to me, as it's what happens at runtime (the whole point of the nonlocal keyword being to add a not-otherwise-local to the locals):

>>> def a():
...   x = 1
...   def b():
...     nonlocal x
...     print(locals())
...   b()
... 
>>> a()
{'x': 1}
>>> def a():
...   x = 1
...   def b():
...     print(locals())
...   b()
... 
>>> a()
{}

[DanielNoord]

Ah you are totally right. I should have checked this in the code:

❯ cat /tmp/test.py
def a():
    x = 1
    y = 2
    def b():
        global x
        nonlocal y
        print(locals())
    b()
a()
❯ python /tmp/test.py
{'y': 2}

Seems like this is actually working as intended? As in, we're mimicking what locals() does? Thanks for pointing this out Jacob, glad I pinged you!

[jacobtylerwalls]

As in, we're mimicking what locals() does?

Hard to say! Docstring just says:

"""A map of the name of a local variable to the node defining it."""

From CPython docs for locals():

each call to locals() instead returns a fresh dictionary containing the current bindings of the function’s local variables and any nonlocal cell references.

So does astroid's locals represent the current bindings of the function’s local variables or the result of locals()? Unless we have a reason to change it, I'd just suggest leaving as is and possibly clarifying in the docstring that we're mimicking locals().

I also wouldn't want to cause unnecessary churn around emitting/not emitting pylint messages for count of max locals, etc.

[yueyinqiu]

I think why CPython includes those nonlocal variables is because by doing that users can use locals() + globals() to access all the names.

However we doesn't provide globals in a FunctionDef. So actually the problem is not only we could access nonlocals in locals, but also we can't access globals here. Of course we could do that when visiting Module instead, so it's really different for runtime and for ast.

Meanwhile CPython does not provide usages of the variables, but we do. With the current implemention, we can't find the usgaes inside those nested function:

import astroid.nodes

module = astroid.parse(
"""
x1 = 1                      # Line 1
def f1():                   # Line 2
    x2 = 2                  # Line 3
    def f2():               # Line 4
        global x1           # Line 5
        nonlocal x2         # Line 6
        x1 = 1              # Line 7
        x2 = 2              # Line 8
""")

f1 = module.locals["f1"][0]
print(module.locals["x1"])  # [<AssignName.x1 l.2 at 0x2511d5c0170>, <AssignName.x1 l.8 at 0x2511d5c0500>]
print(f1.locals["x2"])  # [<AssignName.x2 l.4 at 0x2511d5c03b0>]

And also, it seems that we can't get x1 when visiting f2, unless to go through its body and find the Global node, or to iterate all the global variables and check their lineno.

So will it be a good idea to include both global x1 and nonlocal x2 into f2.locals, and add an extra property to indicate whether they are global or nonlocal and where they come from?

[yueyinqiu]

In fact this problem happens in an actual situation when I'm trying to write this PyLint plugin.

I want to prevent users from adding type annotation for the same variable duplicately. So I visit_functiondef, visit_module, visit_classdef and check whether their locals' parent is AnnAssign.

The globals will not appear in visit_functiondef, and instead, all of them will appear in visit_module. So at least in my case, there is no problem with globals.

But I have no idea how to deal with nonlocals like that. It seems I could only go through the function's body to find the Nonlocal statements.

[jacobtylerwalls]

But I have no idea how to deal with nonlocals like that. It seems I could only go through the function's body to find the Nonlocal statements.

That sounds like a way to meet your use case. You might try calling nodes_of_class(nodes.Nonlocal).

[jacobtylerwalls]

So will it be a good idea to include both global x1 and nonlocal x2 into f2.locals, and add an extra property to indicate whether they are global or nonlocal and where they come from?

I guess I don't think so, I'd rather keep globals and locals separate to avoid astonishing users who expect these variables to track globals() and locals().

[yueyinqiu]

That sounds like a way to meet your use case. You might try calling nodes_of_class(nodes.Nonlocal).

Ah there's such a function!

But I have to use nodes_of_class(Nonlocal, FunctionDef) to skip those Nonlocal nodes inside the nested functions.

Anyway that's a great idea and I will use it currently.

[yueyinqiu]

I guess I don't think so, I'd rather keep globals and locals separate to avoid astonishing users who expect these variables to track globals() and locals().

Umm... I think the most convincing reason to do the change is:

def f1():
    x = 1
    def f2():
        nonlocal x
        x = 2

With the current implemention, f1.locals["x"] does not contains x = 2, which is in f2.locals instead. This could make it hard to track the variable in something like visit_functiondef(f1).

[jacobtylerwalls]

I guess I wouldn't expect x = 2 in f1's locals.