gh-156664: Fix interaction between free variables and comprehensions

[JelleZijlstra]

• Issue: inlined comprehension and closed over variables have buggy interactions #156664

[carljm]

Thanks! Codex found a couple regressions that this causes relative to main.

[carljm]

This conversion can leave a null cell slot when the current block is itself an inlined comprehension. The copied symbol still has DEF_LOCAL, so codegen_push_inlined_comprehension_locals() clears the enclosing cell with LOAD_FAST_AND_CLEAR, but now skips MAKE_CELL because the scope is FREE. After the nested comprehension restores the null slot, a subsequent closure capture or dereference crashes.

def f(x):
    return [([lambda: x for x in [1]], lambda: x) for y in [0]]

f(7)

This completes successfully on main, but aborts on the LOAD_FAST_BORROW non-null assertion with this PR. Replacing the second lambda with [x for _ in [0]] produces a segmentation fault in LOAD_DEREF.

[carljm]

With separate cell and free-variable slots, both slots contain values during the comprehension, and frame.f_locals exposes the shared name twice. Its keys(), items(), values(), and length iterate the slots without deduplicating names. This breaks keyword expansion, for example:

import sys

def outer():
    x = [1]
    def inner():
        return [(lambda: x, dict(**sys._getframe().f_locals))
                for x in x]
    return inner()

outer()

Main succeeds, with {'x': 1} in the result, while this PR raises TypeError: dict() got multiple values for keyword argument 'x'. Likewise, dict(sys._getframe().f_locals.items())["x"] returns the enclosing [1] instead of the comprehension's 1, even though direct lookup returns 1.

[JelleZijlstra]

That's an interesting piece of code, I guess we'll have to filter out cells when creating the locals proxy.

My Codex came up with this variant:

import sys

def write_x(value):
    proxy = sys._getframe(1).f_locals
    proxy["x"] = value
    return proxy["x"]

def outer():
    x = 3

    def f():
        proxy_saw = write_x(4)
        funcs = [lambda: x for x in [1]]
        return proxy_saw, x, funcs[0]()

    return f()

print(outer())

Which works differently on this PR and main. It seemed a bit out there so I didn't post it, but your repro is more likely to break real code.