Quick question: How could we implement dill.dumps for async generators?

[creatorrr]

Hey @pulkin !

Have been having a lot of fun playing around with pyteleport, very educational for me so thanks! :)

I am still learning about python internals but I wanted to know how could we implement dill.dumps for pickling async generator objects? Similar to what you've done for generator objects for instance.

[pulkin]

async is still a matter of research. As far as I can see, you have to implement serializing (and restoring) some of the built-in frames. This will not automatically fix the issue as there is currently no pipeline or API to handle built-in frames.

I added an async branch to play around. The traceback currently looks like this:

pyteleport.snapshot.FrameStackException: Recorded frame stack does not match TOS+1 analysis
Snapshot traceback (most recent call last):
File "_test_teleport_async.py", line 26, in __main__
File "/usr/lib64/python3.10/asyncio/runners.py", line 44, in asyncio.runners
File "/usr/lib64/python3.10/asyncio/base_events.py", line 636, in asyncio.base_events
File "/usr/lib64/python3.10/asyncio/base_events.py", line 603, in asyncio.base_events
File "/usr/lib64/python3.10/asyncio/base_events.py", line 1909, in asyncio.base_events
File "/usr/lib64/python3.10/asyncio/events.py", line 80, in asyncio.events
  TOS+1 in the frame above is a built-in function or method
    <built-in method run of _contextvars.Context object at 0x7f207d50d000>
File "_test_teleport_async.py", line 21, in __main__

Basically, the async frame is run inside _contextvars.Context.run function which is implemented in C (and may even have its own stack). It is called in asyncio/events.py as this:

self._context.run(self._callback, *self._args)

This is probably the implementation of Context.run in cpython sources:

https://github.com/python/cpython/blob/aae84a0e849adc7486115b72e082e6896eebc3d5/Python/context.c#L649-L673

Naively, it reads as this

def run(self, callback, *args):
    with self:
        callback(*args)

but better learn more about context and context variables.