Multiprocess slow start for ops

[joshuataylor]

Hi All

With dagster in production we're seeing ops take 5+ seconds to start, even if nothing else is running at the same time.

We're using:
Python 3.8.12 - just upgraded to 3.10.4 and it has the same issue
Dagster: 0.14.7
Backing store: Postgres (AWS RDS)
Host: Ubuntu on AWS EC2 Graviton2 (but also has issues on AMD64) Linux ip-xxx 5.13.0-1017-aws #19~20.04.1-Ubuntu SMP Mon Mar 7 12:55:31 UTC 2022 aarch64 aarch64 aarch64 GNU/Linux) Ubuntu 20.04.4 LTS

[joshuataylor]

We have found that solids don't seem to have this, thinking it could be a subprocess thing?

[prha]

Yeah, this could be the overhead of spawning a new process for each op. It may be the number of Python imports that have to happen in the process. When comparing against solids, do you see the same latency when using the in_process_executor? You can invoke this by assigning the executor on the job itself:

from dagster import job, op, in_process_executor

@job(executor_def=in_process_executor)
def my_job():
    ...

[joshuataylor]

Yeah, with in_process_executor I see it being pretty much instant, same as the previous way.

We do have a semi-complicated setup where we build a list of jobs dynamically from YAML, so if I had to guess this would be the reason for the delay -- does dagster bootstrap the repository list each time?

[alangenfeld]

does dagster bootstrap the repository list each time?

Yea, when we need to load an artifact in a new subprocess we do it based on how it was loaded originally.

[alangenfeld]

One tool to avoid repeating the cost is to use the start_method forkserver which is available as config on the multiprocess executor. The first op will still pay the init cost to create the template process, but each subsequent op should be faster as it forks that template process instead of starting from scratch. You may need to explicitly set preload_modules in the config if the default behavior doesn't load the necessary modules.

context: https://docs.python.org/3/library/multiprocessing.html#contexts-and-start-methods

[isaac-jordan]

Tried this myself and I think I maybe see a small increase using forkserver and a list of preloaded modules, but it's still taking approx 8 seconds to start any op (Launching subprocess to Starting initialization of resources...).