Multiple worker support

[habdelra]

This PR enables multiple worker support. It introduces a new mechanism, worker-manager, that is responsible for spinning up the desired number of workers in each container. the worker-manager is also responsible for keeping track of when all the workers are ready using a very simple TCP server that the realm server leverages in the test environment. Additionally, the worker-manager is responsible for tagging the stdout and stderr with the worker's PID so that the log messages among multiple workers can be correlated correctly.

The start:all script spins up the workers automatically. Each realm server has a corresponding worker-manager. However, you can also use start:worker-development to spin up the workers manually if you'd like (which is paired with the start:development script for starting the development realm server). The WORKER_COUNT environment variable is used to indicate how many workers the worker-manager should spin up. By default, when this environment variable is not specified, only a single worker is spun up by the worker-manager.

Finally, this PR also updates the deploy scripts to deploy the workers into their own ECS container. Currently we are only deploying a single worker per ECS container. as soon as we confirm everything is working properly, we'll increase the worker count in our hosted envs in a subsequent PR.

This PR is paired with the infra PR https://github.com/cardstack/infra/pull/568

TODO:

• Test deploy in staging

[github-actions]

Host Test Results

    1 files  ±0      1 suites  ±0   21m 47s ⏱️ -20s
728 tests ±0  726 ✔️ ±0  2 💤 ±0  0 ❌ ±0 
733 runs  ±0  731 ✔️ ±0  2 💤 ±0  0 ❌ ±0 

Results for commit 348026b. ± Comparison against base commit 0b30125.

♻️ This comment has been updated with latest results.

[lukemelia]

Looks good. I notice that we are identifying the worker by pid in logging. Might it also be useful to include the hostname?

[habdelra]

Looks good. I notice that we are identifying the worker by pid in logging. Might it also be useful to include the hostname?

I think that it will be moot, since it will be cloudwatch that actually renders the logs. Cloudwatch already takes care of that for us.

[jurgenwerk]

Looks good! Perhaps we can consider adding more managerial logic to the manager down the road, for example respawn the worker in case it dies.

[habdelra]

Looks good! Perhaps we can consider adding more managerial logic to the manager down the road, for example respawn the worker in case it dies.

That’s a good idea

[habdelra]

Looks good! Perhaps we can consider adding more managerial logic to the manager down the road, for example respawn the worker in case it dies.

I like this idea so much i'm adding it to this PR (it's a really simple update)