Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Depend on other services based on health checks for docker compose #2715

Conversation

andresilva91
Copy link
Contributor

@andresilva91 andresilva91 commented Oct 25, 2024

Motivation

When using Docker Compose, we're not currently properly waiting for the service dependencies.

Proposal

Add a health check, and remove the init_done file

Test Plan

CI: we can see from the Docker CI logs that both the proxy and shards services only get created after shard-init exits:

Network docker_default  Creating
 Network docker_default  Created
 Volume "docker_grafana-storage"  Creating
 Volume "docker_grafana-storage"  Created
 Volume "docker_linera-scylla-data"  Creating
 Volume "docker_linera-scylla-data"  Created
 Container scylla  Creating
 Container prometheus  Creating
 Container grafana  Creating
 Container prometheus  Created
 Container scylla  Created
 Container grafana  Created
 Container shard-init  Creating
 Container shard-init  Created
 Container proxy  Creating
 Container docker-shard-4  Creating
 Container docker-shard-2  Creating
 Container docker-shard-1  Creating
 Container docker-shard-3  Creating
 Container docker-shard-1  Created               <----  Creations
 Container proxy  Created                               <----
 Container docker-shard-4  Created              <----
 Container docker-shard-3  Created              <----
 Container docker-shard-2  Created              <----
 Container grafana  Starting
 Container prometheus  Starting
 Container scylla  Starting
 Container scylla  Started
 Container grafana  Started
 Container shard-init  Starting
 Container prometheus  Started
 Container shard-init  Started
 Container shard-init  Waiting
 Container shard-init  Waiting
 Container shard-init  Exited                           <----    Exits
 Container proxy  Starting                               <----   Starts
 Container shard-init  Exited
 Container docker-shard-4  Starting              <----
 Container proxy  Started
 Container docker-shard-4  Started
 Container docker-shard-1  Starting               <----
 Container docker-shard-1  Started
 Container docker-shard-3  Starting               <----
 Container docker-shard-3  Started
 Container docker-shard-2  Starting                <----
 Container docker-shard-2  Started
 Container prometheus  Waiting
 Container grafana  Waiting
 Container scylla  Waiting
 Container proxy  Waiting
 Container docker-shard-4  Waiting
 Container docker-shard-1  Waiting
 Container docker-shard-3  Waiting
 Container docker-shard-2  Waiting
 Container shard-init  Waiting
 Container docker-shard-4  Healthy
 Container docker-shard-1  Healthy
 Container docker-shard-3  Healthy
 Container docker-shard-2  Healthy
 Container proxy  Healthy
 Container shard-init  Exited
 Container scylla  Healthy
 Container grafana  Healthy
 Container prometheus  Healthy

Release Plan

  • Nothing to do / These changes follow the usual release cycle.

Copy link
Contributor Author

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @andresilva91 and the rest of your teammates on Graphite Graphite

@andresilva91 andresilva91 force-pushed the 10-25-depend_on_other_services_based_on_health_checks_for_docker_compose branch from 4251b89 to ddc8ee9 Compare October 25, 2024 21:07
Copy link
Contributor

@ma2bd ma2bd left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice!

Copy link
Contributor Author

andresilva91 commented Oct 25, 2024

Merge activity

  • Oct 25, 5:38 PM EDT: A user started a stack merge that includes this pull request via Graphite.
  • Oct 25, 5:39 PM EDT: A user merged this pull request with Graphite.

@andresilva91 andresilva91 merged commit ac00cc3 into main Oct 25, 2024
2 checks passed
@andresilva91 andresilva91 deleted the 10-25-depend_on_other_services_based_on_health_checks_for_docker_compose branch October 25, 2024 21:39
ma2bd pushed a commit that referenced this pull request Oct 25, 2024
…2715)

## Motivation

When using Docker Compose, we're not currently properly waiting for the service dependencies.

## Proposal

Add a health check, and remove the `init_done` file

## Test Plan

CI: we can see from the Docker CI logs that both the `proxy` and `shard`s services only get created after `shard-init` exits:
```
Network docker_default  Creating
 Network docker_default  Created
 Volume "docker_grafana-storage"  Creating
 Volume "docker_grafana-storage"  Created
 Volume "docker_linera-scylla-data"  Creating
 Volume "docker_linera-scylla-data"  Created
 Container scylla  Creating
 Container prometheus  Creating
 Container grafana  Creating
 Container prometheus  Created
 Container scylla  Created
 Container grafana  Created
 Container shard-init  Creating
 Container shard-init  Created
 Container proxy  Creating
 Container docker-shard-4  Creating
 Container docker-shard-2  Creating
 Container docker-shard-1  Creating
 Container docker-shard-3  Creating
 Container docker-shard-1  Created               <----  Creations
 Container proxy  Created                               <----
 Container docker-shard-4  Created              <----
 Container docker-shard-3  Created              <----
 Container docker-shard-2  Created              <----
 Container grafana  Starting
 Container prometheus  Starting
 Container scylla  Starting
 Container scylla  Started
 Container grafana  Started
 Container shard-init  Starting
 Container prometheus  Started
 Container shard-init  Started
 Container shard-init  Waiting
 Container shard-init  Waiting
 Container shard-init  Exited                           <----    Exits
 Container proxy  Starting                               <----   Starts
 Container shard-init  Exited
 Container docker-shard-4  Starting              <----
 Container proxy  Started
 Container docker-shard-4  Started
 Container docker-shard-1  Starting               <----
 Container docker-shard-1  Started
 Container docker-shard-3  Starting               <----
 Container docker-shard-3  Started
 Container docker-shard-2  Starting                <----
 Container docker-shard-2  Started
 Container prometheus  Waiting
 Container grafana  Waiting
 Container scylla  Waiting
 Container proxy  Waiting
 Container docker-shard-4  Waiting
 Container docker-shard-1  Waiting
 Container docker-shard-3  Waiting
 Container docker-shard-2  Waiting
 Container shard-init  Waiting
 Container docker-shard-4  Healthy
 Container docker-shard-1  Healthy
 Container docker-shard-3  Healthy
 Container docker-shard-2  Healthy
 Container proxy  Healthy
 Container shard-init  Exited
 Container scylla  Healthy
 Container grafana  Healthy
 Container prometheus  Healthy
```

## Release Plan

- Nothing to do / These changes follow the usual release cycle.
ma2bd pushed a commit that referenced this pull request Oct 26, 2024
…2715)

## Motivation

When using Docker Compose, we're not currently properly waiting for the service dependencies.

## Proposal

Add a health check, and remove the `init_done` file

## Test Plan

CI: we can see from the Docker CI logs that both the `proxy` and `shard`s services only get created after `shard-init` exits:
```
Network docker_default  Creating
 Network docker_default  Created
 Volume "docker_grafana-storage"  Creating
 Volume "docker_grafana-storage"  Created
 Volume "docker_linera-scylla-data"  Creating
 Volume "docker_linera-scylla-data"  Created
 Container scylla  Creating
 Container prometheus  Creating
 Container grafana  Creating
 Container prometheus  Created
 Container scylla  Created
 Container grafana  Created
 Container shard-init  Creating
 Container shard-init  Created
 Container proxy  Creating
 Container docker-shard-4  Creating
 Container docker-shard-2  Creating
 Container docker-shard-1  Creating
 Container docker-shard-3  Creating
 Container docker-shard-1  Created               <----  Creations
 Container proxy  Created                               <----
 Container docker-shard-4  Created              <----
 Container docker-shard-3  Created              <----
 Container docker-shard-2  Created              <----
 Container grafana  Starting
 Container prometheus  Starting
 Container scylla  Starting
 Container scylla  Started
 Container grafana  Started
 Container shard-init  Starting
 Container prometheus  Started
 Container shard-init  Started
 Container shard-init  Waiting
 Container shard-init  Waiting
 Container shard-init  Exited                           <----    Exits
 Container proxy  Starting                               <----   Starts
 Container shard-init  Exited
 Container docker-shard-4  Starting              <----
 Container proxy  Started
 Container docker-shard-4  Started
 Container docker-shard-1  Starting               <----
 Container docker-shard-1  Started
 Container docker-shard-3  Starting               <----
 Container docker-shard-3  Started
 Container docker-shard-2  Starting                <----
 Container docker-shard-2  Started
 Container prometheus  Waiting
 Container grafana  Waiting
 Container scylla  Waiting
 Container proxy  Waiting
 Container docker-shard-4  Waiting
 Container docker-shard-1  Waiting
 Container docker-shard-3  Waiting
 Container docker-shard-2  Waiting
 Container shard-init  Waiting
 Container docker-shard-4  Healthy
 Container docker-shard-1  Healthy
 Container docker-shard-3  Healthy
 Container docker-shard-2  Healthy
 Container proxy  Healthy
 Container shard-init  Exited
 Container scylla  Healthy
 Container grafana  Healthy
 Container prometheus  Healthy
```

## Release Plan

- Nothing to do / These changes follow the usual release cycle.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants