Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Healthcheck indicates healthy before processing docker-entrypoint-initdb.d #610

Closed
delanym opened this issue Aug 30, 2024 · 5 comments · Fixed by #612
Closed

Healthcheck indicates healthy before processing docker-entrypoint-initdb.d #610

delanym opened this issue Aug 30, 2024 · 5 comments · Fixed by #612

Comments

@delanym
Copy link

delanym commented Aug 30, 2024

https://mariadb.com/kb/en/using-healthcheck-sh/

When configured like test: ["CMD", "healthcheck.sh", "--connect", "--innodb_initialized"] the healthcheck indicates to dependent containers that it is healthy even though initialization from /docker-entrypoint-initdb.d/ has not completed.

You can see below Redmine performing a migration before the database is fully initialized

mariadb-1  | 2024-08-30 20:43:27 0 [Note] mariadbd: ready for connections.
mariadb-1  | Version: '11.5.2-MariaDB-ubu2404'  socket: '/run/mysqld/mysqld.sock'  port: 0  mariadb.org binary distribution
mariadb-1  | 2024-08-30 20:43:28+00:00 [Note] [Entrypoint]: Temporary server started.
mariadb-1  | 2024-08-30 20:43:28+00:00 [Note] [Entrypoint]: Creating database redmine
mariadb-1  | 2024-08-30 20:43:28+00:00 [Note] [Entrypoint]: Creating user redmine
mariadb-1  | 2024-08-30 20:43:28+00:00 [Note] [Entrypoint]: Giving user redmine access to schema redmine
mariadb-1  | 2024-08-30 20:43:28+00:00 [Note] [Entrypoint]: Securing system users (equivalent to running mysql_secure_installation)
mariadb-1  | 2024-08-30 20:43:28 6 [Warning] Access denied for user 'healthcheck'@'localhost' (using password: YES)
mariadb-1  | 
mariadb-1  | 2024-08-30 20:43:28+00:00 [Note] [Entrypoint]: /usr/local/bin/docker-entrypoint.sh: running /docker-entrypoint-initdb.d/activate-qcache.sql
redmine-1  | Your Gemfile lists the gem redmineup (>= 0) more than once.
redmine-1  | You should probably keep only one of them.
redmine-1  | Remove any duplicate entries and specify the gem only once.
redmine-1  | While it's not a problem now, it could cause errors if you change the version of one of them later.
redmine-1  | Your Gemfile lists the gem puma (>= 0) more than once.
redmine-1  | You should probably keep only one of them.
redmine-1  | Remove any duplicate entries and specify the gem only once.
redmine-1  | While it's not a problem now, it could cause errors if you change the version of one of them later.
redmine-1  | Resolving dependencies...
redmine-1  | The Gemfile's dependencies are satisfied
redmine-1  | W, [2024-08-30T20:43:59.959774 #26]  WARN -- : Creating scope :system. Overwriting existing method Enumeration.system.
redmine-1  | W, [2024-08-30T20:44:00.065795 #26]  WARN -- : Creating scope :sorted. Overwriting existing method User.sorted.
redmine-1  | I, [2024-08-30T20:44:00.658530 #26]  INFO -- : Migrating to AddJournalUpdatedOn (20220714093000)
redmine-1  | == 20220714093000 AddJournalUpdatedOn: migrating ==============================
redmine-1  | -- add_column(:journals, :updated_on, :datetime, {:after=>:created_on})
redmine-1  |    -> 0.0056s
mariadb-1  | 
mariadb-1  | 
redmine-1  | == 20220714093000 AddJournalUpdatedOn: migrated (1.3267s) =====================
redmine-1  | 
redmine-1  | I, [2024-08-30T20:44:01.992322 #26]  INFO -- : Migrating to AddJournalUpdatedBy (20220714093010)
redmine-1  | == 20220714093010 AddJournalUpdatedBy: migrating ==============================
redmine-1  | -- add_column(:journals, :updated_by_id, :integer, {:default=>nil, :after=>:updated_on})
mariadb-1  | 2024-08-30 20:43:28+00:00 [Note] [Entrypoint]: /usr/local/bin/docker-entrypoint.sh: running /docker-entrypoint-initdb.d/backup-2024-08-23.sql
mariadb-1  | 
mariadb-1  | 
mariadb-1  | 2024-08-30 20:43:39+00:00 [Note] [Entrypoint]: /usr/local/bin/docker-entrypoint.sh: running /docker-entrypoint-initdb.d/reports-user.sql
mariadb-1  | 
mariadb-1  | 
mariadb-1  | 2024-08-30 20:43:39+00:00 [Note] [Entrypoint]: Stopping temporary server
mariadb-1  | 2024-08-30 20:43:39 0 [Note] mariadbd (initiated by: unknown): Normal shutdown
mariadb-1  | 2024-08-30 20:43:39 0 [Note] InnoDB: FTS optimize thread exiting.
mariadb-1  | 2024-08-30 20:43:39 0 [Note] InnoDB: Starting shutdown...
mariadb-1  | 2024-08-30 20:43:39 0 [Note] InnoDB: Dumping buffer pool(s) to /var/lib/mysql/ib_buffer_pool
mariadb-1  | 2024-08-30 20:43:39 0 [Note] InnoDB: Restricted to 2016 pages due to innodb_buf_pool_dump_pct=25
mariadb-1  | 2024-08-30 20:43:39 0 [Note] InnoDB: Buffer pool(s) dump completed at 240830 20:43:39
mariadb-1  | 2024-08-30 20:43:39 0 [Note] InnoDB: Removed temporary tablespace data file: "./ibtmp1"
@grooverdan
Copy link
Member

Do you have?

redmine:
    depends_on:
      mariadb:
        condition: service_healthy
```

@delanym
Copy link
Author

delanym commented Aug 30, 2024

Yes

  redmine:
    depends_on:
      mariadb:
        condition: service_healthy
        restart: true
    restart: unless-stopped

@grooverdan
Copy link
Member

on 2611c39

  1. in connect - select @@skip_networking fails to connect - hence "Access denied for user" - falls through to second test
  2. connect() doesn't output "Can't connect", on --protocol tcp - sets connect_s=0 (don't know this happens)
  3. if ! "$test"; then succeeded - next test
  4. innodb_initialized succeeds.
    5.if [ -z "$connect_s" ]; then skipped due to non-0
  5. returns true (0)

grooverdan added a commit to grooverdan/mariadb-docker that referenced this issue Sep 2, 2024
Based on user reports, a `connect` test can observer a non-"Can't
connect" error message.

Because this passes other tests, like innodb_initialized might
succeeded.

The final test -z "$connect_s" is also true, leaving the user
with an incorrect test result.

Maybe the healthcheck cnf hasn't been created yet?

Either way, ruggardise the final test to ensure a healthy connect
test occured.

Closes MariaDB#610
@grooverdan
Copy link
Member

grooverdan commented Sep 2, 2024

I think I have a fix in #612.

If you have a chance to test it that would be appreciated.

FYI @mariushosting

grooverdan added a commit to grooverdan/mariadb-docker that referenced this issue Sep 2, 2024
Based on user reports, a `connect` test can observer a non-"Can't
connect" error message.

Because this passes other tests, like innodb_initialized might
succeeded.

The final test -z "$connect_s" is also true, leaving the user
with an incorrect test result.

Maybe the healthcheck cnf hasn't been created yet?

Either way, ruggardise the final test to ensure a healthy connect
test occured.

Closes MariaDB#610
grooverdan added a commit to grooverdan/mariadb-docker that referenced this issue Sep 2, 2024
Based on user reports, a `connect` test can observer a non-"Can't
connect" error message.

Because this passes other tests, like innodb_initialized might
succeeded.

The final test -z "$connect_s" is also true, leaving the user
with an incorrect test result.

Maybe the healthcheck cnf hasn't been created yet?

Either way, ruggardise the final test to ensure a healthy connect
test occured.

Narrow down the scope of healthcheck --connect failures
to the mariadb client errors "Can't connect".

May as well look at same @skip-networking system variable
on TCP just in case.

28000 errors, per
https://mariadb.com/kb/en/mariadb-error-code-reference/ are client
errors after a connection, indicating that the start of a tcp
connection occured.

Closes MariaDB#610
@grooverdan
Copy link
Member

For testing ghcr.io/mariadb/mariadb:11.5.2-noble

docker-library/official-images#17468 has the official change.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

Successfully merging a pull request may close this issue.

2 participants