Skip to content

Commit

Permalink
icingadb.DB: Retry Schema Checks
Browse files Browse the repository at this point in the history
During testing, I just encountered a race condition where my Galera
cluster was not yet ready, causing the initial schema check to fail.

```
2024-04-11T08:13:40.401Z        INFO    icingadb        Starting Icinga DB daemon (1.1.1)
2024-04-11T08:13:40.401Z        INFO    icingadb        Connecting to database at 'mysql:3306'
2024-04-11T08:13:40.404Z        FATAL   icingadb        Error 1047 (08S01): WSREP has not yet prepared node for application use
can't check database schema version
github.com/icinga/icingadb/pkg/icingadb.(*DB).CheckSchema
        /go/src/github.com/Icinga/icingadb/pkg/icingadb/db.go:115
main.run
        /go/src/github.com/Icinga/icingadb/cmd/icingadb/main.go:74
main.main
        /go/src/github.com/Icinga/icingadb/cmd/icingadb/main.go:37
runtime.main
        /usr/local/go/src/runtime/proc.go:271
runtime.goexit
        /usr/local/go/src/runtime/asm_amd64.s:1695
exit status 1
```

This change now also retries the initial cluster check.

References #698.
  • Loading branch information
oxzi committed Apr 11, 2024
1 parent 6ff3ca5 commit d76f86f
Showing 1 changed file with 8 additions and 1 deletion.
9 changes: 8 additions & 1 deletion pkg/icingadb/db.go
Original file line number Diff line number Diff line change
Expand Up @@ -110,7 +110,14 @@ func (db *DB) CheckSchema(ctx context.Context) error {

var version uint16

err := db.QueryRowxContext(ctx, "SELECT version FROM icingadb_schema ORDER BY id DESC LIMIT 1").Scan(&version)
err := retry.WithBackoff(
ctx,
func(ctx context.Context) error {
return db.QueryRowxContext(ctx, "SELECT version FROM icingadb_schema ORDER BY id DESC LIMIT 1").Scan(&version)
},
shouldRetry,
backoff.NewExponentialWithJitter(128*time.Millisecond, 1*time.Minute),
db.getDefaultRetrySettings())
if err != nil {
return errors.Wrap(err, "can't check database schema version")
}
Expand Down

0 comments on commit d76f86f

Please sign in to comment.