roachtest: backup-restore/small-ranges failed [relation does not exist]

[cockroach-teamcity]

Note: This build has runtime assertions enabled. If the same failure was hit in a run without assertions enabled, there should be a similar failure without this message. If there isn't one, then this failure is likely due to an assertion violation or (assertion) timeout.

roachtest.backup-restore/small-ranges failed with artifacts on release-24.1.14-rc @ c8459c085e2258f589949e970a527a0342871276:

(monitor.go:149).Wait: monitor failure: schema change workload failed: full command output in run_065758.595367333_n4_COCKROACHRANDOMSEED4.log: COMMAND_PROBLEM: exit status 1 [owner=sql-foundations]
test artifacts and logs in: /artifacts/backup-restore/small-ranges/run_1

Parameters:

• arch=amd64
• cloud=gce
• coverageBuild=false
• cpu=4
• encrypted=false
• fs=ext4
• localSSD=true
• runtimeAssertionsBuild=true
• ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

• roachtest: backup-restore/small-ranges failed #139412 roachtest: backup-restore/small-ranges failed [A-disaster-recovery C-test-failure O-roachtest O-robot P-3 T-disaster-recovery branch-release-23.2]

/cc @cockroachdb/sql-foundations

This test on roachdash | Improve this report!

Jira issue: CRDB-48100

[annrpom]

{
 "workerId": 0,
 "clientTimestamp": "06:58:11.949172",
 "ops": [
  "BEGIN",
  {
   "sql": "ALTER TABLE schema_w0_5.table_w3_456 ADD COLUMN \"col_w3_456_w0_447\" REGROLE UNIQUE"
  },
  "COMMIT"
 ],
 "expectedExecErrors": "",
 "expectedCommitErrors": "",
 "message": "TXN RETRY ERROR; ERROR: restart transaction: error executing PreCommitPhase stage 2 of 2 with 15 MutationType ops: persisting catalog mutations: TransactionRetryWithProtoRefreshError: WriteTooOldError: write for key /Table/3/1/268/2/1 at timestamp 1740466691.948659042,0 too old; must write at or above 1740466692.119215005,1: \"sql txn\" meta={id=ad5f186e key=/Table/3/1/268/2/1 iso=Serializable pri=0.06684582 epo=0 ts=1740466692.119215005,1 min=1740466691.948659042,0 seq=1} lock=true stat=PENDING rts=1740466691.948659042,0 wto=false gul=1740466692.448659042,0 (SQLSTATE 40001)"
}
E250225 06:58:12.243547 1 workload/cli/run.go:568  [-] 3  workload run error: ***UNEXPECTED ERROR; Failed to generate a random operation: ERROR: relation "schema_w0_5.table_w3_456" does not exist (SQLSTATE 42P01)

-- the most recent time table_w3_456 was referenced:
 "ops": [
  "BEGIN",
  {
   "sql": "CREATE TABLE schema_w4_392.table_w4_524 (\"coL524_w4_525\" GEOGRAPHY, 🙀col524_w4_526 STRING COLLATE en_US NOT NULL, \"col524 _w4_527\" OID NULL, col524_w4_528 BPCHAR NULL, \"col%v524_w4_529\" INTERVAL NOT NULL, col524_w4_530 REGROLE NULL, col524_w4_531 TSQUERY NULL, col524_w4_532 PG_LSN NOT NULL, col524_w4_533 FLOAT8, col524_w4_534 REFCURSOR NOT NULL, \"col5 24_w4_535\" STRING AS (lower(CAST(\"coL524_w4_525\" AS STRING))) STORED, UNIQUE (lower(col524_w4_528) ASC, col524_w4_533 ASC, 🙀col524_w4_526 DESC), UNIQUE (🙀col524_w4_526 DESC), UNIQUE (col524_w4_532) STORING (col524_w4_528, col524_w4_531, col524_w4_533, \"col5 24_w4_535\"), INDEX (lower(CAST(col524_w4_534 AS STRING)) DESC) STORING (\"col524 _w4_527\", col524_w4_533, col524_w4_534, \"col5 24_w4_535\") NOT VISIBLE, UNIQUE (col524_w4_528 DESC) STORING (\"col524 _w4_527\", col524_w4_533, col524_w4_534, \"col5 24_w4_535\"), UNIQUE (col524_w4_530 DESC) STORING (\"col524 _w4_527\", col524_w4_531, col524_w4_532, col524_w4_533, col524_w4_534), FAMILY (col524_w4_534), FAMILY (\"coL524_w4_525\"), FAMILY (col524_w4_530, col524_w4_533, col524_w4_532), FAMILY (col524_w4_531), FAMILY (\"col5 24_w4_535\"), FAMILY (\"col524 _w4_527\"), FAMILY (col524_w4_528), FAMILY (\"col%v524_w4_529\", 🙀col524_w4_526))"
  },
  {
   "sql": "ALTER TABLE schema_w0_5.table_w3_456 SET LOCALITY REGIONAL BY ROW",
   "expectedExecErr": "42P16"
  }
 ],
 "expectedExecErrors": "42P16",
 "expectedCommitErrors": "",
 "message": "ROLLBACK; Successfully got expected execution error.: ERROR: cannot alter a table's LOCALITY if its database is not multi-region enabled (SQLSTATE 42P16)"
}

[annrpom]

I believe #140613 was supposed to resolve this and the changes did go into this rc

cockroach/pkg/workload/schemachange/schemachange.go

Lines 230 to 232 in c8459c0

	if err := s.setClusterSettings(ctx, urls[0]); err != nil {
	return workload.QueryLoad{}, err
	}