roachtest: tpce/c=5000/nodes=3 failed

[cockroach-teamcity]

roachtest.tpce/c=5000/nodes=3 failed with artifacts on release-24.1.14-rc @ c8459c085e2258f589949e970a527a0342871276:

(test_runner.go:1260).runTest: test timed out (4h0m0s)
test artifacts and logs in: /artifacts/tpce/c=5000/nodes=3/run_1

Parameters:

• arch=amd64
• cloud=gce
• coverageBuild=false
• cpu=4
• encrypted=false
• fs=ext4
• localSSD=true
• runtimeAssertionsBuild=false
• ssd=1

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

• roachtest: tpce/c=5000/nodes=3 failed [auto generic query plans] #137018 roachtest: tpce/c=5000/nodes=3 failed [auto generic query plans] [C-test-failure O-roachtest O-robot P-2 T-sql-queries branch-master branch-release-25.1]

/cc @cockroachdb/test-eng

This test on roachdash | Improve this report!

Jira issue: CRDB-48082

[srosenberg]

Here is a relevant timeline,

11:44:15 teamcity-18912588-1740378613-118-n4cpu4-0004 sudo[21946]:   ubuntu : PWD=/home/ubuntu ; USER=root ; COMMAND=/usr/bin/docker run us-east1-docker.pkg.dev/crl-ci-images/cockroach/tpc-e:master --bucket gs://cockroach-fixtures-us-east1/tpce-csv --customers 5000 --init --pg-password cockroachdb --pg-user roachprod --racks 3 --pg-port=29000 --hosts=10.142.1.25
..
I250224 12:54:08.713842 9217 jobs/registry.go:1554 â‹® [T1,Vsystem,n1] 1771  IMPORT job 1049647706053640193: stepping through state succeeded
...
W250224 13:22:30.368966 219888 sql/schemachanger/scbuild/builder_state.go:186 ⋮ [T1,Vsystem,n1,client=10.142.1.103:41936,hostssl,user=‹roachprod›] 2376  failed building declarative schema change targets for CREATE INDEX with error: ‹descriptor 120 is undergoing another schema change›
...
I250224 13:51:25.358008 219888 sql/schema_change_plan_node.go:178 ⋮ [T1,Vsystem,n1,client=10.142.1.103:41936,hostssl,user=‹roachprod›] 3412  done waiting for concurrent schema changes on descriptor 120 after 28m54.983098413s
...
I250224 13:51:25.401320 307541 sql/gcjob/index_garbage_collection.go:165 â‹® [T1,Vsystem,n1,job=SCHEMA CHANGE GC id=1049672696800968705] 3418  clearing index 3 from table 120
...
I250224 14:14:20.059677 356643 sql/gcjob/index_garbage_collection.go:165 â‹® [T1,Vsystem,n1,job=SCHEMA CHANGE GC id=1049677200892559361] 3583  clearing index 5 from table 120
...
14:15:00 teamcity-18912588-1740378613-118-n4cpu4-0004 sudo[22592]:   ubuntu : PWD=/home/ubuntu ; USER=root ; COMMAND=/usr/bin/docker run us-east1-docker.pkg.dev/crl-ci-images/cockroach/tpc-e:master --bucket gs://cockroach-fixtures-us-east1/tpce-csv --customers 5000 --duration 2h0m0s --pg-password cockroachdb --pg-user roachprod --racks 3 --threads 12 --pg-port=29000 --hosts=10.142.1.25 --hosts=10.142.1.77 --hosts=10.142.1.69

The load took ~2.5h, out of which IMPORT took ~1h, 10m, and the rest was index creation. The workload is roughly ~30m behind because its specified duration is 2h, but the test's timeout is 4h.