Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

roachtest: ssh_problem failed #132980

Open
cockroach-teamcity opened this issue Oct 19, 2024 · 143 comments
Open

roachtest: ssh_problem failed #132980

cockroach-teamcity opened this issue Oct 19, 2024 · 143 comments
Labels
branch-release-24.3 Used to mark GA and release blockers, technical advisories, and bugs for 24.3 O-roachtest O-robot Originated from a bot. T-testeng TestEng Team X-infra-flake the automatically generated issue was closed due to an infrastructure problem not a product issue

Comments

@cockroach-teamcity
Copy link
Member

cockroach-teamcity commented Oct 19, 2024 •
edited by cockroach-jira-scripts
Loading

roachtest.ssh_problem failed with artifacts on release-24.3 @ 4cbedefd790c75cb0f21f77ed8d917c8528a7d15:

test schemachange/leasing-benchmark failed: (multiregion_leasing.go:185).func2: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/schemachange/leasing-benchmark/run_1/ssh/ssh_083711.272178191_n3_time-p-cockroach-sql.log): TRANSIENT_ERROR(ssh_problem): exit status 255
test artifacts and logs in: /artifacts/schemachange/leasing-benchmark/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=aws
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=4
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_runtimeAssertionsBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Grafana is not yet available for aws clusters

Same failure on other branches

/cc @cockroachdb/test-eng

This test on roachdash | Improve this report!

Jira issue: CRDB-43378
@cockroach-teamcity cockroach-teamcity added branch-release-24.3 Used to mark GA and release blockers, technical advisories, and bugs for 24.3 O-roachtest O-robot Originated from a bot. T-testeng TestEng Team X-infra-flake the automatically generated issue was closed due to an infrastructure problem not a product issue labels Oct 19, 2024
This was referenced Oct 19, 2024
Open
Open
@cockroach-teamcity
Copy link
Member Author

roachtest.ssh_problem failed with artifacts on release-24.3 @ e466862ab36572bdc894cec84536b45472bdeb5b:

test kv95/enc=false/nodes=3/cpu=32/size=64kb failed: (cluster.go:2449).Run: full command output in run_074755.528724630_n4_cockroach-workload-r.log: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/kv95/enc=false/nodes=3/cpu=32/size=64kb/run_1/ssh/ssh_074755.679983329_n4_cockroach-workload-r.log): TRANSIENT_ERROR(ssh_problem): exit status 255
(monitor.go:149).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/kv95/enc=false/nodes=3/cpu=32/size=64kb/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=aws
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=32
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_runtimeAssertionsBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Grafana is not yet available for aws clusters

Same failure on other branches

This test on roachdash | Improve this report!

This was referenced Oct 20, 2024
Open
Open
@cockroach-teamcity
Copy link
Member Author

roachtest.ssh_problem failed with artifacts on release-24.3 @ b2d2353b876af1748607e155ecdfed9d4bba29d3:

test kv0/enc=false/nodes=3/batch=16 failed: (cluster.go:2449).Run: full command output in run_064717.580008948_n4_cockroach-workload-r.log: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/kv0/enc=false/nodes=3/batch=16/run_1/ssh/ssh_064717.766004446_n4_cockroach-workload-r.log): TRANSIENT_ERROR(ssh_problem): exit status 255
(monitor.go:149).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/kv0/enc=false/nodes=3/batch=16/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=aws
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=8
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_runtimeAssertionsBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Grafana is not yet available for aws clusters

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.ssh_problem failed with artifacts on release-24.3 @ b2d2353b876af1748607e155ecdfed9d4bba29d3:

test cdc/tpcc-1000/sink=kafka failed: (cluster.go:2449).Run: full command output in run_063816.003489090_n4_cockroach-workload-r.log: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/cdc/tpcc-1000/sink=kafka/run_1/ssh/ssh_063816.124165214_n4_cockroach-workload-r.log): TRANSIENT_ERROR(ssh_problem): exit status 255
(cdc.go:394).Close: error shutting down prometheus/grafana: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/cdc/tpcc-1000/sink=kafka/run_1/ssh/ssh_070502.671249278_n4_sudo-rm-rf-tmppromet.log): TRANSIENT_ERROR(ssh_problem): exit status 255
test artifacts and logs in: /artifacts/cdc/tpcc-1000/sink=kafka/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=aws
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=16
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_runtimeAssertionsBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Grafana is not yet available for aws clusters

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.ssh_problem failed with artifacts on release-24.3 @ b2d2353b876af1748607e155ecdfed9d4bba29d3:

test kv95/enc=false/nodes=3/size=64kb failed: (cluster.go:2449).Run: full command output in run_073828.558591078_n4_cockroach-workload-r.log: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/kv95/enc=false/nodes=3/size=64kb/run_1/ssh/ssh_073828.591333526_n4_cockroach-workload-r.log): TRANSIENT_ERROR(ssh_problem): exit status 255
(monitor.go:149).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/kv95/enc=false/nodes=3/size=64kb/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=aws
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=8
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_runtimeAssertionsBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Grafana is not yet available for aws clusters

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.ssh_problem failed with artifacts on release-24.3 @ 330f659a3f244d1386bb86c89a258d2da6063959:

test schemachange/leasing-benchmark failed: (multiregion_leasing.go:185).func2: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/schemachange/leasing-benchmark/cpu_arch=arm64/run_1/ssh/ssh_090057.433708232_n1_time-p-cockroach-sql.log): TRANSIENT_ERROR(ssh_problem): exit status 255
test artifacts and logs in: /artifacts/schemachange/leasing-benchmark/cpu_arch=arm64/run_1

Parameters:

  • ROACHTEST_arch=arm64
  • ROACHTEST_cloud=aws
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=4
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_runtimeAssertionsBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Grafana is not yet available for aws clusters

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.ssh_problem failed with artifacts on release-24.3 @ f4bf28879d49979d438967379b44c2fe3464f968:

test kv0/enc=false/nodes=3/size=4kb failed: (cluster.go:2449).Run: full command output in run_065721.591288583_n4_cockroach-workload-r.log: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/kv0/enc=false/nodes=3/size=4kb/run_1/ssh/ssh_065721.671060465_n4_cockroach-workload-r.log): TRANSIENT_ERROR(ssh_problem): exit status 255
(monitor.go:149).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/kv0/enc=false/nodes=3/size=4kb/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=aws
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=8
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_runtimeAssertionsBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Grafana is not yet available for aws clusters

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity cockroach-teamcity mentioned this issue Oct 24, 2024
Open
@cockroach-teamcity
Copy link
Member Author

roachtest.ssh_problem failed with artifacts on release-24.3 @ 36f5b311f33775bbeb26d56003f6831ec9ddd837:

test kv0/enc=false/nodes=3/size=64kb failed: (cluster.go:2449).Run: full command output in run_065516.514917978_n4_cockroach-workload-r.log: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/kv0/enc=false/nodes=3/size=64kb/cpu_arch=arm64/run_1/ssh/ssh_065516.544153623_n4_cockroach-workload-r.log): TRANSIENT_ERROR(ssh_problem): exit status 255
(monitor.go:149).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/kv0/enc=false/nodes=3/size=64kb/cpu_arch=arm64/run_1

Parameters:

  • ROACHTEST_arch=arm64
  • ROACHTEST_cloud=aws
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=8
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_runtimeAssertionsBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Grafana is not yet available for aws clusters

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.ssh_problem failed with artifacts on release-24.3 @ 36f5b311f33775bbeb26d56003f6831ec9ddd837:

test kv0/enc=false/nodes=3/lease=expiration failed: (cluster.go:2449).Run: full command output in run_065049.903823631_n4_cockroach-workload-r.log: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/kv0/enc=false/nodes=3/lease=expiration/cpu_arch=arm64/run_1/ssh/ssh_065049.996135013_n4_cockroach-workload-r.log): TRANSIENT_ERROR(ssh_problem): exit status 255
(monitor.go:149).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/kv0/enc=false/nodes=3/lease=expiration/cpu_arch=arm64/run_1

Parameters:

  • ROACHTEST_arch=arm64
  • ROACHTEST_cloud=aws
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=8
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_runtimeAssertionsBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Grafana is not yet available for aws clusters

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.ssh_problem failed with artifacts on release-24.3 @ 36f5b311f33775bbeb26d56003f6831ec9ddd837:

test kv0/enc=false/nodes=3/cpu=32/size=64kb failed: (cluster.go:2449).Run: full command output in run_064926.334981220_n4_cockroach-workload-r.log: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/kv0/enc=false/nodes=3/cpu=32/size=64kb/cpu_arch=arm64/run_1/ssh/ssh_064926.428321082_n4_cockroach-workload-r.log): TRANSIENT_ERROR(ssh_problem): exit status 255
(monitor.go:149).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/kv0/enc=false/nodes=3/cpu=32/size=64kb/cpu_arch=arm64/run_1

Parameters:

  • ROACHTEST_arch=arm64
  • ROACHTEST_cloud=aws
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=32
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_runtimeAssertionsBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Grafana is not yet available for aws clusters

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.ssh_problem failed with artifacts on release-24.3 @ 36f5b311f33775bbeb26d56003f6831ec9ddd837:

test kv0/enc=false/nodes=3/mt-shared-process failed: (cluster.go:2449).Run: full command output in run_065224.573626763_n4_cockroach-workload-r.log: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/kv0/enc=false/nodes=3/mt-shared-process/cpu_arch=arm64/run_1/ssh/ssh_065224.601736097_n4_cockroach-workload-r.log): TRANSIENT_ERROR(ssh_problem): exit status 255
(monitor.go:149).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/kv0/enc=false/nodes=3/mt-shared-process/cpu_arch=arm64/run_1

Parameters:

  • ROACHTEST_arch=arm64
  • ROACHTEST_cloud=aws
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=8
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_runtimeAssertionsBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Grafana is not yet available for aws clusters

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.ssh_problem failed with artifacts on release-24.3 @ 36f5b311f33775bbeb26d56003f6831ec9ddd837:

test admission-control/disk-bandwidth-limiter failed: (cluster.go:2449).Run: full command output in run_061459.654053966_n2_cockroach-workload-r.log: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/admission-control/disk-bandwidth-limiter/run_1/ssh/ssh_061459.683132830_n2_cockroach-workload-r.log): TRANSIENT_ERROR(ssh_problem): exit status 255
(cluster.go:2449).Run: context canceled
(cluster.go:2890).Conn: connecting to node 1: unknown cluster: teamcity-17434700-1729749107-17-n2cpu8
test artifacts and logs in: /artifacts/admission-control/disk-bandwidth-limiter/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=aws
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=8
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_runtimeAssertionsBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Grafana is not yet available for aws clusters

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.ssh_problem failed with artifacts on release-24.3 @ 3839fb143f98e5009a994ad58fb6b82dc7badc55:

test loqrecovery/workload=movr/rangeSize=default failed: (cluster.go:2336).Start: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/loqrecovery/workload=movr/rangeSize=default/run_1/ssh/ssh_172643.519297946_n2_upload-start-script.log): TRANSIENT_ERROR(ssh_problem): exit status 255
test artifacts and logs in: /artifacts/loqrecovery/workload=movr/rangeSize=default/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=4
  • ROACHTEST_encrypted=false
  • ROACHTEST_runtimeAssertionsBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.ssh_problem failed with artifacts on release-24.3 @ 416acdb7560e4548f3b82c5a522ef48592471c7d:

test kv95/enc=false/nodes=1 failed: (cluster.go:2449).Run: full command output in run_071518.773025344_n2_cockroach-workload-r.log: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/kv95/enc=false/nodes=1/cpu_arch=arm64/run_1/ssh/ssh_071518.883881580_n2_cockroach-workload-r.log): TRANSIENT_ERROR(ssh_problem): exit status 255
(monitor.go:149).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/kv95/enc=false/nodes=1/cpu_arch=arm64/run_1

Parameters:

  • ROACHTEST_arch=arm64
  • ROACHTEST_cloud=aws
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=8
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_runtimeAssertionsBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Grafana is not yet available for aws clusters

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.ssh_problem failed with artifacts on release-24.3 @ c357ebbfdd72bb6aecef57c84850845b5955382f:

test kv0/enc=true/nodes=3 failed: (cluster.go:2449).Run: full command output in run_071947.040050133_n4_cockroach-workload-r.log: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/kv0/enc=true/nodes=3/cpu_arch=arm64/run_1/ssh/ssh_071947.131412213_n4_cockroach-workload-r.log): TRANSIENT_ERROR(ssh_problem): exit status 255
(monitor.go:149).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/kv0/enc=true/nodes=3/cpu_arch=arm64/run_1

Parameters:

  • ROACHTEST_arch=arm64
  • ROACHTEST_cloud=aws
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=8
  • ROACHTEST_encrypted=true
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_runtimeAssertionsBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Grafana is not yet available for aws clusters

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.ssh_problem failed with artifacts on release-24.3 @ c357ebbfdd72bb6aecef57c84850845b5955382f:

test c2c/tpcc/warehouses=1000/duration=60/cutover=30 failed: (test_runner.go:846).func4: cluster.PutE: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/_runner-logs/ssh/ssh_093702.109275912_n1_if-hostname-teamcity.log): TRANSIENT_ERROR(ssh_problem): exit status 255 [owner=test-eng]

Parameters:

  • ROACHTEST_arch=arm64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=8
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=false
  • ROACHTEST_runtimeAssertionsBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity cockroach-teamcity mentioned this issue Oct 31, 2024
Open
@cockroach-teamcity
Copy link
Member Author

roachtest.ssh_problem failed with artifacts on release-24.3 @ b1fbd0c00f283c147b0c3a36a8f25ce7eec41dff:

test schemachange/leasing-benchmark failed: (cluster.go:2386).Stop: cluster.StopE: one or more parallel execution failure(s): _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/schemachange/leasing-benchmark/cpu_arch=arm64/run_1/ssh/ssh_084552.497492334_n2_kill.log): TRANSIENT_ERROR(ssh_problem): exit status 255
test artifacts and logs in: /artifacts/schemachange/leasing-benchmark/cpu_arch=arm64/run_1

Parameters:

  • ROACHTEST_arch=arm64
  • ROACHTEST_cloud=aws
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=4
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_runtimeAssertionsBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Grafana is not yet available for aws clusters

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity cockroach-teamcity mentioned this issue Nov 1, 2024
Open
@cockroach-teamcity
Copy link
Member Author

roachtest.ssh_problem failed with artifacts on release-24.3 @ bbce415047c9896ee3b33b1eb4c06e3d2cab5bd6:

test admission-control/disk-bandwidth-limiter failed: (cluster.go:2451).Run: full command output in run_071204.119661807_n2_cockroach-workload-r.log: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/admission-control/disk-bandwidth-limiter/run_1/ssh/ssh_071204.172079649_n2_cockroach-workload-r.log): TRANSIENT_ERROR(ssh_problem): exit status 255
(cluster.go:2451).Run: context canceled
(cluster.go:2892).Conn: connecting to node 1: unknown cluster: teamcity-17578560-1730702910-17-n2cpu8
test artifacts and logs in: /artifacts/admission-control/disk-bandwidth-limiter/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=aws
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=8
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_runtimeAssertionsBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Grafana is not yet available for aws clusters

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.ssh_problem failed with artifacts on release-24.3 @ 3bb549350582192ffa091bfecba15527e545e30b:

test kv0/enc=false/nodes=3/cpu=32/mt-shared-process failed: (cluster.go:2455).Run: full command output in run_071349.595599019_n4_cockroach-workload-r.log: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/kv0/enc=false/nodes=3/cpu=32/mt-shared-process/cpu_arch=arm64/run_1/ssh/ssh_071349.632042335_n4_cockroach-workload-r.log): TRANSIENT_ERROR(ssh_problem): exit status 255
(monitor.go:149).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/kv0/enc=false/nodes=3/cpu=32/mt-shared-process/cpu_arch=arm64/run_1

Parameters:

  • arch=arm64
  • cloud=aws
  • coverageBuild=false
  • cpu=32
  • encrypted=false
  • fs=ext4
  • localSSD=true
  • runtimeAssertionsBuild=false
  • ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Grafana is not yet available for aws clusters

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.ssh_problem failed with artifacts on release-24.3 @ 3bb549350582192ffa091bfecba15527e545e30b:

test kv0/enc=false/nodes=1 failed: (cluster.go:2455).Run: full command output in run_065317.153743329_n2_cockroach-workload-r.log: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/kv0/enc=false/nodes=1/run_1/ssh/ssh_065317.172707602_n2_cockroach-workload-r.log): TRANSIENT_ERROR(ssh_problem): exit status 255
(monitor.go:149).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/kv0/enc=false/nodes=1/run_1

Parameters:

  • arch=amd64
  • cloud=aws
  • coverageBuild=false
  • cpu=8
  • encrypted=false
  • fs=ext4
  • localSSD=true
  • runtimeAssertionsBuild=false
  • ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Grafana is not yet available for aws clusters

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.ssh_problem failed with artifacts on release-24.3 @ 0cf63b7d71789277dc5a235fe54771cfb19b2520:

test jepsen/bank-multitable/subcritical-skews-start-kill-2 failed: (test_runner.go:852).func4: cluster.PutE: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/_runner-logs/ssh/ssh_120428.588006935_n1_if-hostname-teamcity.log): TRANSIENT_ERROR(ssh_problem): exit status 255 [owner=test-eng]

Parameters:

  • arch=arm64
  • cloud=gce
  • coverageBuild=false
  • cpu=4
  • encrypted=false
  • fs=ext4
  • localSSD=false
  • runtimeAssertionsBuild=false
  • ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.ssh_problem failed with artifacts on release-24.3 @ eaa38b5fe50c68d6de15a17b984d937c8af6f94d:

test failover/non-system/pause/lease=leader failed: (cluster.go:2419).Signal: cluster.Signal: one or more parallel execution failure(s): _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/failover/non-system/pause/lease=leader/run_1/ssh/ssh_095856.057657005_n5_kill.log): TRANSIENT_ERROR(ssh_problem): exit status 255
(monitor.go:149).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/failover/non-system/pause/lease=leader/run_1

Parameters:

  • arch=amd64
  • cloud=gce
  • coverageBuild=false
  • cpu=2
  • encrypted=false
  • fs=ext4
  • localSSD=true
  • runtimeAssertionsBuild=false
  • ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.ssh_problem failed with artifacts on release-24.3 @ eaa38b5fe50c68d6de15a17b984d937c8af6f94d:

test rebalance/by-load/replicas/mixed-version failed: (mixedversion.go:744).Run: mixed-version test failure while running step 31 (restart node 6 with binary version release-24.3): cluster.StopE: one or more parallel execution failure(s): _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/rebalance/by-load/replicas/mixed-version/run_1/ssh/ssh_122133.624721774_n6_kill.log): TRANSIENT_ERROR(ssh_problem): exit status 255
test artifacts and logs in: /artifacts/rebalance/by-load/replicas/mixed-version/run_1

Parameters:

  • arch=amd64
  • cloud=gce
  • coverageBuild=false
  • cpu=4
  • encrypted=false
  • fs=ext4
  • localSSD=true
  • mvtDeploymentMode=shared-process
  • mvtVersions=v24.2.8 → release-24.3
  • runtimeAssertionsBuild=false
  • ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.ssh_problem failed with artifacts on release-24.3 @ d0be0aaa78740d6d24a83bf6500d433874b48905:

test kv0/enc=false/nodes=3/cpu=32/mt-shared-process failed: (cluster.go:2455).Run: full command output in run_070442.361514181_n4_cockroach-workload-r.log: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/kv0/enc=false/nodes=3/cpu=32/mt-shared-process/run_1/ssh/ssh_070442.396524886_n4_cockroach-workload-r.log): TRANSIENT_ERROR(ssh_problem): exit status 255
(monitor.go:149).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/kv0/enc=false/nodes=3/cpu=32/mt-shared-process/run_1

Parameters:

  • arch=amd64
  • cloud=aws
  • coverageBuild=false
  • cpu=32
  • encrypted=false
  • fs=ext4
  • localSSD=true
  • runtimeAssertionsBuild=false
  • ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Grafana is not yet available for aws clusters

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.ssh_problem failed with artifacts on release-24.3 @ 97e1ded1bb5a59e86ba43329afac4ca53ca08c1e:

test kv95/enc=false/nodes=3/mt-shared-process failed: (cluster.go:2455).Run: full command output in run_072251.910341352_n4_cockroach-workload-r.log: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/kv95/enc=false/nodes=3/mt-shared-process/cpu_arch=arm64/run_1/ssh/ssh_072251.951075140_n4_cockroach-workload-r.log): TRANSIENT_ERROR(ssh_problem): exit status 255
(monitor.go:149).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/kv95/enc=false/nodes=3/mt-shared-process/cpu_arch=arm64/run_1

Parameters:

  • arch=arm64
  • cloud=aws
  • coverageBuild=false
  • cpu=8
  • encrypted=false
  • runtimeAssertionsBuild=false
  • ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Grafana is not yet available for aws clusters

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.ssh_problem failed with artifacts on release-24.3 @ 97e1ded1bb5a59e86ba43329afac4ca53ca08c1e:

test cdc/tpcc-1000/sink=kafka failed: (cluster.go:2455).Run: full command output in run_064356.316931880_n4_cockroach-workload-r.log: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/cdc/tpcc-1000/sink=kafka/cpu_arch=arm64/run_1/ssh/ssh_064356.363235064_n4_cockroach-workload-r.log): TRANSIENT_ERROR(ssh_problem): exit status 255
(cdc.go:394).Close: error shutting down prometheus/grafana: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/cdc/tpcc-1000/sink=kafka/cpu_arch=arm64/run_1/ssh/ssh_080444.174186670_n4_sudo-rm-rf-tmppromet.log): TRANSIENT_ERROR(ssh_problem): exit status 255
test artifacts and logs in: /artifacts/cdc/tpcc-1000/sink=kafka/cpu_arch=arm64/run_1

Parameters:

  • arch=arm64
  • cloud=aws
  • coverageBuild=false
  • cpu=16
  • encrypted=false
  • fs=ext4
  • localSSD=true
  • runtimeAssertionsBuild=false
  • ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Grafana is not yet available for aws clusters

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.ssh_problem failed with artifacts on release-24.3 @ af56876f2593fbc5c763c3dc389fae31d19de688:

test kv0/enc=false/nodes=1/cpu=32 failed: (cluster.go:2455).Run: full command output in run_065406.397953744_n2_cockroach-workload-r.log: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/kv0/enc=false/nodes=1/cpu=32/cpu_arch=arm64/run_1/ssh/ssh_065406.429907685_n2_cockroach-workload-r.log): TRANSIENT_ERROR(ssh_problem): exit status 255
(monitor.go:149).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/kv0/enc=false/nodes=1/cpu=32/cpu_arch=arm64/run_1

Parameters:

  • arch=arm64
  • cloud=aws
  • coverageBuild=false
  • cpu=32
  • encrypted=false
  • fs=ext4
  • localSSD=true
  • runtimeAssertionsBuild=false
  • ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Grafana is not yet available for aws clusters

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.ssh_problem failed with artifacts on release-24.3 @ af56876f2593fbc5c763c3dc389fae31d19de688:

test kv0/enc=false/nodes=3/cpu=32/mvcc-range-keys=global failed: (cluster.go:2455).Run: full command output in run_070438.021139694_n4_cockroach-workload-r.log: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/kv0/enc=false/nodes=3/cpu=32/mvcc-range-keys=global/cpu_arch=arm64/run_1/ssh/ssh_070438.069104845_n4_cockroach-workload-r.log): TRANSIENT_ERROR(ssh_problem): exit status 255
(monitor.go:149).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/kv0/enc=false/nodes=3/cpu=32/mvcc-range-keys=global/cpu_arch=arm64/run_1

Parameters:

  • arch=arm64
  • cloud=aws
  • coverageBuild=false
  • cpu=32
  • encrypted=false
  • fs=ext4
  • localSSD=true
  • runtimeAssertionsBuild=false
  • ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Grafana is not yet available for aws clusters

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.ssh_problem failed with artifacts on release-24.3 @ af56876f2593fbc5c763c3dc389fae31d19de688:

test cdc/tpcc-1000/sink=kafka failed: (cluster.go:2455).Run: full command output in run_065433.509327746_n4_cockroach-workload-r.log: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/cdc/tpcc-1000/sink=kafka/cpu_arch=arm64/run_1/ssh/ssh_065433.577935180_n4_cockroach-workload-r.log): TRANSIENT_ERROR(ssh_problem): exit status 255
(cdc.go:394).Close: error shutting down prometheus/grafana: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/cdc/tpcc-1000/sink=kafka/cpu_arch=arm64/run_1/ssh/ssh_071918.235735487_n4_sudo-rm-rf-tmppromet.log): TRANSIENT_ERROR(ssh_problem): exit status 255
test artifacts and logs in: /artifacts/cdc/tpcc-1000/sink=kafka/cpu_arch=arm64/run_1

Parameters:

  • arch=arm64
  • cloud=aws
  • coverageBuild=false
  • cpu=16
  • encrypted=false
  • fs=ext4
  • localSSD=true
  • runtimeAssertionsBuild=false
  • ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Grafana is not yet available for aws clusters

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.ssh_problem failed with artifacts on release-24.3 @ af56876f2593fbc5c763c3dc389fae31d19de688:

test tpccbench/nodes=3/cpu=16 failed: (cluster.go:2392).Stop: cluster.StopE: one or more parallel execution failure(s): _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/tpccbench/nodes=3/cpu=16/cpu_arch=arm64/run_1/ssh/ssh_080903.978818270_n2_kill.log): TRANSIENT_ERROR(ssh_problem): exit status 255
test artifacts and logs in: /artifacts/tpccbench/nodes=3/cpu=16/cpu_arch=arm64/run_1

Parameters:

  • arch=arm64
  • cloud=aws
  • coverageBuild=false
  • cpu=16
  • encrypted=false
  • fs=ext4
  • localSSD=true
  • runtimeAssertionsBuild=false
  • ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Grafana is not yet available for aws clusters

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.ssh_problem failed with artifacts on release-24.3 @ 754142b1d85d078c7e1c4f67539ba1e78d413234:

test kv0/enc=false/nodes=3 failed: (cluster.go:2455).Run: full command output in run_065510.724634954_n4_cockroach-workload-r.log: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/kv0/enc=false/nodes=3/cpu_arch=arm64/run_1/ssh/ssh_065510.780773075_n4_cockroach-workload-r.log): TRANSIENT_ERROR(ssh_problem): exit status 255
(monitor.go:149).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/kv0/enc=false/nodes=3/cpu_arch=arm64/run_1

Parameters:

  • arch=arm64
  • cloud=aws
  • coverageBuild=false
  • cpu=8
  • encrypted=false
  • fs=ext4
  • localSSD=true
  • runtimeAssertionsBuild=false
  • ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Grafana is not yet available for aws clusters

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.ssh_problem failed with artifacts on release-24.3 @ 754142b1d85d078c7e1c4f67539ba1e78d413234:

test kv95/enc=false/nodes=3/cpu=32/mvcc-range-keys=global failed: (cluster.go:2455).Run: full command output in run_073521.691222075_n4_cockroach-workload-r.log: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/kv95/enc=false/nodes=3/cpu=32/mvcc-range-keys=global/cpu_arch=arm64/run_1/ssh/ssh_073521.761709707_n4_cockroach-workload-r.log): TRANSIENT_ERROR(ssh_problem): exit status 255
(monitor.go:149).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/kv95/enc=false/nodes=3/cpu=32/mvcc-range-keys=global/cpu_arch=arm64/run_1

Parameters:

  • arch=arm64
  • cloud=aws
  • coverageBuild=false
  • cpu=32
  • encrypted=false
  • fs=ext4
  • localSSD=true
  • runtimeAssertionsBuild=false
  • ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Grafana is not yet available for aws clusters

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.ssh_problem failed with artifacts on release-24.3 @ 754142b1d85d078c7e1c4f67539ba1e78d413234:

test restore/tpce/15GB/aws/nodes=4/cpus=8 failed: (cluster.go:2342).Start: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/restore/tpce/15GB/aws/nodes=4/cpus=8/run_1/ssh/ssh_075500.854879672_n3_upload-start-script.log): TRANSIENT_ERROR(ssh_problem): exit status 255
test artifacts and logs in: /artifacts/restore/tpce/15GB/aws/nodes=4/cpus=8/run_1

Parameters:

  • arch=amd64
  • cloud=aws
  • coverageBuild=false
  • cpu=8
  • encrypted=false
  • fs=ext4
  • localSSD=false
  • runtimeAssertionsBuild=false
  • ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Grafana is not yet available for aws clusters

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.ssh_problem failed with artifacts on release-24.3 @ 754142b1d85d078c7e1c4f67539ba1e78d413234:

test cdc/tpcc-1000/sink=kafka failed: (cluster.go:2455).Run: full command output in run_065039.473774453_n4_cockroach-workload-r.log: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/cdc/tpcc-1000/sink=kafka/run_1/ssh/ssh_065039.550975107_n4_cockroach-workload-r.log): TRANSIENT_ERROR(ssh_problem): exit status 255
(cdc.go:394).Close: error shutting down prometheus/grafana: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/cdc/tpcc-1000/sink=kafka/run_1/ssh/ssh_081630.713543684_n4_sudo-rm-rf-tmppromet.log): TRANSIENT_ERROR(ssh_problem): exit status 255
test artifacts and logs in: /artifacts/cdc/tpcc-1000/sink=kafka/run_1

Parameters:

  • arch=amd64
  • cloud=aws
  • coverageBuild=false
  • cpu=16
  • encrypted=false
  • fs=ext4
  • localSSD=true
  • runtimeAssertionsBuild=false
  • ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Grafana is not yet available for aws clusters

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.ssh_problem failed with artifacts on release-24.3 @ 12043c7a951ecfb224dd2c71289c80a2729197b1:

test kv0/enc=false/nodes=3 failed: (cluster.go:2455).Run: full command output in run_064738.207469709_n4_cockroach-workload-r.log: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/kv0/enc=false/nodes=3/cpu_arch=arm64/run_1/ssh/ssh_064738.244314584_n4_cockroach-workload-r.log): TRANSIENT_ERROR(ssh_problem): exit status 255
(monitor.go:149).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/kv0/enc=false/nodes=3/cpu_arch=arm64/run_1

Parameters:

  • arch=arm64
  • cloud=aws
  • coverageBuild=false
  • cpu=8
  • encrypted=false
  • fs=ext4
  • localSSD=true
  • runtimeAssertionsBuild=false
  • ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Grafana is not yet available for aws clusters

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.ssh_problem failed with artifacts on release-24.3 @ 303bfcd84f0820ab42ac5bc2384e93ad90160d57:

test kv95/enc=false/nodes=3/cpu=32 failed: (cluster.go:2473).Run: full command output in run_072133.504085689_n4_cockroach-workload-r.log: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/kv95/enc=false/nodes=3/cpu=32/cpu_arch=arm64/run_1/ssh/ssh_072133.549025514_n4_cockroach-workload-r.log): TRANSIENT_ERROR(ssh_problem): exit status 255
(monitor.go:149).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/kv95/enc=false/nodes=3/cpu=32/cpu_arch=arm64/run_1

Parameters:

  • arch=arm64
  • cloud=aws
  • coverageBuild=false
  • cpu=32
  • encrypted=false
  • runtimeAssertionsBuild=false
  • ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Grafana is not yet available for aws clusters

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.ssh_problem failed with artifacts on release-24.3 @ 2d069809fb0ffd915a9d76639bd43e1c420b8557:

test kv95/enc=false/nodes=1 failed: (cluster.go:2473).Run: full command output in run_070824.578409989_n2_cockroach-workload-r.log: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/kv95/enc=false/nodes=1/run_1/ssh/ssh_070824.616830396_n2_cockroach-workload-r.log): TRANSIENT_ERROR(ssh_problem): exit status 255
(monitor.go:149).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/kv95/enc=false/nodes=1/run_1

Parameters:

  • arch=amd64
  • cloud=aws
  • coverageBuild=false
  • cpu=8
  • encrypted=false
  • fs=ext4
  • localSSD=true
  • runtimeAssertionsBuild=false
  • ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Grafana is not yet available for aws clusters

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.ssh_problem failed with artifacts on release-24.3 @ 2d069809fb0ffd915a9d76639bd43e1c420b8557:

test kv95/enc=false/nodes=3/lease=expiration failed: (cluster.go:2473).Run: full command output in run_072102.542533797_n4_cockroach-workload-r.log: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/kv95/enc=false/nodes=3/lease=expiration/cpu_arch=arm64/run_1/ssh/ssh_072102.592207843_n4_cockroach-workload-r.log): TRANSIENT_ERROR(ssh_problem): exit status 255
(monitor.go:149).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/kv95/enc=false/nodes=3/lease=expiration/cpu_arch=arm64/run_1

Parameters:

  • arch=arm64
  • cloud=aws
  • coverageBuild=false
  • cpu=8
  • encrypted=false
  • runtimeAssertionsBuild=false
  • ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Grafana is not yet available for aws clusters

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.ssh_problem failed with artifacts on release-24.3 @ c161558d387d45fa8569f7a63428938302ef368b:

test kv0/enc=false/nodes=1/mt-shared-process failed: (cluster.go:2473).Run: full command output in run_064819.231170060_n2_cockroach-workload-r.log: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/kv0/enc=false/nodes=1/mt-shared-process/run_1/ssh/ssh_064819.252999096_n2_cockroach-workload-r.log): TRANSIENT_ERROR(ssh_problem): exit status 255
(monitor.go:149).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/kv0/enc=false/nodes=1/mt-shared-process/run_1

Parameters:

  • arch=amd64
  • cloud=aws
  • coverageBuild=false
  • cpu=8
  • encrypted=false
  • fs=ext4
  • localSSD=true
  • runtimeAssertionsBuild=false
  • ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Grafana is not yet available for aws clusters

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.ssh_problem failed with artifacts on release-24.3 @ 7d3c3da3a2daa0889bee14b1b815a2f5e86b44a7:

test kv0/enc=false/nodes=3/cpu=96 failed: (cluster.go:2473).Run: full command output in run_065316.205443622_n4_cockroach-workload-r.log: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/kv0/enc=false/nodes=3/cpu=96/run_1/ssh/ssh_065316.233107473_n4_cockroach-workload-r.log): TRANSIENT_ERROR(ssh_problem): exit status 255
(monitor.go:149).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/kv0/enc=false/nodes=3/cpu=96/run_1

Parameters:

  • arch=amd64
  • cloud=aws
  • coverageBuild=false
  • cpu=96
  • encrypted=false
  • fs=ext4
  • localSSD=false
  • runtimeAssertionsBuild=false
  • ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Grafana is not yet available for aws clusters

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.ssh_problem failed with artifacts on release-24.3 @ 7d3c3da3a2daa0889bee14b1b815a2f5e86b44a7:

test kv0/enc=true/nodes=3 failed: (cluster.go:2473).Run: full command output in run_071436.947241857_n4_cockroach-workload-r.log: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/kv0/enc=true/nodes=3/run_1/ssh/ssh_071436.971043330_n4_cockroach-workload-r.log): TRANSIENT_ERROR(ssh_problem): exit status 255
(monitor.go:149).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/kv0/enc=true/nodes=3/run_1

Parameters:

  • arch=amd64
  • cloud=aws
  • coverageBuild=false
  • cpu=8
  • encrypted=true
  • runtimeAssertionsBuild=false
  • ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Grafana is not yet available for aws clusters

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.ssh_problem failed with artifacts on release-24.3 @ 7d3c3da3a2daa0889bee14b1b815a2f5e86b44a7:

test ycsb/D/nodes=3 failed: (cluster.go:2473).Run: full command output in run_081500.559086736_n4_cockroach-workload-r.log: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/ycsb/D/nodes=3/run_1/ssh/ssh_081500.581061932_n4_cockroach-workload-r.log): TRANSIENT_ERROR(ssh_problem): exit status 255
(monitor.go:149).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/ycsb/D/nodes=3/run_1

Parameters:

  • arch=amd64
  • cloud=aws
  • coverageBuild=false
  • cpu=8
  • encrypted=false
  • fs=ext4
  • localSSD=true
  • runtimeAssertionsBuild=false
  • ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Grafana is not yet available for aws clusters

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.ssh_problem failed with artifacts on release-24.3 @ 9ca00e38ee8d06d0d0b5fb66eae23dd18d6db3de:

test cdc/tpcc-1000/sink=kafka failed: (cluster.go:2474).Run: full command output in run_064507.487454936_n4_cockroach-workload-r.log: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/cdc/tpcc-1000/sink=kafka/run_1/ssh/ssh_064507.606060908_n4_cockroach-workload-r.log): TRANSIENT_ERROR(ssh_problem): exit status 255
(cdc.go:394).Close: error shutting down prometheus/grafana: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/cdc/tpcc-1000/sink=kafka/run_1/ssh/ssh_071218.754355056_n4_sudo-rm-rf-tmppromet.log): TRANSIENT_ERROR(ssh_problem): exit status 255
test artifacts and logs in: /artifacts/cdc/tpcc-1000/sink=kafka/run_1

Parameters:

  • arch=amd64
  • cloud=aws
  • coverageBuild=false
  • cpu=16
  • encrypted=false
  • fs=ext4
  • localSSD=true
  • runtimeAssertionsBuild=false
  • ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Grafana is not yet available for aws clusters

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

Note: This build has runtime assertions enabled. If the same failure was hit in a run without assertions enabled, there should be a similar failure without this message. If there isn't one, then this failure is likely due to an assertion violation or (assertion) timeout.

roachtest.ssh_problem failed with artifacts on release-24.3 @ b4a3303281391a411cd40f8f208fcae3793e7d86:

test import/tpcc/warehouses=4000/geo failed: (test_runner.go:853).func4: cluster.PutE: put "/go/src/github.com/cockroachdb/cockroach/bin/cockroach-ea.linux-amd64" failed: error persisted after 3 attempts: TRANSIENT_ERROR(ssh_problem): ~ scp -r -C -o StrictHostKeyChecking=no -o ConnectTimeout=10 -i /home/roach/.ssh/id_rsa -i /home/roach/.ssh/google_compute_engine /go/src/github.com/cockroachdb/cockroach/bin/cockroach-ea.linux-amd64 [email protected]:./cockroach
Warning: Permanently added '35.204.247.41' (ECDSA) to the list of known hosts.
Connection to 35.204.247.41 closed by remote host.
lost connection: exit status 1 [owner=test-eng]

Parameters:

  • arch=amd64
  • cloud=gce
  • coverageBuild=false
  • cpu=16
  • encrypted=false
  • fs=ext4
  • localSSD=true
  • runtimeAssertionsBuild=true
  • ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.ssh_problem failed with artifacts on release-24.3 @ 5be83d7c3461add218e04a15e158139ef945ea6c:

test kv0/enc=false/nodes=1 failed: (cluster.go:2474).Run: full command output in run_064624.993847232_n2_cockroach-workload-r.log: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/kv0/enc=false/nodes=1/run_1/ssh/ssh_064625.019319684_n2_cockroach-workload-r.log): TRANSIENT_ERROR(ssh_problem): exit status 255
(monitor.go:149).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/kv0/enc=false/nodes=1/run_1

Parameters:

  • arch=amd64
  • cloud=aws
  • coverageBuild=false
  • cpu=8
  • encrypted=false
  • fs=ext4
  • localSSD=true
  • runtimeAssertionsBuild=false
  • ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Grafana is not yet available for aws clusters

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.ssh_problem failed with artifacts on release-24.3 @ 5be83d7c3461add218e04a15e158139ef945ea6c:

test kv0/enc=false/nodes=1 failed: (cluster.go:2474).Run: full command output in run_064639.084800175_n2_cockroach-workload-r.log: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/kv0/enc=false/nodes=1/cpu_arch=arm64/run_1/ssh/ssh_064639.150709604_n2_cockroach-workload-r.log): TRANSIENT_ERROR(ssh_problem): exit status 255
(monitor.go:149).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/kv0/enc=false/nodes=1/cpu_arch=arm64/run_1

Parameters:

  • arch=arm64
  • cloud=aws
  • coverageBuild=false
  • cpu=8
  • encrypted=false
  • fs=ext4
  • localSSD=true
  • runtimeAssertionsBuild=false
  • ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Grafana is not yet available for aws clusters

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.ssh_problem failed with artifacts on release-24.3 @ 5be83d7c3461add218e04a15e158139ef945ea6c:

test cdc/tpcc-1000/sink=kafka failed: (cluster.go:2474).Run: full command output in run_064553.591138609_n4_cockroach-workload-r.log: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/cdc/tpcc-1000/sink=kafka/cpu_arch=arm64/run_1/ssh/ssh_064553.639349186_n4_cockroach-workload-r.log): TRANSIENT_ERROR(ssh_problem): exit status 255
(cdc.go:394).Close: error shutting down prometheus/grafana: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/cdc/tpcc-1000/sink=kafka/cpu_arch=arm64/run_1/ssh/ssh_071050.418318667_n4_sudo-rm-rf-tmppromet.log): TRANSIENT_ERROR(ssh_problem): exit status 255
test artifacts and logs in: /artifacts/cdc/tpcc-1000/sink=kafka/cpu_arch=arm64/run_1

Parameters:

  • arch=arm64
  • cloud=aws
  • coverageBuild=false
  • cpu=16
  • encrypted=false
  • fs=ext4
  • localSSD=true
  • runtimeAssertionsBuild=false
  • ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Grafana is not yet available for aws clusters

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.ssh_problem failed with artifacts on release-24.3 @ 5be83d7c3461add218e04a15e158139ef945ea6c:

test kv0/enc=false/nodes=3/batch=16 failed: (cluster.go:2474).Run: full command output in run_065441.124222270_n4_cockroach-workload-r.log: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/kv0/enc=false/nodes=3/batch=16/cpu_arch=arm64/run_1/ssh/ssh_065441.160858628_n4_cockroach-workload-r.log): TRANSIENT_ERROR(ssh_problem): exit status 255
(monitor.go:149).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/kv0/enc=false/nodes=3/batch=16/cpu_arch=arm64/run_1

Parameters:

  • arch=arm64
  • cloud=aws
  • coverageBuild=false
  • cpu=8
  • encrypted=false
  • fs=ext4
  • localSSD=true
  • runtimeAssertionsBuild=false
  • ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Grafana is not yet available for aws clusters

Same failure on other branches

This test on roachdash | Improve this report!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
branch-release-24.3 Used to mark GA and release blockers, technical advisories, and bugs for 24.3 O-roachtest O-robot Originated from a bot. T-testeng TestEng Team X-infra-flake the automatically generated issue was closed due to an infrastructure problem not a product issue
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@cockroach-teamcity