test(cli): add retry for cli tests

[bjoerge]

Description

CLI tests are sometimes flaky. This adds retrying to cli tests similar to what we did for e2e tests in #8501, and unit tests in #8371

What to review

Makes sense?

Notes for release

n/a – internal

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
page-building-studio	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Feb 10, 2025 3:57pm
performance-studio	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Feb 10, 2025 3:57pm
test-studio	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Feb 10, 2025 3:57pm

2 Skipped Deployments

Name	Status	Preview	Comments	Updated (UTC)
studio-workshop	⬜️ Ignored (Inspect)			Feb 10, 2025 3:57pm
test-next-studio	⬜️ Ignored (Inspect)			Feb 10, 2025 3:57pm

[github-actions]

No changes to documentation

[github-actions]

Coverage Report

Status	Category	Percentage	Covered / Total
🔵	Lines	42.43%	53726 / 126621
🔵	Statements	42.43%	53726 / 126621
🔵	Functions	47.91%	2767 / 5775
🔵	Branches	78.76%	10278 / 13049

File Coverage

No changed files found.

Generated in workflow #30088 for commit 185d576 by the Vitest Coverage Report Action

[github-actions]

⚡️ Editor Performance Report

Updated Mon, 10 Feb 2025 16:07:28 GMT

Benchmark	reference latency of sanity@latest	experiment latency of this branch	Δ (%) latency difference
article (title)	25.0 efps (40ms)	26.0 efps (39ms)	-2ms (-3.8%)	✅
article (body)	77.5 efps (13ms)	72.5 efps (14ms)	+1ms (-/-%)	✅
article (string inside object)	27.0 efps (37ms)	27.8 efps (36ms)	-1ms (-2.7%)	✅
article (string inside array)	23.5 efps (43ms)	24.4 efps (41ms)	-2ms (-3.5%)	✅
recipe (name)	43.5 efps (23ms)	46.5 efps (22ms)	-2ms (-6.5%)	✅
recipe (description)	47.6 efps (21ms)	52.6 efps (19ms)	-2ms (-9.5%)	✅
recipe (instructions)	99.9+ efps (5ms)	99.9+ efps (5ms)	+0ms (-/-%)	✅
synthetic (title)	20.8 efps (48ms)	21.5 efps (47ms)	-2ms (-3.1%)	✅
synthetic (string inside object)	20.8 efps (48ms)	21.3 efps (47ms)	-1ms (-2.1%)	✅

efps — editor "frames per second". The number of updates assumed to be possible within a second.

Derived from input latency. efps = 1000 / input_latency

Detailed information

🏠 Reference result

The performance result of sanity@latest

Benchmark	latency	p75	p90	p99	blocking time	test duration
article (title)	40ms	44ms	48ms	283ms	160ms	10.8s
article (body)	13ms	15ms	28ms	132ms	177ms	5.0s
article (string inside object)	37ms	40ms	43ms	52ms	129ms	7.0s
article (string inside array)	43ms	45ms	49ms	96ms	384ms	7.5s
recipe (name)	23ms	25ms	30ms	44ms	0ms	7.7s
recipe (description)	21ms	23ms	26ms	46ms	0ms	5.1s
recipe (instructions)	5ms	7ms	8ms	9ms	0ms	3.0s
synthetic (title)	48ms	51ms	58ms	306ms	543ms	12.6s
synthetic (string inside object)	48ms	50ms	79ms	393ms	938ms	8.3s

🧪 Experiment result

The performance result of this branch

Benchmark	latency	p75	p90	p99	blocking time	test duration
article (title)	39ms	60ms	77ms	521ms	932ms	10.9s
article (body)	14ms	16ms	26ms	168ms	205ms	5.3s
article (string inside object)	36ms	39ms	43ms	77ms	207ms	6.5s
article (string inside array)	41ms	43ms	48ms	241ms	256ms	7.0s
recipe (name)	22ms	24ms	27ms	38ms	0ms	8.2s
recipe (description)	19ms	21ms	23ms	49ms	0ms	4.9s
recipe (instructions)	5ms	7ms	8ms	19ms	0ms	3.1s
synthetic (title)	47ms	52ms	64ms	366ms	680ms	12.0s
synthetic (string inside object)	47ms	51ms	67ms	439ms	861ms	8.1s

📚 Glossary

column definitions

• benchmark — the name of the test, e.g. "article", followed by the label of the field being measured, e.g. "(title)".
• latency — the time between when a key was pressed and when it was rendered. derived from a set of samples. the median (p50) is shown to show the most common latency.
• p75 — the 75th percentile of the input latency in the test run. 75% of the sampled inputs in this benchmark were processed faster than this value. this provides insight into the upper range of typical performance.
• p90 — the 90th percentile of the input latency in the test run. 90% of the sampled inputs were faster than this. this metric helps identify slower interactions that occurred less frequently during the benchmark.
• p99 — the 99th percentile of the input latency in the test run. only 1% of sampled inputs were slower than this. this represents the worst-case scenarios encountered during the benchmark, useful for identifying potential performance outliers.
• blocking time — the total time during which the main thread was blocked, preventing user input and UI updates. this metric helps identify performance bottlenecks that may cause the interface to feel unresponsive.
• test duration — how long the test run took to complete.

[github-actions]

Component Testing Report Updated Feb 10, 2025 4:11 PM (UTC)

❌ Failed Tests (3) -- expand for details

File	Status	Duration	Passed	Skipped	Failed
comments/CommentInput.spec.tsx	✅ Passed (Inspect)	1m 12s	15	0	0
formBuilder/ArrayInput.spec.tsx	✅ Passed (Inspect)	13s	3	0	0
formBuilder/inputs/PortableText/Annotations.spec.tsx	❌ Failed (Inspect)	2m 36s	3	0	3
formBuilder/inputs/PortableText/copyPaste/CopyPaste.spec.tsx	✅ Passed (Inspect)	56s	11	7	0
formBuilder/inputs/PortableText/copyPaste/CopyPasteFields.spec.tsx	✅ Passed (Inspect)	0s	0	12	0
formBuilder/inputs/PortableText/Decorators.spec.tsx	✅ Passed (Inspect)	26s	6	0	0
formBuilder/inputs/PortableText/DisableFocusAndUnset.spec.tsx	✅ Passed (Inspect)	15s	3	0	0
formBuilder/inputs/PortableText/DragAndDrop.spec.tsx	✅ Passed (Inspect)	27s	6	0	0
formBuilder/inputs/PortableText/FocusTracking.spec.tsx	✅ Passed (Inspect)	1m 9s	15	0	0
formBuilder/inputs/PortableText/Input.spec.tsx	✅ Passed (Inspect)	1m 37s	21	0	0
formBuilder/inputs/PortableText/ObjectBlock.spec.tsx	✅ Passed (Inspect)	2m 5s	21	0	0
formBuilder/inputs/PortableText/PresenceCursors.spec.tsx	✅ Passed (Inspect)	13s	3	9	0
formBuilder/inputs/PortableText/Styles.spec.tsx	✅ Passed (Inspect)	26s	6	0	0
formBuilder/inputs/PortableText/Toolbar.spec.tsx	✅ Passed (Inspect)	1m 44s	21	0	0
formBuilder/tree-editing/TreeEditing.spec.tsx	✅ Passed (Inspect)	0s	0	3	0
formBuilder/tree-editing/TreeEditingNestedObjects.spec.tsx	✅ Passed (Inspect)	0s	0	3	0

[jordanl17]

Neat - prefer to add the retry on all tests over adding to select flakey tests? Fine with this though

[bjoerge]

prefer to add the retry on all tests over adding to select flakey tests?

Good point. The drawback with retrying all tests is that legit test failures will be retried 4 times for no good reason. I don't think that will be a problem in practice though, and also not sure if it's that easy to identify which exact ones are flaky either?

This is a super low-effort quickfix, so I'll merge for now, but lets reconsider if we find issues with it.