feat(config): failOnFlakyTests option

[jonhermansen]

This patch adds a configuration option to make the test runner exit with a non-zero status when any test is flaky. The config option does not take precedence over the CLI flag.

This is my first time submitting a change to playwright, so any and all guidance is welcome. If approved, I can also update the example configurations in the create-playwright repo too. If any documentation needs to be updated, please point me to what I've missed. Thanks!

Fixes #34397

[jonhermansen]

@microsoft-github-policy-service agree

@microsoft-github-policy-service agree

[Skn0tt]

Looks great, thank you! There's one little thing i'd like changed, happy to merge after that.

[github-actions]

Test results for "tests 1"

5 failed
❌ [playwright-test] › tests/exit-code.spec.ts:131:5 › failOnFlakyTests flag disallows flaky @macos-latest-node18-1
❌ [playwright-test] › tests/exit-code.spec.ts:131:5 › failOnFlakyTests flag disallows flaky @ubuntu-latest-node18-1
❌ [playwright-test] › tests/exit-code.spec.ts:131:5 › failOnFlakyTests flag disallows flaky @ubuntu-latest-node20-1
❌ [playwright-test] › tests/exit-code.spec.ts:131:5 › failOnFlakyTests flag disallows flaky @ubuntu-latest-node22-1
❌ [playwright-test] › tests/exit-code.spec.ts:131:5 › failOnFlakyTests flag disallows flaky @windows-latest-node18-1

17 flaky

⚠️ [chromium-library] › tests/library/role-utils.spec.ts:442:5 › svg role=presentation @chromium-ubuntu-22.04-node18
⚠️ [firefox-page] › tests/page/page-evaluate.spec.ts:403:3 › should throw for too deep reference chain @firefox-ubuntu-22.04-node18
⚠️ [playwright-test] › tests/ui-mode-trace.spec.ts:341:5 › should work behind reverse proxy @macos-latest-node18-1
⚠️ [webkit-library] › tests/library/browsercontext-pages.spec.ts:82:3 › should click the button with offset with page scale @webkit-ubuntu-22.04-node18
⚠️ [webkit-library] › tests/library/browsercontext-pages.spec.ts:105:3 › should return bounding box with page scale @webkit-ubuntu-22.04-node18
⚠️ [webkit-library] › tests/library/inspector/cli-codegen-1.spec.ts:199:7 › cli codegen › should work with TrustedTypes @webkit-ubuntu-22.04-node18
⚠️ [webkit-library] › tests/library/page-event-crash.spec.ts:77:1 › should be able to close context when page crashes @webkit-ubuntu-22.04-node18
⚠️ [webkit-library] › tests/library/proxy.spec.ts:93:11 › should proxy local network requests › with other bypasses › loopback address @webkit-ubuntu-22.04-node18
⚠️ [webkit-library] › tests/library/selector-generator.spec.ts:286:5 › selector generator › should properly join child selectors under nested ordinals @webkit-ubuntu-22.04-node18
⚠️ [webkit-library] › tests/library/trace-viewer.spec.ts:149:1 › should open simple trace viewer @webkit-ubuntu-22.04-node18
⚠️ [webkit-page] › tests/page/page-event-popup.spec.ts:28:3 › should work with window features @webkit-ubuntu-22.04-node18
⚠️ [webkit-page] › tests/page/page-filechooser.spec.ts:24:5 › should upload multiple large files @webkit-ubuntu-22.04-node18
⚠️ [webkit-page] › tests/page/page-leaks.spec.ts:82:5 › click should not leak @webkit-ubuntu-22.04-node18
⚠️ [webkit-page] › tests/page/page-leaks.spec.ts:107:5 › fill should not leak @webkit-ubuntu-22.04-node18
⚠️ [webkit-page] › tests/page/page-screenshot.spec.ts:30:5 › page screenshot › should work @smoke @webkit-ubuntu-22.04-node18
⚠️ [webkit-page] › tests/page/page-screenshot.spec.ts:364:5 › page screenshot › path option should work @webkit-ubuntu-22.04-node18
⚠️ [webkit-page] › tests/page/page-set-input-files.spec.ts:204:5 › should upload large file with relative path @webkit-ubuntu-22.04-node18

38556 passed, 794 skipped
✔️✔️✔️

Merge workflow run.

[Skn0tt]

Looks like there's some failing tests - there's a piece of code writing on the removed parameter.