run-ct returns with exit code 0 when tests fail - breaking CI detection of failed tests #17752
Comments
|
hello Cypress! This is a critical failure at the most core level of the product: the test runner reports success when tests fail. Is there anyone interested in investigating this? Is there a workaround? |
|
tagging @JessicaSachs because you responded to a component test 5 days ago as well, hope you can route it to the right person! |
|
I have exactly the same problem. My CI reported success for a few days even though we broke the tests. |
|
The component runner API has the same problem: the promise always resolves instead of rejecting. |
|
The component runner API promise always resolves with {
status: 'failed',
failures: 0,
message: 'Could not find Cypress test run results'
}regardless of the success or failure of tests. Here is an example with only passing tests: And another one with failing tests: The problem is the same if an assertion within the test fails. Here is how I use the API: async function main() {
let result = await cypress.run({
testingType: "component",
spec: "cypress/component/**/*_spec.js",
reporter: "junit",
reporterOptions: "mochaFile=tests-component.[hash].xml,toConsole=true",
});
console.log(result);
if (result && result.status === "failed") {
throw new Error(result.message);
}
}Stack:
|
|
@cellog your post says
Should this be |
|
Can you run the DEBUG logs for the cypress:cli during |
|
excellent, thank you, I'll report back shortly, probably a few hours once these meetings are dun |
yes thank you for the catch, I copy/pasted the wrong line. Updated in the original comment |
|
@cellog thanks! @DamienCassou I think this is a separate issue, seems there is a problem with webpack when using I'll look into both of these today. |
jennifer-shehane
added
the
stage: needs information
|
I'm also seeing this using Vite as the dev server. Cypress always returns 0 even with failing tests.
|
|
there was surprisingly little output: then webpack compilation, then no other debug output at all, just this: |
|
I re-ran it with and at the end of the whole thing: |
|
oh - it looks like the tests did pass, sorry. I put a forced failure in and got: You can see that |
|
I wonder if somewhere in the release and test pipeline a tool like https://github.com/bahmutov/cypress-expect could be used to verify the failing tests and the exit code |
|
Thanks @cellog, yah I can see from the debug logs that it's reporting Is there a way you could reduce your repro to a version we can run and see the issue? I'm running ct tests with a failure, and one with a failure with retries but they are exiting with the correct exit code, so there's likely something specific to your project's setup that's causing the issue we'd love to track down. Thanks! |
|
ok, I'll try to get something small setup, probably tomorrow since we're close to end of day |
|
whoa - the plot thickens - I'm seeing test failures suddenly from tests that didn't fail before, like cypress was all "oh crap" and suddenly is returning exit code of 1. |
|
ohhhh I have a theory!! These tests failed applitools snapshots. Could it be that applitools is overriding the return value with its own thing? Could the bug be in their extension? |
|
I can reproduce the problem using the module API. I posted it here. It could be the same problem described here. Edit: I was able to reproduce w/o module API as well, see this run. |
|
Here is a more minimal reproduction that you can run locally: lmiller1990/cypress-react-template#8. Identical tests, Near identical reproduction in the cypress repo does not have this problem: https://app.circleci.com/pipelines/github/cypress-io/cypress/23239/workflows/77b84ba0-27c3-4b7e-87f5-b1254839744b/jobs/857552 Something must be different about how we are running Cypress and how it runs when published on npm. I will continue debugging. |
|
I could not reproduce this problem with 7.6.0 as suggested above, but I can verify there was a regression between 8.2.0 and 8.3.0.
Comparison. v8.2.0...v8.3.0 |
|
Ok, I found the problem. v8.2.0...v8.3.0#diff-f2f892654ff3b0ae74472e3606a75f32817c5a029dbafd3112c1be0e8ab5f90bR27 Updating electron from v12 to v13 seems to have changed something around how the exit code is returned. I checked out I wonder why only Edit 1: I found that for Edit 2: As suspected, at this point Edit 3: This appears to only be a problem with the electron browser. |
cypress-bot
bot
added
stage: work in progress
and removed
stage: needs information
|
This should fix it: #17894. Also includes detailed breakdown. I cannot explain how you encountered this on Cypress 7.6, though. I'll need a reproduction for that one. This fixed the bug introduced in Cypress 8.3 where |
cypress-bot
bot
added
stage: needs review
just to follow up this was PEBKAC - I was looking at our integration tests which run the e2e version of cypress, and my brain said "these are component tests". Applitools extension is not the culprit. Going to work on a simple repro |
|
@lmiller1990 I would like to try to apply this simple patch locally, and see if it fixes the bug, as this will be a lot less effort than making a repro case. Where in god's name does that file live? I did a full text search of node_modules for |
|
@cellog I think what you have in |
|
thanks Damien! For future reference the path in Mac OS is
@lmiller1990 thank you! |
|
Great to hear this is fixed, we will be doing a patch release soon to get this out. If anyone is interested, we actually have pretty neat setup to test Cypress in an e2e manner. Here's the new test verifying that this bug does not regress : https://github.com/cypress-io/cypress/pull/17894/files#diff-f01be2f25ace400c3f80f0cd17786e27db5352291a9555ec4f2e5cc206a15330 which would have caught this issue (and will moving forward). |
|
Released in This comment thread has been locked. If you are still experiencing this issue after upgrading to |
cypress-bot
bot
removed
the
stage: needs review
Current behavior
This is a pretty big bug. All CI jobs running with
run-ctwill report success even if tests fail due to this bug.In this screenshot, you are seeing the output of
yarn cypress:component:cli --spec=packages/SearchInput/src/component_tests/ClearButton.test.tsxwhich runs this command:Note that at the bottom, we see
Done in 69.93sbut no error. ThisDoneis reported by yarn. When a test fails, it should look more like this (I addedexpect(false).toBe(true)to a jest test to simulate a failure):cypress.json:
Desired behavior
when a test fails, the shell error code should be non-zero.
cypress rundoes this correctly.Test code to reproduce
cypress run-ctCypress Version
8.3.0
Other
tested with
7.6.0as well, upgraded to be sure it was not already fixed.The text was updated successfully, but these errors were encountered: