[CSR-3] Report results for spec files when cypress has a hard crash

[bilarallen]

Before opening, please confirm:

• I have searched for duplicate or closed issues and discussions.
• I have done my best to include a minimal, self-contained set of instructions for consistently reproducing the issue.
• I acknowledge that I will attach a full debug log, otherwise the issue will be closed with no response.

Environment information

# Put output below this line
System:
    OS: Linux 5.4 Debian GNU/Linux 10 (buster) 10 (buster)
    CPU: (10) x64 AMD EPYC 7282 16-Core Processor
    Memory: 11.78 GB / 58.88 GB
    Container: Yes
    Shell: 5.0.3 - /bin/bash
  Binaries:
    Node: 14.16.0 - /usr/local/bin/node
    Yarn: 1.22.10 - /usr/local/bin/yarn
    npm: 6.14.11 - /usr/local/bin/npm
  npmPackages:
    @currents/cli: ^4.0.3 => 4.0.3 
    @cypress/angular:  0.0.0-development 
    @cypress/mount-utils:  0.0.0-development 
    @cypress/react:  0.0.0-development 
    @cypress/react18:  0.0.0-development 
    @cypress/svelte:  0.0.0-development 
    @cypress/vue:  0.0.0-development 
    @cypress/vue2:  0.0.0-development 
    @cypress/webpack-batteries-included-preprocessor: ^2.4.0 => 2.4.0 
    @cypress/webpack-preprocessor: ^5.17.0 => 5.17.0 
    @faker-js/faker: ^7.6.0 => 7.6.0 
    cypress: ^12.7.0 => 12.7.0 
    cypress-cloud: ^1.4.4 => 1.4.4 
    cypress-recurse: ^1.27.0 => 1.27.0 
    moment: ^2.29.4 => 2.29.4 
    numeral: ^2.0.6 => 2.0.6 
    swagger-client: ^3.18.5 => 3.18.5 
    uuid: ^9.0.0 => 9.0.0 (8.3.2)
    xvfb: ^0.4.0 => 0.4.0 
  npmGlobalPackages:
    npm: 6.14.11
    yarn: 1.22.10

Describe the bug

When the Jenkins Job is already finished and is a success, the run associated with that build in the Currents keeps on running. This only happens whenever we encounter the warning message WARNING Cannot find run result for spec ../spec/Path and WARNING Cannot determine run result for spec ../spec/Path every after Reporting results and artifacts in background... in the console log

Expected behavior

Should be able to finish the run and perhaps give more information as to why it cannot find and determine the result for a particular spec that was run.

Command and Setup

Jenkins File Snippet

 stage('CBS UI Test') { 
            environment {
                CURRENTS_RECORD_KEY = credentials('currents-record-key')
                CURRENTS_PROJECT_ID = credentials('currents-master-projectid')
                CURRENTS_API_URL = credentials('currents-cloud-url')
                CYPRESS_trashAssetsBeforeRuns = 'false'
            }

            parallel {
                stage('Tester A') {
                    steps {
                        echo "Running build ${env.BUILD_ID} on Tester A"
                        sh "NO_COLOR=1 npx cypress-cloud run --parallel --record --key ${env.CURRENTS_RECORD_KEY} --ci-build-id ${env.GIT_BRANCH}-${env.BUILD_ID}"
                    }
                }

                stage('Tester B') {
                    steps {
                        echo "Running build ${env.BUILD_ID} on Tester B"
                        sh "NO_COLOR=1 npx cypress-cloud run --parallel --record --key ${env.CURRENTS_RECORD_KEY} --ci-build-id ${env.GIT_BRANCH}-${env.BUILD_ID}"
                    }
                }

                stage('Tester C') {
                    steps {
                        echo "Running build ${env.BUILD_ID} on Tester C"
                        sh "NO_COLOR=1 npx cypress-cloud run --parallel --record --key ${env.CURRENTS_RECORD_KEY} --ci-build-id ${env.GIT_BRANCH}-${env.BUILD_ID}"
                    }
                }

            }
           
        }

Full log and debug output

This is an intermittent issue and produces a lot of logs if we do it in Jenkins CI and doesn't happen all the time but here's a screen recording of it

2023-03-14_12h31_02.mp4

Edit by @agoldis

Also, make sure to mark the run as failed in CI when a crash is detected

[bilarallen]

Randomly encountered it again but this time before the warning message it displays:

[1910:0314/060518.104073:ERROR:connection.cc(46)] X connection error received.
Gdk-Message: 06:05:18.108: Cypress: Fatal IO error 11 (Resource temporarily unavailable) on X server :99.

[agoldis]

Thanks for reporting that @bilarallen. It seems like cypress had a hard crash without returning anything at all.

[bilarallen]

Hi @agoldis I think I found the possible root cause of the issue but I'm not that really sure, it seems that based on my comment #79 (comment) its a problem with the xvfb since I'm running the test in parallel all threads are using the same port so it resulted in not finding the test result of the recent run. Here's additional links that can connect the dot for this problem:

tnicola/cypress-parallel#89 (comment)
cypress-io/xvfb#98
https://docs.cypress.io/guides/continuous-integration/introduction#Xvfb

I get some point in the comment to run a particular command for xvfb but I don't know how to do it in my jenkins pipeline since it is using docker cypress/base image

[agoldis]

@bilarallen are you running cypress in parallel on the same machine? It'd be still great to have the crash reported to the dashboard.

[bilarallen]

Hi @agoldis yes I am running the parallel test on the same machine.

[MaltePetersen]

Hello, we have the same issue. We are running our test in parallel on the same machine as well. @agoldis Do I understand you correctly the problem is that the Cypress cloud processes are running on the same machine. So running each cypress cloud process in a different docker container might solve the issue, right?

[agoldis]

@MaltePetersen
Running on the same machine increases the chances of a crash because Cypress is quite resources-heavy.
Using isolated environments can prevent some crashes or side effects.

This issue deals is aboutcypress-cloud not reporting the results for crashed cypress runs and causing the whole run to time out

[bilarallen]

Hi @agoldis, would just like to ask what's the update for this one, please? As we have experienced it again. Here is the debug logs. Thanks!
Cypress_Cloud_Issue_79_New_Logs.txt

[agoldis]

Hello @bilarallen I am very sorry that this issue is taking so much to resolve, we are working on a fix, expecting to roll out a release next week.

[bilarallen]

Hi @agoldis cool! Looking forward to it 😄 thanks!

[agoldis]

@bilarallen @MaltePetersen
Very sorry it took so long to resolve. Please try using [email protected] - along with a bit faster reporting, it handles the failures in an appropriate way.

[MaltePetersen]

@agoldis Thanks for the update. Through the appropriate error handling, I recognized that my concurrent approach on the same machine did not work because xvfb was sometimes locked from the other cypress-cloud instance accessing it on initialization. Just putting a sleep 5 && before my second cypress-cloud initialization solved the issue.

[agoldis]

That's great! Will release to the latest npm channel