Find and implement a replacement for codecov

[jristau1984]

Codecov is rate limiting all CI in github (and thus causing commits to fail), so it is being temporarily disabled in a lot of places. We cannot be in this state for very long, s owe need a replacement solution put in place asap.

[huniafatima-arbi]

Hello @jristau1984, I have mentioned the detail of the possible replacement for the codecov. Please give it a review and then we can discuss which one to use further.

Third Party Apps

Codecov (open source)

Pros

• Smooth integration in CI/CD workflows

• Comprehensive coverage reports.

• Shows coverage of files individually

• Gives insights on covered and uncovered lines by test cases

Cons

• Free for open source repos only

• Pricing plans for private repos

Coveralls

Pros

• Smooth integration in CI/CD workflows

• Comprehensive coverage reports.

• Shows coverage of files individually

• Gives insights on covered and uncovered lines by test cases

Cons

• Free for open source repos only

• Pricing plans for private repos

Github Actions

Jest coverage report

Pros

• Easy integration by just adding the action in your workflow

• Free for all repos

Cons

• Does not provide coverage stats up to files and lines of code

Coverage Sample

Jest Coverage Comment

Pros

• Easy integration by just adding the action in your workflow

• Free for all repos

Cons

• Does not provide coverage stats up to files and lines of code

Coverage Sample

Note: To use github actions we shall be using different for both python repos and javascript ones.

Python Coverage Comment

Pros

• Easy integration by just adding the action in your workflow

• Free for all repos

Cons

• Does not provide coverage stats up to files and lines of code

Coverage Sample

[jristau1984]

@UsamaSadiq @huniafatima-arbi - is there anything left for this ticket? I think just documenting the final findings and decision should be enough to call this complete. It may require follow up tickets to configure codecov the way we want it internally. Does that sound correct?

[huniafatima-arbi]

@UsamaSadiq @huniafatima-arbi - is there anything left for this ticket? I think just documenting the final findings and decision should be enough to call this complete. It may require follow up tickets to configure codecov the way we want it internally. Does that sound correct?

@jristau1984 Yup, I shall get on the documentation and will share here once done. Afterwards we can go for the follow up tickets for codecov configuration.

[huniafatima-arbi]

This sheet contains the list of private repos which are currently using codecov in edX org