feat(gnovm): Test Coverage Support

[notJoon]

Description

closes #1121

Support testing coverage functionality into previous testing framework. This feature tracks and reports which parts of the code were executed during test runs.

Key Changes

1. Added coverage data structures
   • Implemented CoverageData and FileCoverage structs
   • Per-file executed line tracker
2. Coverage collection machanism
   • Collect current location information after PopOp in Machine.Run() method
     • Inserted recordCoverage function for each statement to obtaining accurate coverage tracking result
   • Record executed lines throught AddHit method
3. Coverage reporting
   • Calculate and output coverage percentage per file
     • Excludes non-executable lines (e.g., comments) from the calculation
   • Terminal visualization using ANSI color codes
4. Connect with external tools
   • Convert coverage data to JSON format

CLI Options

CLI Options

Flag	Desc	Example
-cover	Activate coverage analysis	gno test <path> -cover
-view=<pattern>	View coverage for files matching the pattern	gno test <path> -cover -view=demo/foo
-show-hits	Show number of times each line was executed	gno test <path> -cover -view=demo/foo -show-hits
-html=<file>	Generate HTML coverage report	gno test <path> -cover -html=coverage.html

All coverage-related flags must be preceded by the -cover flag to be active.

Notes:

• The report displays a list of files with their coverage percentages, color-coded based on coverage level.
• The -view option supports partial file path patterns and can match multiple files.
• When using -view, the output is color-coded: green for executed lines, yellow for executable but not executed lines, and white for non-executable lines.
• The -show-hits option adds an orange hit count next to each executed line when used with -view.

The currently added CLI was added for testing during development. I plan to modify it later to provide the same functionality as go tool.

[codecov]

Codecov Report

Attention: Patch coverage is 74.42681% with 145 lines in your changes missing coverage. Please review.

Project coverage is 63.59%. Comparing base (534e652) to head (56cc61f).

Files with missing lines	Patch %	Lines
gnovm/pkg/gnolang/coverage.go	81.33%	70 Missing and 11 partials ⚠️
gnovm/cmd/gno/test.go	53.57%	34 Missing and 5 partials ⚠️
gnovm/pkg/gnolang/machine.go	43.18%	22 Missing and 3 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #2616      +/-   ##
==========================================
+ Coverage   63.38%   63.59%   +0.21%     
==========================================
  Files         566      567       +1     
  Lines       79490    80332     +842     
==========================================
+ Hits        50381    51088     +707     
- Misses      25717    25833     +116     
- Partials     3392     3411      +19     

Flag	Coverage Δ
contribs/gnodev	60.57% <ø> (ø)
contribs/gnofaucet	14.82% <ø> (ø)
gno.land	67.37% <ø> (ø)
gnovm	67.98% <74.42%> (+0.11%)	⬆️
misc/genstd	79.72% <ø> (ø)
tm2	62.41% <ø> (-0.02%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[notJoon]

For simplicity for reviewing I'll change this PR as ready for review.

[notJoon]

@zivkovicmilos It seems the problem is occurring when retrieving and updating the line information executed in the stdlib part. If I fix this, I'll ping you.

[thehowl]

Hey @notJoon, thanks for covering this. If I get a chance, I'll try to get a review in.

I just wanted to comment because I just added a bounty on the original issue: #1121

This has been a long time coming; we had set out the requirements for a small while before you published this PR; but we have been slow in getting the bounties published 🫠

The requirements are slightly different to what you implemented here in general, and reflect the general comment I'd make on the PR right now: I like the terminal visualization, but ultimately it doesn't matter. Terminal visualization should be the responsibility of other tools based on the eventual coverprofile. If we use coverprofiles properly, it means we can re-use the tools Go already has; and that's more important.

So, I'd recommend you move forward more in the direction of ensuring compatibility in that direction and as specified in the requirements. Note that the bounties exclude AiB/NT employees from working on the bounty, but not grantees; so at your discretion, you can also claim the bounty.

Thanks!

[notJoon]

@thehowl Looks like the bounty program is gonna start 😮

The CLI output was temporarily added to view during development, so it doesn't matter actually. Ultimately, the goal is to make it as similar as possible to Go's test coverage functionality.

Originally, before the bounty was added, I was planning to split the work to reduce the PR size, but in this case, I wonder if all the work needs to be completed in a single PR?

Or should I submit a separate work plan? There seems to be more information in the attached link, but I don't have permission to access it. Thank you!

[thehowl]

separate please! smaller PRs are better in general :)

once the issue is fully addressed, we'll propose how we intend to distribute the bounty (mostly in case you're not the only once who contributed to it) :)

[thehowl]

apologies, misclicked.

[notJoon]

separate please! smaller PRs are better in general :)

Good. I'll get this done as quickly as possible 👍

[notJoon]

@zivkovicmilos @thehowl

Now all CI passes successfully. Previously, I had set it up to always generate and store coverage data whenever tests were run. But this caused timeouts in the strings and bytes packages due to increase overhead (#2935). So I modified it to collect data only when the coverage flag (-flag) is activated.

Currently, since it only perfoems basic operations, there are cases where lines that should be executed are not recognized during the execution. However, this case requires branch coverage and other static analysis features to be implemented. So, I plan to make these in a separate PR to prevent the current PR size from becoming too large.

The planned tasks are as follows:

1. Run static analysis based on the collected line information to calculate more accurate coverage
2. Provide the same commands and functionalities as go tool cover (ref: gno test -cover ... #1121)

[jefft0]

Hello @notJoon . It says "This branch has conflicts that must be resolved". Are there merge conflicts with master?

[notJoon]

Hello @notJoon . It says "This branch has conflicts that must be resolved". Are there merge conflicts with master?

@jefft0 now fixed

[jefft0]

The changes look reasonable to me, but we need a core dev to decide if they are needed. Do you need anything else from me for the "triage" review?

[notJoon]

The changes look reasonable to me, but we need a core dev to decide if they are needed. Do you need anything else from me for the "triage" review?

@jefft0

I think checking just the basic things should be sufficient. I've added flags as per convention.

[jefft0]

@notJoon , can you resolve the conflicts with the master branch?

[notJoon]

@jefft0 resolved. thanks 👍

[jefft0]

@notJoon, the failing CI checks are because of a misconfiguration of codecov. This has been fixed. Can you please push an empty commit? This will make the CI checks run again and hopefully pass.