Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(cve): implement CVE scanning as background tasks #1833

Merged
merged 1 commit into from
Sep 22, 2023

Conversation

andaaron
Copy link
Contributor

@andaaron andaaron commented Sep 19, 2023

  1. Move existing CVE DB download generator/task login under the cve package
  2. Add a new CVE scanner task generator and task type to run in the background, as well as tests for it
  3. Move the CVE cache in its own package
  4. Add a CVE scanner methods to check if an entry is present in the cache, and to retreive the results
  5. Modify the FilterTags MetaDB method to not exit on first error
    This is needed in order to pass all tags to the generator,
    instead of the generator stopping at the first set of invalid data
  6. Integrate the new scanning task generator with the existing zot code.
  7. Fix an issue where the CVE scan results for multiarch images was not cached
  8. Rewrite some of the older CVE tests to use the new image-utils test package
  9. Use the CVE scanner as attribute of the controller instead of CveInfo.
    Remove functionality of CVE DB update from CveInfo, it is now responsible,
    as the name states, only for providing CVE information.
  10. The logic to get maximum severity and cve count for image sumaries now uses only the scanner cache.
  11. Removed the GetCVESummaryForImage method from CveInfo as it was only used in tests

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

@andaaron
Copy link
Contributor Author

In today's implementation the scanner populates the cache, but the API calls are still blocking.
An improvement would be to make the API calls non-blocking and to have them push new tasks for scanning images not present in cache.

@codecov
Copy link

codecov bot commented Sep 19, 2023

Codecov Report

Merging #1833 (3f029a8) into main (8c55944) will increase coverage by 0.06%.
Report is 1 commits behind head on main.
The diff coverage is 92.60%.

❗ Current head 3f029a8 differs from pull request most recent head e019a5b. Consider uploading reports for the commit e019a5b to get more accurate results

@@            Coverage Diff             @@
##             main    #1833      +/-   ##
==========================================
+ Coverage   91.72%   91.79%   +0.06%     
==========================================
  Files         143      145       +2     
  Lines       26127    26275     +148     
==========================================
+ Hits        23965    24119     +154     
+ Misses       1597     1594       -3     
+ Partials      565      562       -3     
Files Changed Coverage Δ
pkg/meta/boltdb/boltdb.go 89.14% <56.00%> (+<0.01%) ⬆️
pkg/meta/dynamodb/dynamodb.go 92.92% <66.66%> (-0.28%) ⬇️
pkg/extensions/search/cve/cve.go 97.76% <90.90%> (-0.44%) ⬇️
pkg/api/config/config.go 97.35% <100.00%> (+0.03%) ⬆️
pkg/api/controller.go 94.20% <100.00%> (ø)
pkg/api/routes.go 94.26% <100.00%> (ø)
pkg/extensions/extension_search.go 100.00% <100.00%> (ø)
pkg/extensions/extension_search_disabled.go 100.00% <100.00%> (ø)
pkg/extensions/search/cve/cache/cache.go 100.00% <100.00%> (ø)
pkg/extensions/search/cve/scan.go 100.00% <100.00%> (ø)
... and 2 more

... and 3 files with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

@andaaron
Copy link
Contributor Author

Related to #1815

@andaaron andaaron marked this pull request as ready for review September 20, 2023 16:44
@andaaron andaaron self-assigned this Sep 20, 2023
@andaaron andaaron force-pushed the cve-bg-2 branch 4 times, most recently from a4d9c01 to 3f029a8 Compare September 21, 2023 22:14
1. Move existing CVE DB download generator/task login under the cve package
2. Add a new CVE scanner task generator and task type to run in the background, as well as tests for it
3. Move the CVE cache in its own package
4. Add a CVE scanner methods to check if an entry is present in the cache, and to retreive the results
5. Modify the FilterTags MetaDB method to not exit on first error
This is needed in order to pass all tags to the generator,
instead of the generator stopping at the first set of invalid data
6. Integrate the new scanning task generator with the existing zot code.
7. Fix an issue where the CVE scan results for multiarch images was not cached
8. Rewrite some of the older CVE tests to use the new image-utils test package
9. Use the CVE scanner as attribute of the controller instead of CveInfo.
Remove functionality of CVE DB update from CveInfo, it is now responsible,
as the name states, only for providing CVE information.
10. The logic to get maximum severity and cve count for image sumaries now uses only the scanner cache.
11. Removed the GetCVESummaryForImage method from CveInfo as it was only used in tests

Signed-off-by: Andrei Aaron <[email protected]>
Copy link
Contributor

@rchincha rchincha left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@rchincha rchincha merged commit 7c78f80 into project-zot:main Sep 22, 2023
29 checks passed
@andaaron andaaron linked an issue Sep 23, 2023 that may be closed by this pull request
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Feat] CVE scanner: use background task for scanning images
2 participants