intermittent failures in k8s scanning

[itaysk]

Description

SOMETIMES a k8s scan will fail, with an error like:

2024-10-07T14:26:31+03:00       FATAL   Fatal error
  - k8s scan error:
    github.com/aquasecurity/trivy/pkg/k8s/commands.(*runner).run
        github.com/aquasecurity/trivy/pkg/k8s/commands/run.go:91
  - scanning misconfigurations error:
    github.com/aquasecurity/trivy/pkg/k8s/scanner.(*Scanner).Scan.func1
        github.com/aquasecurity/trivy/pkg/k8s/scanner/scanner.go:116
  - scan error:
    github.com/aquasecurity/trivy/pkg/commands/artifact.(*runner).scanArtifact
        github.com/aquasecurity/trivy/pkg/commands/artifact/run.go:260
  - scan failed:
    github.com/aquasecurity/trivy/pkg/commands/artifact.(*runner).scan
        github.com/aquasecurity/trivy/pkg/commands/artifact/run.go:615
  - failed analysis:
    github.com/aquasecurity/trivy/pkg/scanner.Scanner.ScanArtifact
        github.com/aquasecurity/trivy/pkg/scanner/scan.go:158
  - walk filesystem:
    github.com/aquasecurity/trivy/pkg/fanal/artifact/local.Artifact.Inspect
        github.com/aquasecurity/trivy/pkg/fanal/artifact/local/fs.go:113
  - walk dir error:
    github.com/aquasecurity/trivy/pkg/fanal/walker.(*FS).Walk
        github.com/aquasecurity/trivy/pkg/fanal/walker/fs.go:34
  - unknown error with /var/folders/kz/5_0qf4vn66v6521s9f4nm3rm0000gn/T/appy-Service-appy-frontend-4034367594.yaml:
    github.com/aquasecurity/trivy/pkg/fanal/walker.(*FS).Walk.(*FS).onError.func2
        github.com/aquasecurity/trivy/pkg/fanal/walker/fs.go:91

every time it occured, I rerun the same command and it worked. maybe a race condition?

Desired Behavior

shouldn't fail, or error out

Actual Behavior

see error

Reproduction Steps

I cannot consistently reproduce it, it happens sometimes, every time it occured I rerun the command and it worked.

Target

Kubernetes

Scanner

None

Output Format

None

Mode

None

Debug Output

$ trivy k8s --report all --include-namespaces appy --severity HIGH --ignorefile "$demodir/backend/trivyignore.yaml" --debug
2024-10-07T14:26:26+03:00       DEBUG   No plugins loaded
2024-10-07T14:26:26+03:00       DEBUG   Default config file "file_path=trivy.yaml" not found, using built in values
2024-10-07T14:26:26+03:00       DEBUG   Cache dir       dir="/Users/itaysk/Library/Caches/trivy"
2024-10-07T14:26:26+03:00       DEBUG   Cache dir       dir="/Users/itaysk/Library/Caches/trivy"
2024-10-07T14:26:26+03:00       DEBUG   Parsed severities       severities=[HIGH]
2024-10-07T14:26:26+03:00       DEBUG   Ignore statuses statuses=[]
2024-10-07T14:26:28+03:00       INFO    Node scanning is enabled
2024-10-07T14:26:28+03:00       INFO    If you want to disable Node scanning via an in-cluster Job, please try '--disable-node-collector' to disable the Node-Collector job.
2024-10-07T14:26:28+03:00       DEBUG   DB update was skipped because the local DB is the latest
2024-10-07T14:26:28+03:00       DEBUG   DB info schema=2 updated_at=2024-10-07T06:17:04.462736974Z next_update=2024-10-08T06:17:04.462736814Z downloaded_at=2024-10-07T09:25:56.846879Z
6 / 6 [-------------------------------------------------------------------------------------------------------------------------------------] 100.00% 2 p/s
2024-10-07T14:26:31+03:00       FATAL   Fatal error
  - k8s scan error:
    github.com/aquasecurity/trivy/pkg/k8s/commands.(*runner).run
        github.com/aquasecurity/trivy/pkg/k8s/commands/run.go:91
  - scanning misconfigurations error:
    github.com/aquasecurity/trivy/pkg/k8s/scanner.(*Scanner).Scan.func1
        github.com/aquasecurity/trivy/pkg/k8s/scanner/scanner.go:116
  - scan error:
    github.com/aquasecurity/trivy/pkg/commands/artifact.(*runner).scanArtifact
        github.com/aquasecurity/trivy/pkg/commands/artifact/run.go:260
  - scan failed:
    github.com/aquasecurity/trivy/pkg/commands/artifact.(*runner).scan
        github.com/aquasecurity/trivy/pkg/commands/artifact/run.go:615
  - failed analysis:
    github.com/aquasecurity/trivy/pkg/scanner.Scanner.ScanArtifact
        github.com/aquasecurity/trivy/pkg/scanner/scan.go:158
  - walk filesystem:
    github.com/aquasecurity/trivy/pkg/fanal/artifact/local.Artifact.Inspect
        github.com/aquasecurity/trivy/pkg/fanal/artifact/local/fs.go:113
  - walk dir error:
    github.com/aquasecurity/trivy/pkg/fanal/walker.(*FS).Walk
        github.com/aquasecurity/trivy/pkg/fanal/walker/fs.go:34
  - unknown error with /var/folders/kz/5_0qf4vn66v6521s9f4nm3rm0000gn/T/appy-Service-appy-frontend-4034367594.yaml:
    github.com/aquasecurity/trivy/pkg/fanal/walker.(*FS).Walk.(*FS).onError.func2
        github.com/aquasecurity/trivy/pkg/fanal/walker/fs.go:91
  - lstat /var/folders/kz/5_0qf4vn66v6521s9f4nm3rm0000gn/T/appy-Service-appy-frontend-4034367594.yaml: no such file or directory

Operating System

MacOS

Version

Version: 0.55.2
Vulnerability DB:
  Version: 2
  UpdatedAt: 2024-10-07 06:17:04.462736974 +0000 UTC
  NextUpdate: 2024-10-08 06:17:04.462736814 +0000 UTC
  DownloadedAt: 2024-10-07 09:25:56.846879 +0000 UTC
Check Bundle:
  Digest: sha256:ef2d9ad4fce0f933b20a662004d7e55bf200987c180e7f2cd531af631f408bb3
  DownloadedAt: 2024-10-07 10:33:31.13529 +0000 UTC

Checklist

• Run trivy clean --all
• Read the troubleshooting

[itaysk]

Here it is failing 3 times in a row, succeeding in the 4th time. No change between the commands, run immediately one after another (maybe 2 seconds apart)

~/dev/trivy-demo/appy/deploy ❯ trivy k8s --report all --include-namespaces appy --compliance k8s-pss-baseline-0.1
2024-10-08T10:47:59+03:00       INFO    Node scanning is enabled
2024-10-08T10:47:59+03:00       INFO    If you want to disable Node scanning via an in-cluster Job, please try '--disable-node-collector' to disable the Node-Collector job.
6 / 6 [--------------------------------------------------------------------------------------------] 100.00% 3 p/s
2024-10-08T10:48:02+03:00       FATAL   Fatal error     k8s scan error: scanning misconfigurations error: scan error: scan failed: failed analysis: walk filesystem: walk dir error: unknown error with /var/folders/kz/5_0qf4vn66v6521s9f4nm3rm0000gn/T/appy-Service-appy-backend-1781616976.yaml: lstat /var/folders/kz/5_0qf4vn66v6521s9f4nm3rm0000gn/T/appy-Service-appy-backend-1781616976.yaml: no such file or directory
~/dev/trivy-demo/appy/deploy ❯ trivy k8s --report all --include-namespaces appy --compliance k8s-pss-baseline-0.1
2024-10-08T10:48:07+03:00       INFO    Node scanning is enabled
2024-10-08T10:48:07+03:00       INFO    If you want to disable Node scanning via an in-cluster Job, please try '--disable-node-collector' to disable the Node-Collector job.
6 / 6 [--------------------------------------------------------------------------------------------] 100.00% 3 p/s
2024-10-08T10:48:10+03:00       FATAL   Fatal error     k8s scan error: scanning misconfigurations error: scan error: scan failed: failed analysis: walk filesystem: walk dir error: unknown error with /var/folders/kz/5_0qf4vn66v6521s9f4nm3rm0000gn/T/appy-ServiceAccount-default-325800095.yaml: lstat /var/folders/kz/5_0qf4vn66v6521s9f4nm3rm0000gn/T/appy-ServiceAccount-default-325800095.yaml: no such file or directory
~/dev/trivy-demo/appy/deploy ❯ trivy k8s --report all --include-namespaces appy --compliance k8s-pss-baseline-0.1
2024-10-08T10:48:15+03:00       INFO    Node scanning is enabled
2024-10-08T10:48:15+03:00       INFO    If you want to disable Node scanning via an in-cluster Job, please try '--disable-node-collector' to disable the Node-Collector job.
6 / 6 [--------------------------------------------------------------------------------------------] 100.00% 2 p/s
2024-10-08T10:48:18+03:00       FATAL   Fatal error     k8s scan error: scanning misconfigurations error: scan error: scan failed: failed analysis: walk filesystem: walk dir error: unknown error with /var/folders/kz/5_0qf4vn66v6521s9f4nm3rm0000gn/T/appy-Deployment-appy-frontend-2459893250.yaml: lstat /var/folders/kz/5_0qf4vn66v6521s9f4nm3rm0000gn/T/appy-Deployment-appy-frontend-2459893250.yaml: no such file or directory
~/dev/trivy-demo/appy/deploy ❯ trivy k8s --report all --include-namespaces appy --compliance k8s-pss-baseline-0.1
2024-10-08T10:48:31+03:00       INFO    Node scanning is enabled
2024-10-08T10:48:31+03:00       INFO    If you want to disable Node scanning via an in-cluster Job, please try '--disable-node-collector' to disable the Node-Collector job.
6 / 6 [--------------------------------------------------------------------------------------------] 100.00% 2 p/s

[afdesk]

hi Itay!
I could reproduce the issue in my environment:

trivy k8s --report all --include-namespaces rbac-test --compliance k8s-pss-baseline-0.1
2024-10-08T16:58:16+06:00	INFO	Node scanning is enabled
2024-10-08T16:58:16+06:00	INFO	If you want to disable Node scanning via an in-cluster Job, please try '--disable-node-collector' to disable the Node-Collector job.
6 / 6 [-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------] 100.00% 2 p/s
2024-10-08T16:58:20+06:00	FATAL	Fatal error	k8s scan error: scanning misconfigurations error: scan error: scan failed: failed analysis: walk filesystem: walk dir error: unknown error with /var/folders/q0/ykp5nvjx0j5_t4llxyh5hcm40000gn/T/rbac-test-ServiceAccount-default-1786258627.yaml: lstat /var/folders/q0/ykp5nvjx0j5_t4llxyh5hcm40000gn/T/rbac-test-ServiceAccount-default-1786258627.yaml: no such file or directory

will investigate it and create a new issue.

[afdesk]

closed in favor of #7684