Additional information on Evaluate

[meganwolf0]

Is your feature request related to a problem? Please describe.

When evaluate runs, and fails, only the control IDs are output. More information would be helpful for debugging what actually failed, since the control satisfaction is often a function of many observations.

Describe the solution you'd like

• Given assessment-results
• When lula evaluate fails
• Then list observation changes

Additional context

Thinking a decent solution would be:

• Identify old -> new observation pairs by the observation.description text matching. Print out those with mis-matched relevant-evidence.description values (i.e., the result: not-satisfied/satisfied)
• For any new observations that are not-satisfied without a previous old observation, print those.

This is just my idea for evaluate, but perhaps this rolls into a larger function to interrogate the assessment results more granularly and specific to individual controls.

[mjnagel]

From a user looking at lula evaluate output in CI, this would be super helpful to either expand the default information or provide a debug/verbose mode. For reference my current process to understand a failing evaluation is:

• Get a list of control IDs from the CI/command output
• Download the assessment results (in our case they are in CI artifacts)
• For each control ID listed as not-satisfied, check the results to get a list of related-observations (worth noting at this point I only see that the control was not-satisfied, not necessarily which observations/evidence were not satisfied)
• For each observation-uuid check the relevant-evidence to see if it was satisfied or not-satisfied
• If not-satisfied, review the remarks

If lula evaluate output the specific not-satisfied observations, or even further the associated remarks that would hugely simplify this process.

[meganwolf0]

@mjnagel - thanks for the feedback! Yeah agreed this process is not great and is like trying to untangle a spaghetti of uuids. The draft PR I'm working has the output listing all different observations and all new or missing ones for each failed control:

From a user perspective, if you have thoughts on how to make this more informative, cleaner, whatever lmk! Just trying to get the information in there as a first cut at solving this traceability problem.

[mjnagel]

Yeah I think that's a super helpful start. Two things that I would find useful:

• Ability to filter out any observations that were satisfied - maybe not by default but if I could reduce the noise to just focus on what failed that would be great so that I don't have to read through all the observations (maybe I'm just misunderstanding the above screenshot and this is already true - I think the first observation block with satisfied is throwing me off).
• Not really sure how to do this but if there were an easy way to link back/provide the details of the validation check that was used (i.e. the actual opa check) that could be helpful. This might just be (1) me getting used to the format and remembering that the UUID in the description test is the UUID in the oscal and (2) a bit of a short term issue as the validations are still maturing.

This looks great though and is exactly what I was hoping to see!

[meganwolf0]

Yep, this is supposed to just show the ones that changed, that formatting is probably confusing - but the first in the list is like the old version (where it was passing + the "evidence" and the second is it's pair that is now failing - I feel like I could probably rearrange this to make it more obvious that's what's happening.

On the validation part, are you thinking you'd like to see the rego? and/or the input data? We have some other issues open to add like "human friendly" readme + composing that in the OSCAL, but that's something we would still need to implement but could possibly add the raw rego though...

[mjnagel]

Yep, this is supposed to just show the ones that changed, that formatting is probably confusing - but the first in the list is like the old version (where it was passing + the "evidence" and the second is it's pair that is now failing - I feel like I could probably rearrange this to make it more obvious that's what's happening.

Gotcha, maybe my examples are limited but I'm thinking most of the time I'd only care about the new evidence section, not necessarily seeing the "passing" version?

On the validation part, are you thinking you'd like to see the rego? and/or the input data? We have some other issues open to add like "human friendly" readme + composing that in the OSCAL, but that's something we would still need to implement but could possibly add the raw rego though...

I think the rego could be useful, but probably should should be part of a verbose mode flag or something like that since it could be a lot to log out by default. It's mostly just in the line of thinking "well this failed, I see the validation error message, but was that because the check needs an update?"

A good example related to your screenshotted validation - if we switched from our current sidecar implementation to a "native sidecar" implementation I'd expect that to fail the validation. The message would just say "Istio sidecar proxy not found..." but in reality the rego check is really the "problem"/thing that needs to be reviewed.