Performance instrumentation for individual and overall constraint processing

[aj-stein-gsa]

User Story

As a developer of Metaschema-enabled software, models, and data, I would like performance instrumentation to measure individual constraints and overall model and external constraint processing to determine hotspots, performance bottlenecks, and areas for improvement.

Goals

• Determine micro and macro-level performance of this library and dependent tools
• Analyze performance to add, change, or remove features and a specific implementation to meet ongoing needs of Metaschema users for heterogenous use cases (e.g. OSCAL and others)

Dependencies

No response

Acceptance Criteria

• All website and readme documentation affected by the changes in this issue have been updated.
• A Pull Request (PR) is submitted that fully addresses the goals of this User Story. This issue is referenced in the PR.
• The CI-CD build process runs without any reported errors on the PR. This can be confirmed by reviewing that all checks have passed in the PR.

Revisions

No response

[aj-stein-gsa]

@wandmagic I created this issue based upon our discussion in standup today about more precise perf counters.

[wandmagic]

This would be really handy especially as we scale up ssp data size

[david-waltermire]

How should this work? We need to get to some form of a spec we can implement.

[aj-stein-gsa]

How should this work? We need to get to some form of a spec we can implement.

Agreed. I was looking into instrumentation systems for Java applications.

[wandmagic]

we could just have a flag for --instrumentation in the CLI (or a flag to turn it off --skip-instrumentation)
and then in the sarif output, a custom prop with how much processing time was used during this rule:

  "version": "2.1.0",
  "$schema": "https://raw.githubusercontent.com/oasis-tcs/sarif-spec/master/Schemata/sarif-schema-2.1.0.json",
  "runs": [{
    "tool": {
      "driver": {
        "name": "oscal-cli",
        "rules": [{
          "id": "{uuid}",
          "name": "resolves-to-valid-content",
          "properties": {
            "executionTimeMs": 324
          }
        }]
      }
    },
    "results": [
      // ... regular results
    ]
  }]
}

[aj-stein-gsa]

Before extending SARIF data and reinventing the wheel, one very rough (not so granular) data source we could tap into (but current do not) is the JUnit/Surefire reports we could store, but do not, in GitHub or elsewhere given we use that plugin with Maven.

That said, it only tells us what the macro-level "I ran this test that calls of this other code across modules in one or more function calls," and nothing more granular, like I said. I have been researching this on and off all morning and found nothing very compelling about time measurement and profiling, but I will have to read up on this area.

That said, if we could find a way outside of m-j and oscal-cli code to actually annotate code with time runs in SARIF since we know what code paths are used for tests, and annotate function calls that exceeds a thresshold or deserve investigation, we may be onto something I think no one else is doing (open source or on the inner source proprietary side, I'll have to ask; no one has ever hinted they do something like that, so we would be trendsetters).