Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Restructure Journey Tests #1597

Open
48 tasks
samayer12 opened this issue Dec 17, 2024 · 1 comment
Open
48 tasks

Restructure Journey Tests #1597

samayer12 opened this issue Dec 17, 2024 · 1 comment
Labels
complexity complexity github_actions Pull requests that update GitHub Actions code large large tech-debt Testing Testing

Comments

@samayer12
Copy link
Collaborator

samayer12 commented Dec 17, 2024

Describe what should be investigated or refactored

The tests in journey/ are one of the slower aspects of our CI/CD pipeline. They are also interwoven and are not isolated/independent from each other. In general, we see the journey/ directory and think "here be dragons."

We should improve our Journey tests by doing the following smaller, well-scoped actions:

Entrypoint Changes

  • Adopt common naming scheme (i.e., *.test.ts)
  • Adopt BDD test-naming patterns
  • Enable independent execution of peprInit.
  • Enable independent execution of peprFormat.
  • Enable independent execution of peprBuild.
  • Enable independent execution of peprDeploy.
  • Enable independent execution of peprDev.
  • Remove circular dependency between entrypoint.test.ts and other test files
  • Reassess global test timeout of 5 minutes

General Changes

  • Remove need for TEST_MODE environment variable.
  • Present clear feedback messages when tests fail
  • Do not attempt to open IDEs during test execution
  • Make Journey tests & wasm tests execute independently of each other (scope?)

pepr-init.ts

  • Use expect() statements in tests to assert on some condition
  • Add error-handling to execSync() commands
  • Adopt BDD test-naming patterns
  • Use test setup and teardown functions to avoid conflicts when running in parallel.

pepr-format.ts

  • Use expect() statements in tests to assert on some condition
  • Add error-handling to execSync() commands
  • Adopt BDD test-naming patterns
  • Use test setup and teardown functions to avoid conflicts when running in parallel.

pepr-build.ts

  • Add error-handling to execSync() commands
  • Reduce tight coupling between test logic & validation logic in validate*() functions
  • Parameterize tests that call npx pepr build
  • Adopt BDD test-naming patterns
  • Provide complete test descriptions in test names
  • Remove inline type definitions
  • Avoid implicit assumptions on calls that could fail silently
  • Remove hardcoded paths in test implementation

pepr-deploy.ts

  • Break up monolithic structure for test logic, validation logic, and utility functions
  • Use parameterized tests for similar calls to waitForConfigMap()
  • Adopt BDD test-naming patterns
  • Provide complete test descriptions in test names
  • Use setup and teardown test hooks for common tasks (see cleanupSamples())
  • Apply error handling to spawnSync() and execSync() calls
  • Reduce code duplication between testMutate, testValidate, and testStore
  • Use consistent assertions (includes vs exact matches)
  • Remove hardcoded paths in test implementation
  • Consider splitting this file into smaller files aligned with each describe() block

pepr-dev.ts

  • Add timeouts to async calls to avoid hanging tests
  • Adopt BDD test-naming patterns
  • Provide complete test descriptions in test names
  • Use setup and teardown test hooks for common tasks
  • Remove use of shared state (see success and expectedLines)
  • Remove use of unnecessary calls to done()
  • Simplify afterAll() by simply killing process and avoiding manual teardown
  • Avoid use of hardcoded variables and endpoints

CI/CD Changes

  • Parallelize journey test execution in CI/CD pipeline.

Additional context

Related issues: #1140, #1106, #978
Related branches: 1140-journey-refactor

@samayer12 samayer12 added github_actions Pull requests that update GitHub Actions code tech-debt Testing Testing large large complexity complexity labels Dec 17, 2024
@samayer12
Copy link
Collaborator Author

There's also value for an audit of test cases against what we cover in https://github.com/defenseunicorns/pepr-excellent-examples

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
complexity complexity github_actions Pull requests that update GitHub Actions code large large tech-debt Testing Testing
Projects
Status: 🆕 New
Development

No branches or pull requests

1 participant