Testing

Testing Policy

• All changes must be tested
• Tests must be run before pushing changes to the master branch

What this means

It is the responsibility of the person submitting a change for code review to adequately write tests for their change. Whether they use unit tests or ktest is up to them, however, certain types of code changes are not easily tested using ktest (e.g. error cases).

Sample unit tests

https://github.com/kframework/k/blob/master/src/javasources/KTool/test/org/kframework/backend/java/builtins/BuiltinIOOperationsTest.java https://github.com/kframework/k/blob/master/src/javasources/KTool/test/org/kframework/krun/ioserver/filesystem/portable/PortableFileSystemTest.java

Sample end-to-end tests

https://github.com/kframework/k/tree/master/tests/regression/java-rewrite-engine/strings https://github.com/kframework/k/tree/master/tests/regression/java-rewrite-engine/set

Merging changes into master

It is your responsibility as a developer to make sure the test suite is run and all tests pass prior to a commit. Jenkins will facilitate this, however it is ultimately your responsibility. If you commit a change that breaks the build without code reviewing it first, you will be asked to provide a very clear explanation for why it happened, because code review is a simple expectation on all commits. If you do it a second time, your commit access will be revoked.

Solving failed tests

Sometimes tests will fail when you merge into master. Assuming you have taken appropriate steps to test your change (i.e. run all tests before committing, and merging changes with the upstream if you expect conflicts), and assuming it has been code reviewed and I signed off on it having passed tests, you will not be blamed for this. However, regardless of whether blame falls on you for causing tests to fail in master, there are certain expectations you must perform if that occurs.

1. Determine whether the tests failed due to some kind of transient failure or because of your change. If you can demonstrate clearly that the failure has nothing to do with your change, rerun the test suite. If it passes, you are done. Otherwise, or if the tests failed due to your change, proceed to step 2.
2. Revert your change. This does not require a code review. If the first commit you want to revert has hash , and the last commit you want to revert has hash , run git revert <commit1>^..<commit2>. Note that this will not behave as you expect it to if you try to select a range of commits that are not ancestors of each other, for example if you try to select two commits that were parts of different branches without also selecting the revisions that merge these branches.
3. Push your revert upstream, once you have verified that it looks as you expect it to.
4. Cherry-pick your change back into your local repository with git cherry-pick <commit1>^..<commit2>.
5. Fix your change so that it passes all the tests (making sure to merge with upstream to ensure that you have considered any tests that might fail due to conflicts).
6. Submit your modified change as a pull request.

Failure to promptly follow steps 1-3 will put you on the same type of probation you would be put under for failing to code review your changes.