Automated tests

[SeanTAllen]

At the moment, the best we can do when checking a PR is to verify that stable builds. We have no automated tests of basic functionality. I'm opening this issue to collect and track thoughts on what tests we should add.

[SeanTAllen]

@jemc @Theodus any thoughts on what automated tests for stable would look like?

[jemc]

Maybe it could be an integration test that sets up an example bundle.json that clones its own repo (pony-stable), and imports some types to prove that it worked?

[SeanTAllen]

@jemc i was just thinking the same thing. and then compile the program with stable env ponyc.

we can verify

• bundle file is correct after the add (this could be expanded for each type eventually)
• stable env ponyc compiles program

eventually i think we'd want to test other features like

• tag
• local-git
• github
• stable add

[SeanTAllen]

@rawkode any interest in this one? it's a little more advanced but probably manageable as a start. there's a bit of "how much can we use ponytest for" (a lot see https://github.com/ponylang/changelog-tool/blob/master/tests/main.pony tests)? and some would have to be some kind of automated integration test(s), so there's a bit of complication but its pretty straightforward. if you are interested, i'd say drop idea here for tests and what the runner would be and @jemc and I could give feedback.

[rawkode]

@SeanTAllen Yeah, I'll look at what I can do and come back with a plan 👍

[SeanTAllen]

@rawkode assume. i just sent you an invite to be part of our "contributor" team. it doesn't give you much in the way of privileges but does allow for you to appear in the "Assignee" list and be assigned to issues you are interested in working on. Baby steps towards committer!

[rawkode]

So this is pretty tough.

My initial thought is that we need a JSON Schema that will evolve with the tool and provide a means of documenting the format for other integrations; as well a confidence that stable conforms with said schema.

From what I can tell, we could probably use ponytest for arbitary comparison tests, but it wouldn't be pretty and we'd need to provide the json for comparison. Which means, in order for the test not to be flaky, we need to add branching/tag/commit sha support to stable add

assert_eq("json", loaded_bundle.json)

Hopefully any assumptions I've made are correct 😄

[srenatus]

I'll try pushing this forward. #70 gets some scaffolding in place, leaving open the following test TODOs:

• stable fetch
• stable add
• stable env: Add integration tests for 'stable env' #73