Unit Tests

[mkaito]

Testing addons is... controversial. I do not expect to get any help from carbine (See here).

As far as I can tell, there are two major approaches to the problem:

Write a test harness as an addon.

This addon would check all other installed addons for tests, run them, and report back. To simplify the "interface", just run everything on one of the later Apollo events (to make sure everything else is loaded), and report back via GeminiConsole. We would have to figure out whether tests can simulate user interaction and whether we can assert the state of the interface.

The upside to this is that we test correctly, where the code is intended to run, and tests will help us keep up with any API changes. We do not need to worry about Apollo. When Apollo breaks out code, the tests will fail and tell us exactly where things broke.

The downside is a massive feedback loop. Every test run needs to be initiated manually via /reloadui, takes a few seconds at best (about 3-4s on my SSD/quadcore), and runs the entire test suite every time. We also depend on the game itself to run tests. If the servers are down, no tests. If you're on a shitty computer that can't run Wildstar, no tests. No continuous integration, automatic testing, and other test related goodies. Manual tests can and will be forgotten and/or ignored.

Write a mock library for Apollo

This would entail manually writing out mocks for every Apollo function we use, and keeping the mocks up to date with any API changes. We could write an addon that ran all declared mocks and compared them to actual Apollo calls to make this easier. We can run the interface forms through an XML parser and write asserts for properties and hierarchy.

The upside is automatic testing. We get the usual testing goodies: automatic test runs on file save, fast feedback loops, continuous integration, etc. Github integrates well with travis so you don't necessarily have to pull and check every pull request locally. We get to use anything ever written for Lua testing.

The downside is that some parts of Apollo might be tricky to "mock". I'm thinking specifically of calls to Apollo.getPackage() and the likes. We don't have to mock out the entire game API to start. Just mock what we actually need and then incrementally build the library as we need it. Also, keep the mock library in a separate repo, because it's not really part of VikingUI, and because there's a chance it'd be picked up by the commmunity.

[mgasparel]

On the surface, it seems simpler to write a test harness as an addon... but there are just so many pitfalls vs the mock library. Also, I agree that if we get a mock library working, it's something that would benefit the entire community and game.

[mkaito]

At first glance, these are sticking points I can see:

• Apollo.GetPackage
• Apollo.LoadForm needs to return window handle mocks
• Apollo.RegisterEventHandler and actually calling those events as part of the test run

[mkaito]

Another idea is to build a "testing mode" switch into addons, in a style similar to how the D or Rust language include unit test blocks right alongside the code they're testing. Have assert blocks delimited by if settings.testing, or ran by a lua expression.

I think we should do both this and the Apollo mocks.

[mkaito]

We're not the first to have the idea, as you would have thought:

https://github.com/vikinghug/Jarod