RFC: testing framework

[bruchar1]

Most of the time, tests are written using test frameworks: gtest, catch2, cppunits, pytest, etc. Each framework uses its own conventions for displaying the list of tests, filtering tests to execute, reporting results, etc.

My proposition is to add a new objet in meson, to unify calling conventions between test frameworks. This would be particularly useful on big projects where different test frameworks are used together.

Here is how I see it:

testing = import('testing')

gtest = testing.framework(
    # cmd: ,  # e.g. pytest, when the exe is always the same (and only the arguments vary)
    # args: [],  # default args
    env: {},
    list_arg: '--gtest-list-tests',
    filter_arg: '--gtest_filter=@FILTER_ARGS@',
    filter_format: '*@FILTER_TAG@*',  # I have to think more about this...
    protocol: 'gtest',  # should be 'junit' in fact...
    timeout_multiplier: 1,  # not sure if this is needed, or if it should be 'timeout'
)

gtest.test('mytest', ...)  # same as the test() function

args from the framework would be added to the args of the test command. Things like protocol, if provided, would be default values when not provided by the test function.

Now some arguments could be added to the meson test command line:

--list-individual-tests (how should it be the behavior of --list in verbose mode?)
--filter "sub-test-name" (exact syntax of the filter TBD)

We could also imagine all test functions moving to the testing module (test, benchmark, add_test_setup), or the testing module being a top-level object instead, but in either case it would break compatibility, so it's probably not a good idea.

Before going too deep in the implementation, I'd like to know what you think about this? Is it a good idea? Am I missing something?