Allow test() to check a boolean value

[jkrenzer]

Hi!

Currently I am working on a project where we need to test negative features of a library, meaning that certain usecases MUST NOT compile - even though they are valid code. This way we move checking of proper usage of the lib as far into compile time as possible.

To do this with meson's test functionality, I currently use sh as the testing program with a cmd line, that invoces meson compile and asserts the return value to be 1.

Looking at the documentation of the compiler object, the usecase could be implemented with compiles() much more readable, if test() could consume a boolean value as input. This could also open the door to checking other conditions implementable in meson or values returned by a runresult for code assurance testing without rigging a meson subprocess or something else into a less readable shell command.

Cheers,

Jörn

[eli-schwartz]

Compiles would return that boolean at configure time, not at test time, so the test function seems like it would be useless. Why not just emit an error at configure time?

[jkrenzer]

Ok, then the compiles() function would be useless as test subject, indeed.

Issuing an error at configure time is not what we need. We run meson test to decide if the library is healthy and can be packaged. So we run unit tests and also some tests to make sure compilation of forbidden usecases is stopped by asserts with a clear error message to the including developer.

Right now the tests are somewhat jury rigged by recursively calling meson compile from the test with some shell expressions. I'd think it would be much more transparent for the user if the test could use a runresult or a boolean value variable in meson (which could be set reusing the already implemented conditionals in meson on a runresult) to decide if it was successful. It would be platform-independent as well n contrast to the shell scripting.

[eli-schwartz]

I think maybe a better way to phrase what you want is that you'd like a hypothetical new function called:

testcase_compile(
    'name',
    sources: ['a.c', 'b.c'], dependencies: [...],
    other_kwargs_from_build_target,
    priority: 0,
    should_fail: false,
    suite: '',
    verbose: false,
)

It would accept some kwargs from build_target(), and some kwargs from test(), and write out the same kind of rules normally written out to build.ninja except instead written out to the testdata and run from the testdata.

[jkrenzer]

Yes, you could also phrase it this way. But frankly test() is almost very nearly there. Currently a negatice compilation test looks like this:

nc_check_exe_framework_pus = executable(
    'nc_check_exe_framework_pus',
    'framework/nc_check_framework_pus.cpp',
    include_directories: cpp_test_include_dirs,
    dependencies: [dep_libtappp, dep_fmt],
    native: true,
    build_by_default: false,
    override_options: ['buildtype=debugoptimized'],
    cpp_args: args_cpp,
    link_args: args_link,
)

test(
    'nc_check_framework_pus',
    find_program('sh'),
    args: [
        '-c',
        '! meson compile nc_check_exe_framework_pus',
    ],
    verbose: false,
    suite: ['nc_checks'],
)

Introducing a fictional option compile_test: true, which would make test() check the return value of the compilation, could transform the example to a much more descriptive:

nc_check_exe_framework_pus = executable(
    'nc_check_exe_framework_pus',
    'framework/nc_check_framework_pus.cpp',
    include_directories: cpp_test_include_dirs,
    dependencies: [dep_libtappp, dep_fmt],
    native: true,
    build_by_default: false,
    override_options: ['buildtype=debugoptimized'],
    cpp_args: args_cpp,
    link_args: args_link,
)

test(
    'nc_check_framework_pus',
    nc_check_exe_framework_pus,
    compile_test: true,
    should_fail: true,
    verbose: false,
    suite: ['nc_checks'],
)

I could use should_fail: true here but one could argue, that the should_fail state was originally intended to signify not yet implemented tests/features, and our expected failure is the intended final implementation state of the negative compilation checks, which should be a 'pass' state. Kind of a philosophical consideration on the interpretation of the test-suit results according to team guidelines (expected fails might be counted as todo issues by trackers) and intended observer impression.

Introducing a fictional parameter return_code_pass one could modify the test to expect a return code of 1 from the executable as a signal for a passed test

test(
    'nc_check_framework_pus',
    nc_check_exe_framework_pus,
    compile_test: true,
    return_code_pass: 1,
    verbose: false,
    suite: ['nc_checks'],
)

If we consider touching the syntax, we could also think about introducing return_code_skip so this could be customized, too.
Both return_code_pass and return_code_skip could of course be used independend of the value of compile_test.