How to invoke a test using Pytest as a library/api, not as a cli tool

[velle]

I know how to call pytest.main, possible setting a long list of arguments. This is basically like calling the CLI from shell. It means that the test results are printed to stdout/stderr, and it means the output is controlled by flags.

I would like to somehow set up the test in my Python script, run the tests, and get a list of executed tests, and the result. E.g. a list of some object or maybe a dictionary.

Is that possible?

[The-Compiler]

See Calling pytest from Python code, which shows an example of passing a plugin object, on which you can implement any of the plugin hooks.

[velle]

Thanks :) I will soon look into that. But first, I would like to understand a little better, if it really can't be used in a "simpler" way.

I was hoping to write code that "feels like calling a library" (in lack of better words). Let me try to make it clear, what I mean. Supposedly the following is possible with unittest (source).

from unittest import TestLoader, TestCase, TestResult

class TestMy(TestCase):
    def test_silly(self):
        assert False

runner = TestLoader()
test_suite = runner.loadTestsFromTestCase(TestMy)
test_result = TestResult()
test_suite.run(test_result)
print(test_result)

The above code is very, very simple (for me at least), while using hooks is not. Both because I am much less used to thinking/coding with hooks, but also because such an approach requires a much better understanding of the inner workings of the library at hand (in this case pytest).

Do you understand what I'm trying to explain?

Is it "impossible" to use pytest in a way similar to how unittest is used above? I have been looking thoroughly for code examples, but I have not yet found anything like what I was expecting/hoping to find.

[The-Compiler]

That's indeed not possible. pytest is a framework and primarily a CLI tool, not a library.

[velle]

That's indeed not possible. pytest is a framework and primarily a CLI tool, not a library

Thanks for stating it that clearly :) I find it very surprising (and disappointing) though.

I wanted to build a file validator, for a specific file format. I have implemented a number of tests (in pytest), that check this, and then I run pytest using a custom --filepath argument. That was quite simple to build.

Now I want to wrap that tool in a simpler interface, such that this tool can be used by others without needing any understanding of pytest. Is that possible? I guess it is possible by using odd hacks and parsing stderr/stdout. But that is hardly any better than if I wrote the wrapper in bash or ruby. Am I right? Or is it possible to build something like this using hooks/plugins, and achieve something which is not a bunch of hacks? :)

[The-Compiler]

No need to parse the output, that's what the reporting hooks are here for. But as you already were told over in #13195, you're using pytest in a way it was never intended to be used.

[velle]

But as you already were told over in #13195, you're using pytest in a way it was never intended to be used.

True :) I understood that the answers discouraged my idea, however, I did not understand the explanation why of why pytest was such a bad choice. And I believe I actually found a solution to the part about compiling a list of all violations of a rule.

I did follow the advice, and went looking for other solutions, and in particular I spent several evenings learning about flake8, until I realized that flake8 is only for parsing/checking .py files. So in the end I had the choice between building it all from scratch, or giving pytest another try. And I decided to come back and ask again.

And I did learn more in this thread; in particular that pytest should not be used as a library. Thanks :)