ci: add test_api job

[afuetterer]

Proposed Changes

Add a CI job that:

• spins up the docker-compose stack
• adds the test corpus
• loads the data for the test corpus
• runs a suite of API tests against the running API service

Requirements

• add a GitHub actions workflow that runs on PRs
• start the docker-compose stack
• add test corpus
• load data for test corpus
• run test suite with schemathesis
• reporting should be done with schemathesis

Problems

1. I cannot start the frontend service with docker-compose up.
2. If I run docker-compose up --detach api fuseki metrics to not start the frontend:

Tasks

• rebase branch
• use compose.t.yml
• remove ci.yml workflow file

Add run: docker images to list used images

Related issue: #219

[cmil]

• http://localhost:8080/exist/restxq/v1/info does not accept GET (HTTP ERROR 405 HTTP method GET is not supported by this URL)

Not sure how you got this error, but in the test run I see Error: Process completed with exit code 52. This indicates that curl doesn't get any response from the server. I suspect that the eXist DB has not yet completely started when you run the request. You may need to wait for the eXist to get ready to serve.

[afuetterer]

I think to test setup is working now. What do you need to test at this stage?

There are a few higher level groups of checks.

• not_a_server_error. The response has 5xx HTTP status;
• status_code_conformance. The response status is not defined in the API schema;
• content_type_conformance. The response content type is not defined in the API schema;
• response_schema_conformance. The response content does not conform to the schema defined for this specific response;
• response_headers_conformance. The response headers does not contain all defined headers.

See: https://schemathesis.readthedocs.io/en/stable/cli.html#how-are-responses-checked

And you can select all of them.

Which ones do you need?

If you select all, you get a lot of errors.

[afuetterer]

How would you like to receive the output of the tests? Are the logs in GitHub actions sufficient? Do you need custom summaries?

If you register with schemathesis.io, they can generate reports for you and they will post comments in PRs.

It is your decision, if you want to use a third-party provider for testing the API.

[afuetterer]

What do you think?

Do you know what you to test?

Should this be not too strict, tests green and be merged soonish? Then you have the skeleton to test the API, additional stricter testing can then be enabled as a next step.

If you test "everything" now, the tests are red and you will not be able to merge without a lot of work fixing the found issues.

[cmil]

How would you like to receive the output of the tests? Are the logs in GitHub actions sufficient? Do you need custom summaries?

If you register with schemathesis.io, they can generate reports for you and they will post comments in PRs.

I have set up a dracor.org organization at https://app.schemathesis.io/organizations/dracor-org/. @afuetterer I could add you as a member if you let me know your schemathesis account or email address. So we could use schemathesis.io to generate reports on test runs. I have briefly tried to do a local test run following their documentation but wasn't able to upload the result to schemathesis.io. Maybe you could figure out how to do it.

[cmil]

What do you think?

Do you know what you to test?

Should this be not too strict, tests green and be merged soonish? Then you have the skeleton to test the API, additional stricter testing can then be enabled as a next step.

If you test "everything" now, the tests are red and you will not be able to merge without a lot of work fixing the found issues.

For the PR to be merged we should arrive at some minimal set of tests that passes. I haven't looked into schemathesis enough to say which tests that could be. Can you suggest what would be possible to test successfully at the moment_

[afuetterer]

How would you like to receive the output of the tests? Are the logs in GitHub actions sufficient? Do you need custom summaries?
If you register with schemathesis.io, they can generate reports for you and they will post comments in PRs.

I have set up a dracor.org organization at https://app.schemathesis.io/organizations/dracor-org/. @afuetterer I could add you as a member if you let me know your schemathesis account or email address. So we could use schemathesis.io to generate reports on test runs. I have briefly tried to do a local test run following their documentation but wasn't able to upload the result to schemathesis.io. Maybe you could figure out how to do it.

If you want to use the schemathesis reporting, fine. Then "report: false" should be "report: true", and you need to set the token as a secret. Then it should work.

[afuetterer]

What do you think?
Do you know what you to test?
Should this be not too strict, tests green and be merged soonish? Then you have the skeleton to test the API, additional stricter testing can then be enabled as a next step.
If you test "everything" now, the tests are red and you will not be able to merge without a lot of work fixing the found issues.

For the PR to be merged we should arrive at some minimal set of tests that passes. I haven't looked into schemathesis enough to say which tests that could be. Can you suggest what would be possible to test successfully at the moment_

The not_a_server_error checks group is working with a single failure, which can be ignored for now.
This should let the tests pass and give you a starting ground for adding more tests/checks in the future.

And this actually tests a lot of endpoints already!

[afuetterer]

Tests are passing.

[Stranger6667]

How would you like to receive the output of the tests? Are the logs in GitHub actions sufficient? Do you need custom summaries?
If you register with schemathesis.io, they can generate reports for you and they will post comments in PRs.

I have set up a dracor.org organization at https://app.schemathesis.io/organizations/dracor-org/. @afuetterer I could add you as a member if you let me know your schemathesis account or email address. So we could use schemathesis.io to generate reports on test runs. I have briefly tried to do a local test run following their documentation but wasn't able to upload the result to schemathesis.io. Maybe you could figure out how to do it.

Could you please, share what was the issue during uploading? I am eager to fix it or either update the docs. In any event I’d be happy to work on any improvements you might need from the Schemathesis side :)

[afuetterer]

What do you say @cmil?

[cmil]

How would you like to receive the output of the tests? Are the logs in GitHub actions sufficient? Do you need custom summaries?
If you register with schemathesis.io, they can generate reports for you and they will post comments in PRs.

I have set up a dracor.org organization at https://app.schemathesis.io/organizations/dracor-org/. @afuetterer I could add you as a member if you let me know your schemathesis account or email address. So we could use schemathesis.io to generate reports on test runs. I have briefly tried to do a local test run following their documentation but wasn't able to upload the result to schemathesis.io. Maybe you could figure out how to do it.

Could you please, share what was the issue during uploading? I am eager to fix it or either update the docs. In any event I’d be happy to work on any improvements you might need from the Schemathesis side :)

I think the problem was that the report size was too large and I may just have missed the warning about this. I have now been able to upload reports. Although I have only managed to do this with my personal account. Is there a way to upload the reports to the dracor-org organisation? I thought maybe I need an organization token but couldn't figure out how to obtain one.

[cmil]

What do you say @cmil?

Thanks a lot @afuetterer. I added a few minor things, mostly cosmetic or, in the case of schemathesis.sh, as a memory aid. I think that's a great base to build on and extend the tests.

Just for the record, there was one runtime error in https://github.com/dracor-org/dracor-api/actions/runs/7438586065/job/20237499097, which I was not able to reproduce locally or in later workflow runs:

==================================== ERRORS ====================================
__________________________ GET /exist/restxq/v1/info ___________________________
Runtime Error

`register_random` was passed `r=<random.Random object at 0x7f8e1400eeb0>` which will be garbage collected immediately after `register_random` creates a weakref to it. This will prevent Hypothesis from managing this PRNG. See the docs for `register_random` for more details.

I can't see how any of my changes would have caused or fixed this. So I assume it was a temporary glitch and think that we are ready to merge. 🎉

[afuetterer]

Awesome.

If you enjoy using schemathesis for the testing, consider giving them a star on GitHub.

Also maybe this is worth a blog post and/or some other sort of publication?

[cmil]

I gave them a star. If you feel to write a blog post, go ahead. You were the main implementer of this after all. :)

[Stranger6667]

I think the problem was that the report size was too large and I may just have missed the warning about this. I have now been able to upload reports. Although I have only managed to do this with my personal account. Is there a way to upload the reports to the dracor-org organisation? I thought maybe I need an organization token but couldn't figure out how to obtain one.

Sorry for the delay! Thank you for sharing - I increased the limits for anonymous uploads, for registered users they are 10MB (let me know if it is not enough).

There are organization tokens, but I didn't add a UI for them just yet. Gimme a few days and it will be there.

[Stranger6667]

Just for the record, there was one runtime error in https://github.com/dracor-org/dracor-api/actions/runs/7438586065/job/20237499097, which I was not able to reproduce locally or in later workflow runs:

I also observed such a sporadic failure on multi-worker test execution (the test run uses 2 workers), and it likely comes from the Hypothesis framework (which Schemathesis uses under the hood). I didn't check it in detail but in my case, it was resolved after updating the Hypothesis version.

[cmil]

Sorry for the delay! Thank you for sharing - I increased the limits for anonymous uploads, for registered users they are 10MB (let me know if it is not enough).

Thank you! That's great and should be more than enough for now.

There are organization tokens, but I didn't add a UI for them just yet. Gimme a few days and it will be there.

Looking forward to the new feature. But no rush!

[cmil]

Just for the record, there was one runtime error in https://github.com/dracor-org/dracor-api/actions/runs/7438586065/job/20237499097, which I was not able to reproduce locally or in later workflow runs:

I also observed such a sporadic failure on multi-worker test execution (the test run uses 2 workers), and it likely comes from the Hypothesis framework (which Schemathesis uses under the hood). I didn't check it in detail but in my case, it was resolved after updating the Hypothesis version.

I saw a few more failures as well. Would an update of the Schemathesis Github action then fix this in the future? We are currently using version 1.0.3 which seems to be the latest.