Hultnér Technologies AB | @ahultner | Blog | Slides | Nordisk Python Community | GitHub
Automatically generate test-cases based on your API-schemas.
The goal with this talk is to introduce the audience to property-based testing for API’s using schemas to automatically generate test scenarios, enabling them to write more powerful tests faster.
The talk will focus on a subset of the field of property-based testing where we focus on testing by automatically generating properties and test strategies from the API Schemas that we often already have. These tests ensure that our APIs conform to their specified schema and enables us to write a much larger amount of tests in less time.
I will focus on the schemathesis library which leverages the strong hypothesis library as well as the hypothesis-JSONSchema extension strategies, and will in the future also support GraphQL via the hypothesis-graphql strategies. I’m a contributor to schemathesis and currently working on the future GraphQL support with schemathesis creator, Dmitry Dygalo.
I will also compare it with its predecessor “swagger-conformance”, pure property-based testing through hypothesis, schema strategies with hypothesis-graphql and hypothesis-jsonschema, and discuss their advantages and disadvantages. I will also briefly talk about “QuickREST: Property-based Test Generation of OpenAPI-Described RESTful APIs”, a research paper that’s inspired part of the functionality in these tools.
By focusing on property-based test generation using schemas we already have, I will show that a field like property-based testing, which can seem quite daunting at first, actually can have a low barrier to entry while yielding large amounts of value in return and is useful for most common web projects today.
The talk will show how formal schemas for APIs can and will continue to provide additional value outside the scope of documentation.
$ schemathesis run http://127.0.0.1:5000/swagger.json
================== Schemathesis test session starts ==================
platform …
rootdir: /~/dev/hultner_technologies/Schema-based-API-Testing/code
hypothesis profile 'default' -> database=Directory…
Schema location: http://127.0.0.1:5000/swagger.json
Base URL: http://127.0.0.1:5000/
Specification version: Swagger 2.0
Workers: 1
collected endpoints: 5
GET /todos/ . [ 20%]
POST /todos/ . [ 40%]
DELETE /todos/{id} . [ 60%]
GET /todos/{id} . [ 80%]
PUT /todos/{id} . [100%]
============================== SUMMARY ===============================
Performed checks:
not_a_server_error 500 / 500 passed PASSED
========================= 5 passed in 5.32s ==========================# Minimal usage example
import requests
import schemathesis
schema = schemathesis.from_uri("http://0.0.0.0:8080/swagger.json")
@schema.parametrize()
def test_no_server_errors(case):
# `requests` will make an appropriate call under the hood
response = case.call() # use `call_wsgi` if you used `schemathesis.from_wsgi`
# You could use built-in checks
case.validate_response(response)
# Or assert the response manually
assert response.status_code < 500- Video: Automatically generate test-cases – Schema-based API testing - Alexander Hultnér 📯 - YouTube
- Type: Talk (20 mins)
- Schedule
- Slides
- Video: Alexander Hultnér - API-schema-based testing with schemathesis
- Type: Talk (30 mins)
- Python level: Intermediate
- Domain level: Intermediate
- Schedule
- Details on europython.eu
- Type: Talk (40 mins)
- Python level: Intermediate
- Domain level: Intermediate
- Schedule
Stateful testing?
Yes, using links.
GraphQL?
In-progress, check link for latest progress.
OpenAPI, Swagger?
Yes, Schemathesis supports both the older Swagger 2.0 standard and the newer OpenAPI 3 standard
Incompabilities?
Can be addressed using fixups, a built in fixup for FastAPI is included.
Activate in CLI with: --fixups=all
Do you have to use the network? No, Schemathesis can import your app directly and hit it without using the network if you use either WSGI or ASGI.
I have a question not covered here, where can I ask it?
I'm @ahultner on twitter, otherwise you can also email me (address in slides).