Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for custom values in corpus generation paths for better API test coverage #37

Open
kranonetka opened this issue Nov 5, 2024 · 9 comments

Comments

@kranonetka
Copy link

Is your feature request related to a problem? Please describe.
The corpus generation feature in the tool does not currently allow specifying custom data values that could improve test coverage. For example, in Keycloak, using the path /admin/realms/{realm}/clients/{client-uuid}/management/permissions, specifying a particular {realm} value can lead to higher coverage, as it triggers specific states or permission settings within that realm. Currently, there’s no way to instruct the fuzzer to generate corpora with this specific value for {realm}.

Describe the solution you'd like
I would like the tool to provide a way to specify custom values for certain parameters in corpus generation. For instance, having the option to define {realm=myrealm} for the path /admin/realms/{realm}/clients/{client-uuid}/management/permissions. The fuzzer could then use this custom value when generating corpora, leading to improved coverage and more thorough testing.

Describe alternatives you've considered
An alternative approach could involve manually generating corpora or scripting to include specific values for {realm}, but this is a labor-intensive process and doesn’t scale as efficiently as a built-in feature.

Additional context
This feature would be particularly beneficial for testing APIs with multiple parameters, where some values are known to trigger unique states or conditions. In Keycloak, for example, different realms or client configurations may expose distinct security or permission-related scenarios that are essential to cover in testing.

@grebnetiew
Copy link
Contributor

Hi! There is one way this can be done (and your issue tells me it should be documented better 😅): you can define an example in the specification for parameter values or object schemas. Corpus generation then prefers the example (but might generate additional values too).

It should also, but doesn't yet, honor the examples (plural) field. I'll add an issue for that.

    name: username
    in: path
    description: username to fetch
    required: true
    example: freddy
    schema:
      type: string

Apart from that, the separate corpus generation step was made so that it is possible for users to edit the corpus before starting the fuzzing campaign. So if you have a more elaborate 'custom corpus entry' consisting of e.g. multiple requests, you can add it to the corpus after it's generated (or delete entries you consider useless) before starting the fuzzing campaign.

@kranonetka
Copy link
Author

does this really work? I tried adding an example to both objects:

    name: username
    in: path
    description: username to fetch
    required: true
    example: freddy
    schema:
      type: string
      example: freddy

Generated the corpus and then tried

grep freddy corpus_directory/*

And nothing was found

@ThomasTNO
Copy link
Contributor

@kranonetka they should end up in the queue directory when fuzzing is started. Do you find them there?

@kranonetka
Copy link
Author

@ThomasTNO tryed after end of fuzzing and after 5, 20 secs after start, grep returns 0 matches

@ThomasTNO
Copy link
Contributor

@grebnetiew, this relates to #39 ?

@koltiradw
Copy link

@ThomasTNO, @grebnetiew, hello! Is it possible to specify a field in a query that has not been changed?

@ThomasTNO
Copy link
Contributor

@ThomasTNO, @grebnetiew, hello! Is it possible to specify a field in a query that has not been changed?

You mean, exclude specific fields from being mutated?

@koltiradw
Copy link

Yep!

@ThomasTNO
Copy link
Contributor

ThomasTNO commented Nov 7, 2024

Currently not, if I recall correctly. This would be an interesting feature. It does require quite some thought on how to incorporate this in terms of design choices. Modifying the OpenAPI spec is insensible. Somehow specifying a denylist could be a viable option. One would need to spend some time to draft out what format to use to specify this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants