Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Multiple model parameter with same value result in multiple evaluations #220

Open
Munsio opened this issue Jun 26, 2024 · 0 comments
Open
Assignees
Labels
bug Something isn't working question Further information is requested
Milestone

Comments

@Munsio
Copy link
Contributor

Munsio commented Jun 26, 2024

When running the evaluation and specifying the same model multiple times currently the evaluation is run X amount of times for the model.

Example:

eval-dev-quality evaluate --runtime docker --result-path ./docker-test --runs 5 --model symflower/symbolic-execution --model symflower/symbolic-execution --model symflower/symbolic-execution --repository golang/plain

This runs the symflower/symbolic-execution 3 times with 5 runs as the model was 3 times specified as parameter.

Question:
Do we want this behavior or should we unique the list of models after parsing?

@Munsio Munsio added bug Something isn't working question Further information is requested labels Jun 26, 2024
@Munsio Munsio self-assigned this Jun 26, 2024
@bauersimon bauersimon modified the milestones: v0.6.0, v0.7.0 Jul 30, 2024
@bauersimon bauersimon modified the milestones: v1.0.0, v1.1.0 Jan 15, 2025
@bauersimon bauersimon mentioned this issue Jan 15, 2025
27 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working question Further information is requested
Projects
None yet
Development

No branches or pull requests

2 participants