Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor requirements and clarify benchmark harness instructions #8

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Refactor requirements and clarify benchmark harness instructions #8

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -95,11 +95,18 @@ git clone https://github.com/paul-gauthier/aider-swe-bench
cd aider-swe-bench
git clone https://github.com/aorwall/SWE-bench-docker

# Create a virtual environment
python -m venv venv
source venv/bin/activate # On Windows use `venv\Scripts\activate`

# Install pip requirements
pip install -r requirements.txt

# You may want to install the latest main branch of aider
python -m pip install --upgrade git+https://github.com/paul-gauthier/aider.git

# Note: The SWE-Bench docker images are automatically pulled when needed.
# The harness.py script interacts with these images to run the benchmarks.
```

See the
Expand Down
5 changes: 2 additions & 3 deletions harness.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,8 +14,7 @@

from dump import dump
from tests import run_tests
from utils import get_full_dataset # noqa: F401
from utils import get_lite_dataset # noqa: F401
from utils import get_full_dataset, get_lite_dataset # noqa: F401
from utils import get_devin_instance_ids, get_plausible, load_predictions, pick_winner

REPOS_DNAME = Path("repos")
Expand Down Expand Up @@ -482,7 +481,7 @@ def main():
if just_devin_570:
# Filter it to the Devin 570
devin_insts = get_devin_instance_ids()
dataset = dict((inst, entry) for inst, entry in dataset.items() if inst in devin_insts)
dataset = {inst: entry for inst, entry in dataset.items() if inst in devin_insts}

#dataset = {"sympy__sympy-18532" : dataset["sympy__sympy-18532"]}

Expand Down