Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor requirements and clarify benchmark harness instructions #8

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

bscriver123
Copy link

Pull Request Description

Issue Addressed

Fixes several issues encountered while rerunning swe-bench-lite with aider.

Changes Made

  1. Updated installation instructions in README.md to include creating a virtual environment before running pip install -r requirements.txt.
  2. Clarified the suggestion to read SWE-Bench docker docs for image building/pulling procedures.
  3. Modified harness.py:
    • Uncommented the import of get_lite_dataset.
    • Imported all necessary functions and added a # noqa comment.
    • Addressed the use of just_devin_570, ensuring it is appropriately disabled for lite datasets.
    • Resolved the NotImplementedError related to loading cached datasets in a LocalFileSystem.

Additional Improvements

  1. Included a frozen requirements file to prevent future issues with external dependencies.
  2. Enhanced clarity and consistency throughout the codebase for a more streamlined experience.

Resolves

  • Issue Title: A few issues while trying to rerun swe-bench-lite with aider

Suggestions for Reviewers

  • Please review the modifications in the installation instructions, harness.py script, and the inclusion of a frozen requirements file.
  • Feedback on the clarity and effectiveness of the code improvements is appreciated.

Related Links

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant