Skip to content

Baselines for all tasks from Long Code Arena benchmarks ๐ŸŸ๏ธ

License

Notifications You must be signed in to change notification settings

JetBrains-Research/lca-baselines

Repository files navigation

๐ŸŸ๏ธ Long Code Arena Baselines

What is Long Code Arena?

Long Code Arena is a suite of benchmarks for code-related tasks with large contexts, up to a whole code repository. It currently spans six different tasks and contains six datasets:

Where are the baselines?

For each task, we have different approaches and environments. You can find the baseline solutions and the necessary information about each task in the corresponding directories.

How can I submit my results?

We are excited to invite you to participate in solving our benchmarks! To submit your results, please send the following materials to our ๐Ÿ“ฉ email ([email protected]):

  • Results: Include the summary of your benchmark outcomes.
  • Reproduction Package: To ensure the integrity and reproducibility of your results, please include the code for context collection (if any), generation of predictions, and evaluating. You can follow our baselines as a reference.
  • Metadata: Model information, organization name, licence of your model, context size, and other information you find relevant.

About

Baselines for all tasks from Long Code Arena benchmarks ๐ŸŸ๏ธ

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published