Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add the BlueBench benchmark #2369

Open
wants to merge 48 commits into
base: main
Choose a base branch
from
Open

Add the BlueBench benchmark #2369

wants to merge 48 commits into from

Commits on Oct 1, 2024

  1. add bluebench files

    @shachardon
    shachardon committed Oct 1, 2024
    Configuration menu
    Copy the full SHA
    f5282e6 View commit details
    Browse the repository at this point in the history

  2. tmp - run with local unitxt installation

    @shachardon
    shachardon committed Oct 1, 2024
    Configuration menu
    Copy the full SHA
    8537fe5 View commit details
    Browse the repository at this point in the history

  3. Update README.md (EleutherAI#2297) 

    * Update README.md

I encounter some Git buffer size limits when trying to download all commits history of the repository, such as:
```error: RPC failed; curl 18 transfer closed with outstanding read data remaining
error: 5815 bytes of body are still expected
fetch-pack: unexpected disconnect while reading sideband packet
fatal: early EOF```

therefore the installation is faster and there are not errors when I download only the last version of the repository

* Fix linting issue
    @shachardon
    shachardon committed Oct 1, 2024
    Configuration menu
    Copy the full SHA
    79d6066 View commit details
    Browse the repository at this point in the history

  4. repr bug (EleutherAI#2315)

    @baberabb @shachardon
    baberabb authored and shachardon committed Oct 1, 2024
    Configuration menu
    Copy the full SHA
    390ef6f View commit details
    Browse the repository at this point in the history

  5. Update neuron backend (EleutherAI#2314) 

    * feat(neuron): align with latest optimum-neuron

* feat(neuron): support pre-exported neuron models

* fix(neuron): correctly use max_length

* fix(neuron): adapt loglikelihood

The evaluation of log likelihood was not working for neuron models
using continuous batching, such as all cached neuron LLama models.

* refactor(neuron): remove dead code
    @dacorvo @shachardon
    dacorvo authored and shachardon committed Oct 1, 2024
    Configuration menu
    Copy the full SHA
    0e963c9 View commit details
    Browse the repository at this point in the history

  6. add version metadata to configs

    @shachardon
    shachardon committed Oct 1, 2024
    Configuration menu
    Copy the full SHA
    c06d2b2 View commit details
    Browse the repository at this point in the history

  7. update template list names

    @shachardon
    shachardon committed Oct 1, 2024
    Configuration menu
    Copy the full SHA
    66c058b View commit details
    Browse the repository at this point in the history

  8. change naming convention to lowercase

    @shachardon
    shachardon committed Oct 1, 2024
    Configuration menu
    Copy the full SHA
    6bf4f3b View commit details
    Browse the repository at this point in the history

  9. use recipes instead of cards + add descriptions to the README

    @shachardon
    shachardon committed Oct 1, 2024
    Configuration menu
    Copy the full SHA
    8aec6e1 View commit details
    Browse the repository at this point in the history

  10. add benchmarks results

    @shachardon
    shachardon committed Oct 1, 2024
    Configuration menu
    Copy the full SHA
    7844627 View commit details
    Browse the repository at this point in the history

  11. fix model names

    @shachardon
    shachardon committed Oct 1, 2024
    Configuration menu
    Copy the full SHA
    a5a70ef View commit details
    Browse the repository at this point in the history

  12. update readme

    @shachardon
    shachardon committed Oct 1, 2024
    Configuration menu
    Copy the full SHA
    98066cb View commit details
    Browse the repository at this point in the history

  13. transpose results table

    @shachardon
    shachardon committed Oct 1, 2024
    Configuration menu
    Copy the full SHA
    9bbe5ef View commit details
    Browse the repository at this point in the history

  14. another change to the results table

    @shachardon
    shachardon committed Oct 1, 2024
    Configuration menu
    Copy the full SHA
    33f8a6e View commit details
    Browse the repository at this point in the history

  15. Add unitxt to the README

    @perlitz @shachardon
    perlitz authored and shachardon committed Oct 1, 2024
    Configuration menu
    Copy the full SHA
    6523683 View commit details
    Browse the repository at this point in the history

  16. fill cehcklist

    @perlitz @shachardon
    perlitz authored and shachardon committed Oct 1, 2024
    Configuration menu
    Copy the full SHA
    e28919c View commit details
    Browse the repository at this point in the history

  17. Update README.md

    @perlitz @shachardon
    perlitz authored and shachardon committed Oct 1, 2024
    Configuration menu
    Copy the full SHA
    9d8b795 View commit details
    Browse the repository at this point in the history

  18. finalize checklist

    @perlitz @shachardon
    perlitz authored and shachardon committed Oct 1, 2024
    Configuration menu
    Copy the full SHA
    d52e5dc View commit details
    Browse the repository at this point in the history

  19. fix typo

    @perlitz @shachardon
    perlitz authored and shachardon committed Oct 1, 2024
    Configuration menu
    Copy the full SHA
    980ec2d View commit details
    Browse the repository at this point in the history

  20. scenarios -> groups

    @perlitz @shachardon
    perlitz authored and shachardon committed Oct 1, 2024
    Configuration menu
    Copy the full SHA
    c2ca175 View commit details
    Browse the repository at this point in the history

  21. fix type

    @perlitz @shachardon
    perlitz authored and shachardon committed Oct 1, 2024
    Configuration menu
    Copy the full SHA
    76f2c68 View commit details
    Browse the repository at this point in the history

  22. fix typos

    @perlitz @shachardon
    perlitz authored and shachardon committed Oct 1, 2024
    Configuration menu
    Copy the full SHA
    0385c13 View commit details
    Browse the repository at this point in the history

  23. clarify checklist

    @perlitz @shachardon
    perlitz authored and shachardon committed Oct 1, 2024
    Configuration menu
    Copy the full SHA
    ffcd078 View commit details
    Browse the repository at this point in the history

  24. change cards links to recipes links

    @shachardon
    shachardon committed Oct 1, 2024
    Configuration menu
    Copy the full SHA
    3cfacd9 View commit details
    Browse the repository at this point in the history

  25. Fixed dummy model (EleutherAI#2339)

    @Am1n3e @shachardon
    Am1n3e authored and shachardon committed Oct 1, 2024
    Configuration menu
    Copy the full SHA
    46dc531 View commit details
    Browse the repository at this point in the history

  26. add a note for missing dependencies (EleutherAI#2336)

    @eldarkurtic @shachardon
    eldarkurtic authored and shachardon committed Oct 1, 2024
    Configuration menu
    Copy the full SHA
    4b8e739 View commit details
    Browse the repository at this point in the history

  27. load metric with evaluate (EleutherAI#2351)

    @baberabb @shachardon
    baberabb authored and shachardon committed Oct 1, 2024
    Configuration menu
    Copy the full SHA
    4227dbc View commit details
    Browse the repository at this point in the history

  28. fix writeout script (EleutherAI#2350)

    @baberabb @shachardon
    baberabb authored and shachardon committed Oct 1, 2024
    Configuration menu
    Copy the full SHA
    f1a9ac8 View commit details
    Browse the repository at this point in the history

  29. Treat tags in python tasks the same as yaml tasks (EleutherAI#2288) 

    * Treat python tasks same as yaml tasks.

* Add tests.

* Re-add fixture decorators.

* Fix typing specification error for Python 3.9.
    @giuliolovisotto @shachardon
    giuliolovisotto authored and shachardon committed Oct 1, 2024
    Configuration menu
    Copy the full SHA
    b4eb477 View commit details
    Browse the repository at this point in the history

  30. change group to tags in task eus_exams task configs (EleutherAI#2320)

    @baberabb @shachardon
    baberabb authored and shachardon committed Oct 1, 2024
    Configuration menu
    Copy the full SHA
    2939e2f View commit details
    Browse the repository at this point in the history

  31. change glianorex to test split (EleutherAI#2332) 

    * change glianorex to test set

* nit

* fix test; doc_to_target can be str for multiple_choice

* nit
    @baberabb @shachardon
    baberabb authored and shachardon committed Oct 1, 2024
    Configuration menu
    Copy the full SHA
    ce1e8e9 View commit details
    Browse the repository at this point in the history

  32. mmlu-pro: add newlines to task descriptions (not leaderboard) (Eleuth… 

    …erAI#2334)

* add newlines to task descriptions; increment versions

* fix task tests (with groups)

* Apply suggestions from code review

---------

Co-authored-by: Hailey Schoelkopf <[email protected]>
    @baberabb @haileyschoelkopf @shachardon
    2 people authored and shachardon committed Oct 1, 2024
    Configuration menu
    Copy the full SHA
    1036191 View commit details
    Browse the repository at this point in the history

  33. Added TurkishMMLU to LM Evaluation Harness (EleutherAI#2283) 

    * Added TurkishMMLU to LM Evaluation Harness

* Fixed COT name

* Fixed COT name

* Updated Readme

* Fixed Test issues

* Completed  Scan for changed tasks

* Updated Readme

* Update README.md

* fixup task naming casing + ensure yaml template stubs aren't registered

---------

Co-authored-by: Hailey Schoelkopf <[email protected]>
Co-authored-by: haileyschoelkopf <[email protected]>
    @ArdaYueksel @haileyschoelkopf @shachardon
    3 people authored and shachardon committed Oct 1, 2024
    Configuration menu
    Copy the full SHA
    79b506c View commit details
    Browse the repository at this point in the history

  34. add mmlu readme (EleutherAI#2282)

    @baberabb @shachardon
    baberabb authored and shachardon committed Oct 1, 2024
    Configuration menu
    Copy the full SHA
    5e3fc76 View commit details
    Browse the repository at this point in the history

  35. openai: better error messages; fix greedy matching (EleutherAI#2327) 

    * better error message; fix greedy matching

* Update lm_eval/models/openai_completions.py

Co-authored-by: Hailey Schoelkopf <[email protected]>

* Update lm_eval/models/openai_completions.py

Co-authored-by: Hailey Schoelkopf <[email protected]>

* pre-commit

---------

Co-authored-by: Hailey Schoelkopf <[email protected]>
    @baberabb @haileyschoelkopf @shachardon
    2 people authored and shachardon committed Oct 1, 2024
    Configuration menu
    Copy the full SHA
    820e1a4 View commit details
    Browse the repository at this point in the history

  36. fix some bugs of mmlu (EleutherAI#2299) 

    * fix some bugs of mmlu

* Fix end of file newline issue

---------

Co-authored-by: eyuansu62 <[email protected]>
    @eyuansu62 @shachardon
    2 people authored and shachardon committed Oct 1, 2024
    Configuration menu
    Copy the full SHA
    94e8b79 View commit details
    Browse the repository at this point in the history

  37. Add new benchmark: Portuguese bench (EleutherAI#2156) 

    * Add portuguese_bench

* Add flores_pt group

* Update _flores_common_yaml

* Run linters and update flores and readme
    @zxcvuser @shachardon
    zxcvuser authored and shachardon committed Oct 1, 2024
    Configuration menu
    Copy the full SHA
    b7b6169 View commit details
    Browse the repository at this point in the history

  38. Fix missing key in custom task loading. (EleutherAI#2304)

    @giuliolovisotto @shachardon
    giuliolovisotto authored and shachardon committed Oct 1, 2024
    Configuration menu
    Copy the full SHA
    dca2cb1 View commit details
    Browse the repository at this point in the history

  39. use the unitxt HF implementation

    @shachardon
    shachardon committed Oct 1, 2024
    Configuration menu
    Copy the full SHA
    2b6a6e4 View commit details
    Browse the repository at this point in the history

  40. remove file used to format the yaml files

    @shachardon
    shachardon committed Oct 1, 2024
    Configuration menu
    Copy the full SHA
    da0a3ae View commit details
    Browse the repository at this point in the history

Commits on Oct 2, 2024

  1. uncomment bluebench_translation_mt_flores_101_eng_kor

    @shachardon
    shachardon committed Oct 2, 2024
    Configuration menu
    Copy the full SHA
    4461ff1 View commit details
    Browse the repository at this point in the history

Commits on Oct 7, 2024

  1. Merge branch 'main' into bluebench_pr

    @shachardon
    shachardon authored Oct 7, 2024
    Configuration menu
    Copy the full SHA
    0d1eeb2 View commit details
    Browse the repository at this point in the history

Commits on Oct 23, 2024

  1. Merge branch 'EleutherAI:main' into bluebench_pr

    @perlitz
    perlitz authored Oct 23, 2024
    Configuration menu
    Copy the full SHA
    da73a22 View commit details
    Browse the repository at this point in the history

  2. accomodate for args_dict 

    Signed-off-by: Yotam-Perlitz <[email protected]>
    @perlitz
    perlitz committed Oct 23, 2024
    Configuration menu
    Copy the full SHA
    b2c489e View commit details
    Browse the repository at this point in the history

  3. Align to format 

    Signed-off-by: Yotam-Perlitz <[email protected]>
    @perlitz
    perlitz committed Oct 23, 2024
    Configuration menu
    Copy the full SHA
    8447856 View commit details
    Browse the repository at this point in the history

  4. Align to formatting 

    Signed-off-by: Yotam-Perlitz <[email protected]>
    @perlitz
    perlitz committed Oct 23, 2024
    Configuration menu
    Copy the full SHA
    b6ad7b2 View commit details
    Browse the repository at this point in the history

  5. add blank libe 

    Signed-off-by: Yotam-Perlitz <[email protected]>
    @perlitz
    perlitz committed Oct 23, 2024
    Configuration menu
    Copy the full SHA
    57235ff View commit details
    Browse the repository at this point in the history

  6. Add trailing blank line 

    Signed-off-by: Yotam-Perlitz <[email protected]>
    @perlitz
    perlitz committed Oct 23, 2024
    Configuration menu
    Copy the full SHA
    55fb5c0 View commit details
    Browse the repository at this point in the history