Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

mmlu translated professionally by OpenAI #2312

Open
wants to merge 25 commits into
base: main
Choose a base branch
from

Conversation

giuliolovisotto
Copy link
Contributor

@giuliolovisotto giuliolovisotto commented Sep 17, 2024

Following up from #2305

@baberabb

  • I grouped by language in subfolders
  • I created a group per language
  • I mostly copied the existing mmlu/default so I dropped some options to do cot/generative/fewshots.

I'll try to run something like llama 70b in the next days to make sure numbers make sense.

@giuliolovisotto giuliolovisotto changed the title Draft: mmlu but it is translated professionally by OpenAI Draft: mmlu translated professionally by OpenAI Sep 24, 2024
SYusupov and others added 19 commits September 27, 2024 16:58
* Update README.md

I encounter some Git buffer size limits when trying to download all commits history of the repository, such as:
```error: RPC failed; curl 18 transfer closed with outstanding read data remaining
error: 5815 bytes of body are still expected
fetch-pack: unexpected disconnect while reading sideband packet
fatal: early EOF```

therefore the installation is faster and there are not errors when I download only the last version of the repository

* Fix linting issue
* feat(neuron): align with latest optimum-neuron

* feat(neuron): support pre-exported neuron models

* fix(neuron): correctly use max_length

* fix(neuron): adapt loglikelihood

The evaluation of log likelihood was not working for neuron models
using continuous batching, such as all cached neuron LLama models.

* refactor(neuron): remove dead code
* Treat python tasks same as yaml tasks.

* Add tests.

* Re-add fixture decorators.

* Fix typing specification error for Python 3.9.
* change glianorex to test set

* nit

* fix test; doc_to_target can be str for multiple_choice

* nit
…erAI#2334)

* add newlines to task descriptions; increment versions

* fix task tests (with groups)

* Apply suggestions from code review

---------

Co-authored-by: Hailey Schoelkopf <[email protected]>
* Added TurkishMMLU to LM Evaluation Harness

* Fixed COT name

* Fixed COT name

* Updated Readme

* Fixed Test issues

* Completed  Scan for changed tasks

* Updated Readme

* Update README.md

* fixup task naming casing + ensure yaml template stubs aren't registered

---------

Co-authored-by: Hailey Schoelkopf <[email protected]>
Co-authored-by: haileyschoelkopf <[email protected]>
* better error message; fix greedy matching

* Update lm_eval/models/openai_completions.py

Co-authored-by: Hailey Schoelkopf <[email protected]>

* Update lm_eval/models/openai_completions.py

Co-authored-by: Hailey Schoelkopf <[email protected]>

* pre-commit

---------

Co-authored-by: Hailey Schoelkopf <[email protected]>
@giuliolovisotto giuliolovisotto changed the title Draft: mmlu translated professionally by OpenAI mmlu translated professionally by OpenAI Sep 27, 2024
}


def normalize_extracted_answer(extracted_answer: str) -> str:

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi giuliolovisotto,
Thank you for your efforts. You've greatly saved me a lot of time.
I have a question regarding this line.
It seems that this function and QUERY_TEMPLATE_MULTICHOICE are not used anywhere, so I believe they can be removed.
Is that correct?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

8 participants