Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add LAMA to Full Benchmark #26

Open
epavlick opened this issue Aug 10, 2021 · 6 comments
Open

Add LAMA to Full Benchmark #26

epavlick opened this issue Aug 10, 2021 · 6 comments
Assignees
@JanKalo
Labels
In-Progress linguistic_structure Benchmark Tasks for CoreNLP/linguistic structure prediction

Comments

@epavlick
Copy link
Contributor

No description provided.

@epavlick epavlick added the linguistic_structure Benchmark Tasks for CoreNLP/linguistic structure prediction label Aug 10, 2021
@epavlick epavlick changed the title Add LAMDA to Full Benchmark Add LAMA to Full Benchmark Aug 10, 2021
@JanKalo
Copy link
Contributor

JanKalo commented Aug 10, 2021

I can take care of this.

@JanKalo
Copy link
Contributor

JanKalo commented Sep 24, 2021

LAMA consists of 4 different subtasks: trex, conceptnet, squad, and google re.
Should I create 4 separate tasks, similar to TidyQA primary/secondary?

@wilsonyhlee
Copy link
Member

LAMA consists of 4 different subtasks: trex, conceptnet, squad, and google re.
Should I create 4 separate tasks, similar to TidyQA primary/secondary?

Please do!

@KhalidAlt
Copy link

I can help with this. I already wrote prompts for trex. It is on promptsource repository. We can complete the prompts for the reset as first step.

@JanKalo
Copy link
Contributor

JanKalo commented Apr 26, 2022

I would also continue working on this.
I think we should have a look on how to transform the MASK task into appropriate prompts to make them comparable to previous evaluations.

@jzf2101
Copy link
Collaborator

jzf2101 commented Apr 27, 2022 •
edited
Loading

See bigscience-workshop/promptsource#737 and bigscience-workshop/promptsource#741

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@JanKalo JanKalo

Labels
In-Progress linguistic_structure Benchmark Tasks for CoreNLP/linguistic structure prediction
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@epavlick @jzf2101 @wilsonyhlee @JanKalo @KhalidAlt