Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add eval for correct_forking_fraction #167

Open
valedan opened this issue Apr 12, 2023 · 1 comment
Open

Add eval for correct_forking_fraction #167

valedan opened this issue Apr 12, 2023 · 1 comment
Assignees
Labels
evals Model evaluations

Comments

@valedan
Copy link
Contributor

valedan commented Apr 12, 2023

What percentage of the time does the model take the correct path at a fork. Some complexity here - we are uncertain how best to deal with forks occuring after the model has already deviated from the solution. There's a version of this in alex's experiments folder to use as inspiration.

@traeuker
Copy link
Member

@afspies you want to simply add your jazz somewhere or should we descope this?

@mivanit mivanit added the evals Model evaluations label Sep 4, 2023
@mivanit mivanit added this to the Fixing evals & training loop milestone Dec 27, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
evals Model evaluations
Projects
None yet
Development

No branches or pull requests

4 participants