Question related to how to use the validation and training splits.

[sorobedio]

Hello, I would like to know how I can use the validation split to evaluate the models, and similarly, how to use the training split for evaluation if needed. I haven't found an option where the user can specify the dataset split they want to use for model evaluation. Could you provide guidance on how to set this up?
thank you

[baberabb]

Hi! You can switch up the sets used in the task yamls. We use the test split if provided, otherwise the validation split is used. Example:

lm-evaluation-harness/lm_eval/tasks/arc/arc_easy.yaml

Lines 7 to 9 in 1185e89

	training_split: train
	validation_split: validation
	test_split: test

If a fewshot_split is not provided, then the priority is training > val > test to extract the fewshot examples.

[sorobedio]

I see. So the validation set is never used when both the training and test sets are present. Is the Open LM leaderboard following the same approach?
Thank you.

[baberabb]

I see. So the validation set is never used when both the training and test sets are present. Is the Open LM leaderboard following the same approach? Thank you.

yes!