How to generate exemplars combination after training.

[LeonChengg]

After executing the train_example_selection.py script, I obtained the checkpoint files for the trained model. However, I am unsure how to generate and output the exemplars combination using this model. Could you provide guidance on the necessary steps?

[SHUMKASHUN]

Thanks for the interest in our work. Basically after training the model, you can check the self.sample_probs which contains the probability distribution for each examplar entry (i.e. 8 examplars for gsm8k). Then you take the argmax (just as the code in validation_step), you will get the index for exemplars combination.

[LeonChengg]

I ran the training for 10 epochs, but unfortunately sadly, it still hasn’t converged, and the validation accuracy remains at 0.3. 😞

[SHUMKASHUN]

For most experiments we did with code-davinci-002 and text-davinci, they normally converge (or doesn't change much) at 4,5 epoches. Maybe you need to check if the variance-reduced algorithm works properly, i.e., produce meaningful loss. Because I remember for 10 runs (1 iteration update) need to have different accuracy.