You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
creating an issue to track progress on optional BOS prefix in all windows for llh_rolling methods. This is useful for models such as Gemma2 family.
Based on original issue reported in discord
I investigated some strange numbers on language modeling tasks (evaluated with ppl), and I assume BOS is added just once (not in subsequent frames of loglikelihood_rolling, when document is windowized). Is this the case? And shouldn't BOS be added for gemma everytime?
I stopped model call randomly in debugger and verified its inputs to see, that BOS is actually not there in llh_rolling everytime
The text was updated successfully, but these errors were encountered:
Hi,
creating an issue to track progress on optional BOS prefix in all windows for llh_rolling methods. This is useful for models such as Gemma2 family.
Based on original issue reported in discord
The text was updated successfully, but these errors were encountered: