Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add secs tests equalization #152

Merged
merged 1 commit into from
Jan 14, 2025

Conversation

1DWalker
Copy link
Contributor

Adds --equalize_test_with_non_secs flag to test secs models on the same reviews as non-secs models.
See: #149.

Model: FSRS-5-secs-equalize_test_with_non_secs
Total number of users: 100
Total number of reviews: 2097825
Weighted average by reviews:
FSRS-5-secs-equalize_test_with_non_secs LogLoss (mean±std): 0.3841±0.1827
FSRS-5-secs-equalize_test_with_non_secs RMSE(bins) (mean±std): 0.0793±0.0377
FSRS-5-secs-equalize_test_with_non_secs AUC (mean±std): 0.6981±0.0752

Model: FSRS-5
Total number of users: 100
Total number of reviews: 2097825
Weighted average by reviews:
FSRS-5 LogLoss (mean±std): 0.3761±0.1779
FSRS-5 RMSE(bins) (mean±std): 0.0687±0.0339
FSRS-5 AUC (mean±std): 0.7080±0.0744

Copy link
Member

@L-M-Sherlock L-M-Sherlock left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@L-M-Sherlock L-M-Sherlock merged commit e7d578a into open-spaced-repetition:main Jan 14, 2025
@Expertium
Copy link
Contributor

@L-M-Sherlock this is good, but I still need a way to not calculate the loss on same-day reviews, neither in training nor in evaluation. So I want you you to add a flag for that.

@L-M-Sherlock
Copy link
Member

@1DWalker would you mind developing that?

@Expertium
Copy link
Contributor

Is it that difficult? I mean, it's just FSRS-5 with fractional intervals. Take FSRS-5 code (including the optimizer code), change the intervals and that's all.

@L-M-Sherlock
Copy link
Member

I'm working on other things.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants