Re-make the workload-retention graph #686
Comments
|
I generate a new one with FSRS-5:
|
|
Huh. That's strange, here the workload above 95% is much lower. |
|
Do we replace the graph in manual? Actually, I think we should smoothen the lines a bit. Less accuracy but looks better when it's a beautiful curved line. |
|
@L-M-Sherlock sorry for bothering you, but would you please generate it with more precision? Instead of incrementing desired retention by 1%, increment it by 0.2%. And increase the sample size for the simulation. |
|
OK. Here you are:
|
|
I wanted to give you some more suggestions, but I think it's better if you just give me the raw data and I plot it myself |
|
I didn't save the raw data. You can plot this image via this commend in python ./src/fsrs_optimizer/fsrs_simulator.py |
|
Please save it and send it to me |
|
|
This was much harder than I thought. I had to do a whole bunch of math + crude hacks. I am NOT sharing the code for this abomination. Anyway, it seems that it is, indeed, quite different from the last one. The workload doesn't increase as steeply as we previously thought, and the optimal value is around 0.85 rather than 0.75. This suggests that previous estimates of optimal retention were underestimates. |
Do you have any intuitive explanation of why it might be so different? (If we can't think of any logical reason, I would tend to believe that this shift results from a bug or a bias.) |
It's caused by the short-term memory module. FSRS-5 allows the stability to increase when the retrievability is 100%. |
|
And also the fact that the simulation accounts for the time it takes to do a same-day review. |
|
Ok, then it looks good to me. |
|
But, it would be better if the red region is also shown at the right side of the graph. Based on a quick glance at the data, it seems that the red region will start from DR = 0.993 |
|
I don't think there is a point in doing that, since in Anki you can't set DR above 0.99, not even a little bit. |
|
Can we make the word _retention_ in graph _desired retention_? It can be
used in the manual in the topic titled "Desired Retention". Expertium,
please?
|
|
LMSherlock already merged it in the guide. Man...sigh fine. |
Fair point. By the way, I am still slightly skeptical about the new graph. According to this graph, 0.97 is in the green zone. But, if I use the FSRS previewer with the default FSRS 5 parameters and DR = 0.97, I get this result: rating history: 3,3,3,3,3,3,3,3,3,3,3 Aren't the intervals too short? Should we advise users to set such a high value of DR?
Good suggestion. |
|
Colors are kinda arbitrary. On this graph, green is just "between min. workload and 2*min. workload". |
|
Do you mean to say that the definition of the green, yellow and red zones should be updated? |
|
I mean to say that those definitions are arbitrary. I'm not sure if they need to be updated, though. |
|
Can we do a gradient transition instead of the solid blocks of colour? |
|
I think that the yellow zone should start where the slope of the graph increases drastically. I am not sure of the best way to find that point. But, on doing a crude analysis with Excel, I think that DR = 0.96 should be the transition point.
|
Oh god, that would be a nightmare.
Let's just say it starts at min. workload x1.5. |
|
You know, I think most people probably don't get these colours. I am aware that I am a sample of one, but I really had no idea what these were signifying, particularly the yellow colour that is shoehorned between other colours. It's a bit hard to make head and tails of that. |
|
I'd rather we discuss whether changing the output range of CMRR is a good idea instead of discussing the color of the |
|
But maybe we should add this, too? |
|
Why is this graph different? It starts slanting from a different point. |
|
Read what the Y axis says, and what the text above the black horizontal lines says 😉 |
|
😮 cool. CMRR value seems much higher than expected. |
|
It's cool, indeed. But I'm somewhat worried that users will be confused by two different graphs. |
|
That? Majority of people will be confused from workload:retention being in x-axis. Having two variables together like this complicates it (I remember I used to find them so hard). |
|
@L-M-Sherlock the graphs here also look like they need to be updated: https://github.com/open-spaced-repetition/fsrs4anki/wiki/The-Optimal-Retention |
It's unnecessary to divide the workload by retention because the code has taken it into account. |
|
Wait, I'm confused. So the original graph in this comment doesn't show workload, but rather workload divided by knowledge? |
|
Because I use the same function used by CMRR. def workload_graph(default_params):
R = [x / 100 for x in range(70, 100)]
cost_per_memorization = [sample(r=r, **default_params) for r in R]
...
def sample(
r,
w,
deck_size=10000,
learn_span=365,
max_cost_perday=1800,
learn_limit_perday=math.inf,
review_limit_perday=math.inf,
max_ivl=36500,
learn_costs=DEFAULT_LEARN_COSTS,
review_costs=DEFAULT_REVIEW_COSTS,
first_rating_prob=DEFAULT_FIRST_RATING_PROB,
review_rating_prob=DEFAULT_REVIEW_RATING_PROB,
first_rating_offset=DEFAULT_FIRST_RATING_OFFSETS,
first_session_len=DEFAULT_FIRST_SESSION_LENS,
forget_rating_offset=DEFAULT_FORGET_RATING_OFFSET,
forget_session_len=DEFAULT_FORGET_SESSION_LEN,
loss_aversion=2.5,
):
memorization = []
if learn_span < 100:
SAMPLE_SIZE = 16
elif learn_span < 365:
SAMPLE_SIZE = 8
else:
SAMPLE_SIZE = 4
for i in range(SAMPLE_SIZE):
_, _, _, memorized_cnt_per_day, cost_per_day = simulate(
w,
r,
deck_size,
learn_span,
max_cost_perday,
learn_limit_perday,
review_limit_perday,
max_ivl,
learn_costs,
review_costs,
first_rating_prob,
review_rating_prob,
first_rating_offset,
first_session_len,
forget_rating_offset,
forget_session_len,
loss_aversion,
seed=42 + i,
)
memorization.append(cost_per_day.sum() / memorized_cnt_per_day[-1])
return np.mean(memorization) |
|
Is cost_per_day just time, or time/knowledge? That's important. |
|
|
|
So this graph is just workload, right? |
Nope. All of previous graph generated by the optimizer is cost/memorization. If you need the workload graph, I will refactor the function. |
|
We will need it for ankitects/anki-manual#263 On a different note, can we explain people the significance of the current graph in simple words? |
Yes, we need workload graph for the tutorial. Also, if the current graph plots cost/memorization, its labels (y axis and the dashed horizontal lines) should be corrected. |
|
@L-M-Sherlock Yep, we need just workload |
|
|
|
Add this right before plt.show(): |
|
Could you please make the upward slope a bit more smoother? Also make retention in graph desired retention? |
|
|
@Expertium, if you has some suggestions to the plotting of this graph, feel free to share your code in open-spaced-repetition/fsrs-optimizer#135 |
|
Well, something is wrong here. 1546464.3 min = 1073.9 days So, the values can't be minutes of study per day (as the label of the graph suggests). |
|
|
This makes sense. But, to avoid confusion, I think that the function should output values in the same unit as the label on the graph. That is, divide the workload by 60 * 365 |
|
|
This graph was generated before FSRS-5, when same-day reviews weren't accounted for. I believe that it needs to be re-made, since same-day reviews likely increase the workload and therefore affect the graph.
The text was updated successfully, but these errors were encountered: