Loading feeding data order effect

[JRMeyer]

>>> SamahZaro
[August 12, 2019, 9:51am]

I was checking the code. In util/feeding.py file, data is sorted in
ascending order by wav_filesize, that is shorter audios will be
processed first. I have a large data of audios with duration from 0.5
sec - 24 sec. How could the order of the data affect the training loss? slash
I am trying now a run with descending order, and already noticed the
loss is better... but maybe the initial random weights is the reason.

Ascending/train (2 epochs): slash

Descending/train (1 epoch): slash

What is the idea of the sorting anyway?

[This is an archived TTS discussion thread from discourse.mozilla.org/t/loading-feeding-data-order-effect]

[JRMeyer]

>>> yv001
[August 12, 2019, 1:20pm]

I think this is a type of curriculum learning - start on easier samples
first and continue with the difficult ones later in the training. This
type of training often leads to better results, e.g. see here
curriculum learning
paper

[Archived Post]

[JRMeyer]

>>> SamahZaro
[August 12, 2019, 2:49pm]

Good to know, thank you.

[Archived Post]

[JRMeyer]

>>> SamahZaro
[August 13, 2019, 5:10pm]



> Ascending/train (2 epochs): slash

I have this plot for training loss. What I noticed is that the loss is
decreasing in a good way until it starts processing large wavs by which
it starts increasing, see around 13k step. Then first epoch finished
around 25k and then the same is occurring in the second epoch. slash
Does that mean that long wavs are reducing the accuracy of my model? or
this is normal?

My audios ranges from 0.5 - 24 seconds.

, looking forward your
suggestions.

[Archived Post]

[JRMeyer]

>>> yv001
[August 21, 2019, 10:17am]

This looks correct to me, for longer wavs, the network has lower
accuracy. The important thing is that overall after each epoch the curve
ends up lower so the network is learning.

[Archived Post]