I need some clarification on ignore longer outputs than inputs flag

[JRMeyer]

>>> Sushantmkarande
[June 17, 2019, 4:45am]

[ slash kdavis](
I was training data I scraped
from youtube and its cc aka vtt aka subtitle as transcript on deepspeech
0.5.0 model when I get this error.

Not enough time for target transition sequence (required: 102, available: 0)0You can turn this error into a warning by using the flag ignore_longer_outputs_than_inputs

I gave ignore_longer_outputs_than_inputs=True this flag in
tf.nn.ctc_loss and model started training again but I need some
clarification on this.

what does it mean?..

why i get this error... it might be true that my transcript is not 100%
match to audio but I remember giving this model completely wrong
transcript and it still trained on it, slash
and how to know how many training sample its ignoring after giving this
flag. what if its skipping over all of the sample because I am not
seeing even slightest effect on model after training all day...

[This is an archived TTS discussion thread from discourse.mozilla.org/t/i-need-some-clarification-on-ignore-longer-outputs-than-inputs-flag]

[JRMeyer]

>>> lissyx
[June 18, 2019, 6:36pm]



> how to know how many training sample its ignoring after giving this
> flag.

So far there's no better solution than either filtering on min / max
length and / or do some binary search to find offending samples.

[Archived Post]

[JRMeyer]

>>> Sushantmkarande
[June 20, 2019, 5:31am]

how do i filter on min/max length. Sry I did not fully understand that.
![:roll_eyes:](
how do i find offending samples error do not specify anything about on
which sample it is stuck...

[Archived Post]

[JRMeyer]

>>> reuben
[June 20, 2019, 12:26pm]

You can look at the data directly. If the audio is too short for its
transcript, it won't work. Audio windows have a 20ms step between them,
so to get the number of windows from an audio file you can just divide
its duration by 20ms, and then compare that with the length of the
transcript.

[
[[SamahZaro
[August 12, 2019, 3:35am]

Good answer. However, the CTC loss calculation, as far as I know, adds
blank character '-' between repetitive characters of the transcript or
something like this... this will make comparing with the length of the
transcript just an indicator but not accurate.
, what do you think?

[Archived Post]

[JRMeyer]

>>> reuben
[August 12, 2019, 5:08am]

I don't think CTC blanks are relevant here.

[Archived Post]

[JRMeyer]

>>> agarwalaashish20
[September 14, 2019, 3:00pm]

,
: I am using Deep Speech
v0.5.0, and I am also encountering this error. I have set
ignore_longer_outputs_than_inputs=True

total_loss = tf.nn.ctc_loss(labels=batch_y, inputs=logits,
sequence_length=batch_seq_len, ignore_longer_outputs_than_inputs=True)

Now, when I run the training my Training Loss is always infinity. Kindly
guide, how to resolve it?

Epoch 0 slash | Training slash | Elapsed Time: 0:12:42 slash | Steps: 1142 slash | Loss:
inf slash
Epoch 0 slash | Validation slash | Elapsed Time: 0:01:39 slash | Steps: 163 slash | Loss:
146.396210 slash | Dataset: .../german-speech-corpus/data_mailabs/dev.csv slash
I Saved new best validating model with loss 146.396210 to:
/home/agarwal/.local/share/deepspeech/checkpoints/best_dev-1142 slash
Epoch 1 slash | Training slash | Elapsed Time: 0:12:32 slash | Steps: 1142 slash | Loss:
inf slash
Epoch 1 slash | Validation slash | Elapsed Time: 0:00:58 slash | Steps: 163 slash | Loss:
131.277453 slash | Dataset: .../german-speech-corpus/data_mailabs/dev.csv slash
WARNING:tensorflow:From
/home/agarwal/python-environments/env/lib/python3.5/site-packages/tensorflow/python/training/saver.py:966:
remove_checkpoint (from
tensorflow.python.training.checkpoint_management) is deprecated and will
be removed in a future version. slash
Instructions for updating: slash
Use standard file APIs to delete files with this prefix. slash
I Saved new best validating model with loss 131.277453 to:
/home/agarwal/.local/share/deepspeech/checkpoints/best_dev-2284 slash
Epoch 2 slash | Training slash | Elapsed Time: 0:12:33 slash | Steps: 1142 slash | Loss:
inf slash
Epoch 2 slash | Validation slash | Elapsed Time: 0:00:58 slash | Steps: 163 slash | Loss:
125.264005 slash | Dataset: .../german-speech-corpus/data_mailabs/dev.csv slash
I Saved new best validating model with loss 125.264005 to:
/home/agarwal/.local/share/deepspeech/checkpoints/best_dev-3426 slash
Epoch 3 slash | Training slash | Elapsed Time: 0:12:34 slash | Steps: 1142 slash | Loss:
inf slash
Epoch 3 slash | Validation slash | Elapsed Time: 0:00:58 slash | Steps: 163 slash | Loss:
128.504051 slash | Dataset: .../german-speech-corpus/data_mailabs/dev.csv slash
Epoch 4 slash | Training slash | Elapsed Time: 0:08:50 slash | Steps: 918 slash | Loss: inf slash
(env) agarwal slash wika: slash ~/DeepSpeech slash $

[Archived Post]

[JRMeyer]

>>> agarwalaashish20
[November 26, 2019, 4:17pm]

, could you please help on the
above issue. Even after setting the flag, it didn't work.

The training loss is inf and validation loss is decreasing. I am using
German-Mailabs dataset.

[
[[Andreea_Georgiana_Sarca
[June 17, 2020, 3:04pm]

Hello!

I started training the system with another dataset, the training worked
well (I allowed it to train for about 100 epochs to see if everything
works fine), but when it started the testing I also got the wonderful
error:

slash 'tensorflow.python.framework.errors_impl.InvalidArgumentError: Not
enough time for target transition sequence (required: 20, available:
17)1You can turn this error into a warning by using the flag
ignore_longer_outputs_than_inputs slash
slash [ slash [{{node CTCLoss}} slash ] slash ] slash ' slash
I need to mention that the flag ignore_longer_outputs_than_inputs=True
was already added to DeepSpeech.py / for the ctc_loss and I still get
the error ! ! !

Any ideas?

[Archived Post]

[JRMeyer]

>>> othiele
[June 17, 2020, 3:11pm]

I guess this could well be a new thread, but if I remember correctly,
this is a different line in the DeepSpeech.py script?

Anyway, this means you have an input that probably doesn't match the
transcript. Are you sure about your data? Especially for testing this
could be a problem.

[
[[Andreea_Georgiana_Sarca
[June 17, 2020, 3:25pm]

In DeepSpeech.py I modified and added the flag, because without it the
training wouldn't start:

# Compute the CTC loss using TensorFlow's ctc_loss

total_loss = tfv1.nn.ctc_loss(labels=batch_y, inputs=logits, sequence_length=batch_seq_len, ignore_longer_outputs_than_inputs=True)

As I mentioned before, the error occurs again when it starts to test and
it asks me again to set the flag that is already there.

Unfortunately I can't say that I am sure of my data, because it was
provided by another University that used it for other things (not speech
recognition). For testing there are around 27000 audio files and I
randomly checked some of them by listening if they match with the
transcript and I didn't notice any problem, probably somewhere in the
dataset it is a mistake

[Archived Post]

[JRMeyer]

>>> lissyx
[June 17, 2020, 3:26pm]

> I need to mention that the flag
> ignore_longer_outputs_than_inputs=True was already added to
> DeepSpeech.py / for the ctc_loss and I still get the error ! ! !
>
> Any ideas?

Please search on the forum, this is already extensively documented as a
data-level issue.

[Archived Post]

[JRMeyer]

>>> othiele
[June 17, 2020, 4:12pm]

Write a small script that checks transcription length vs. audio length
in the csv. Then check outliers manually.

[Archived Post]

[JRMeyer]

>>> carlfm01
[June 17, 2020, 6:06pm]

A long time ago this solved my issue:
mozilla/DeepSpeech#1629 (comment)

> audio files and I randomly checked some of them by listening if they
> match with the transcrip

What works for me to know the quality of the set is the usage of
ignore_longer_outputs_than_inputs on evaluate and the flag
test_output_file to generate a json file sorted using the loss, then I
use my own .NET app to listen to the worst examples from the generated
json.

[Archived Post]

[JRMeyer]

>>> carlfm01
[June 17, 2020, 6:08pm]

Never mind audiofile_to_input_vector is no longer a thing

[Archived Post]