Float zero division error while doing transfer learning on custom data

[JRMeyer]

>>> javi.rahman
[August 18, 2019, 6:04pm]

Hi Deepspeech,

16.04 LTS
NO

Here are the command and output of the error

deepspeech-venv) javi slash thinkpad-e470: slash ~/speech/tools/DeepSpeechWorking slash $
bin/run-alfred.sh

/home/javi/speech/tools/backup/train/train.csv slash --dev_files
/home/javi/speech/tools/backup/train/dev.csv slash --test_files
/home/javi/speech/tools/backup/train/test.csv slash --train_batch_size 80
slash --dev_batch_size 80 slash --test_batch_size 40 slash --n_hidden 375 slash --epoch
33 slash --early_stop True slash --earlystop_nsteps 6 slash --estop_mean_thresh 0.1
slash --estop_std_thresh 0.1 slash --dropout_rate 0.22 slash --learning_rate
0.00095 slash --report_count 100 slash --use_seq_length False slash --export_dir
/home/javi/speech/tools/backup/export_modal/ slash --checkpoint_dir
/home/javi/speech/tools/backup/checkout/ slash --alphabet_config_path
/home/javi/speech/tools/backup/DeepSpeech/data/alphabet.txt
slash --lm_binary_path
/home/javi/speech/tools/backup/DeepSpeech/data/lm.binary slash
WARNING:tensorflow:From
/home/javi/tmp/deepspeech-venv/lib/python3.6/site-packages/tensorflow/python/data/ops/dataset_ops.py:429:
py_func (from tensorflow.python.ops.script_ops) is deprecated and
will be removed in a future version. slash
Instructions for updating: slash
tf.py_func is deprecated in TF V2. Instead, use slash
tf.py_function, which takes a python function which manipulates tf
eager slash
tensors instead of numpy arrays. It's easy to convert a tf eager
tensor to slash
an ndarray (just call tensor.numpy()) but having access to eager
tensors slash
means tf.py_functions can use accelerators such as GPUs as well
as slash
being differentiable using a gradient tape.

WARNING:tensorflow:From
/home/javi/tmp/deepspeech-venv/lib/python3.6/site-packages/tensorflow/python/data/ops/iterator_ops.py:358:
colocate_with (from tensorflow.python.framework.ops) is deprecated and
will be removed in a future version. slash
Instructions for updating: slash
Colocations handled automatically by placer. slash
WARNING:tensorflow:From
/home/javi/tmp/deepspeech-venv/lib/python3.6/site-packages/tensorflow/contrib/rnn/python/ops/lstm_ops.py:696:
to_int64 (from tensorflow.python.ops.math_ops) is deprecated and will be
removed in a future version. slash
Instructions for updating: slash
Use tf.cast instead. slash
WARNING:tensorflow:From
/home/javi/tmp/deepspeech-venv/lib/python3.6/site-packages/tensorflow/python/training/saver.py:1266:
checkpoint_exists (from
tensorflow.python.training.checkpoint_management) is deprecated and will
be removed in a future version. slash
Instructions for updating: slash
Use standard file APIs to check for files with this prefix. slash
I Restored variables from most recent checkpoint at
/home/javi/speech/tools/backup/checkout/train-0, step 0 slash
I STARTING Optimization slash
Epoch 0 slash | Training slash | Elapsed Time: 0:00:00 slash | Steps: 0 slash | Loss:
0.000000 slash
Epoch 0 slash | Validation slash | Elapsed Time: 0:00:00 slash | Steps: 0 slash | Loss:
0.000000 slash | Dataset: /home/javi/speech/tools/backup/train/dev.csv slash
Traceback (most recent call last): slash
File 'DeepSpeech.py', line 836, in slash
tf.app.run(main) slash
File
'/home/javi/tmp/deepspeech-venv/lib/python3.6/site-packages/tensorflow/python/platform/app.py',
line 125, in run slash
slash _sys.exit(main(argv)) slash
File 'DeepSpeech.py', line 820, in main slash
train() slash
File 'DeepSpeech.py', line 524, in train slash
dev_loss = dev_loss / total_steps slash
ZeroDivisionError: float division by zero

[This is an archived TTS discussion thread from discourse.mozilla.org/t/float-zero-division-error-while-doing-transfer-learning-on-custom-data]

[JRMeyer]

>>> lissyx
[August 18, 2019, 6:59pm]



> ZeroDivisionError: float division by zero

That's usually a hint of a problem in your dataset.

[Archived Post]