Automatically session get restarted in between the steps

[JRMeyer]

>>> vigneshgig
[August 31, 2019, 1:31pm]

Epoch 0 slash | Training slash | Elapsed Time: 1 day, 9:20:19 slash | Steps: 87858 slash |
Loss: 6.972850 Killed slash
WARNING: Logging before flag parsing goes to stderr. slash
W0831 04:18:35.381413 140353130383104 deprecation_wrapper.py:119 slash ] From
DeepSpeech_testing.py:853: The name tf.app.run is deprecated. Please use
tf.compat.v1.app.run instead.

W0831 04:18:37.063208 140353130383104 deprecation_wrapper.py:119 slash ] From
/home/administrator/deepspeech/DeepSpeech/DeepSpeech-0.5.1/util/config.py:60:
The name tf.ConfigProto is deprecated. Please use
tf.compat.v1.ConfigProto instead.

W0831 04:18:37.087497 140353130383104 deprecation_wrapper.py:119 slash ] From
DeepSpeech_testing.py:835: The name tf.reset_default_graph is
deprecated. Please use tf.compat.v1.reset_default_graph instead.

W0831 04:18:37.087680 140353130383104 deprecation_wrapper.py:119 slash ] From
DeepSpeech_testing.py:836: The name tf.set_random_seed is deprecated.
Please use tf.compat.v1.set_random_seed instead.

W0831 04:36:03.456215 140353130383104 deprecation.py:323 slash ] From
/usr/local/lib/python3.7/site-packages/tensorflow/python/data/ops/dataset_ops.py:494:
py_func (from tensorflow.python.ops.script_ops) is deprecated and will
be removed in a future version. slash
Instructions for updating: slash
tf.py_func is deprecated in TF V2. Instead, there are two slash
options available in V2. slash
tensors instead of numpy arrays. It's easy to convert a tf eager tensor
to slash
an ndarray (just call tensor.numpy()) but having access to eager
tensors slash
means tf.py_functions can use accelerators such as GPUs as well as slash
being differentiable using a gradient tape. slash
tf.py_func slash
(it is not differentiable, and manipulates numpy arrays). It drops the slash
stateful argument making all functions stateful.

W0831 04:36:03.543363 140353130383104 deprecation_wrapper.py:119 slash ] From
DeepSpeech_testing.py:383: The name tf.data.Iterator is deprecated.
Please use tf.compat.v1.data.Iterator instead.

W0831 04:36:03.543499 140353130383104 deprecation.py:323 slash ] From
DeepSpeech_testing.py:383: DatasetV1.output_types (from
tensorflow.python.data.ops.dataset_ops) is deprecated and will be
removed in a future version. slash
Instructions for updating: slash
Use tf.compat.v1.data.get_output_types(dataset). slash
W0831 04:36:03.543672 140353130383104 deprecation.py:323 slash ] From
DeepSpeech_testing.py:384: DatasetV1.output_shapes (from
tensorflow.python.data.ops.dataset_ops) is deprecated and will be
removed in a future version. slash
Instructions for updating: slash
Use tf.compat.v1.data.get_output_shapes(dataset). slash
W0831 04:36:03.543839 140353130383104 deprecation.py:323 slash ] From
DeepSpeech_testing.py:385: DatasetV1.output_classes (from
tensorflow.python.data.ops.dataset_ops) is deprecated and will be
removed in a future version. slash
Instructions for updating: slash
Use tf.compat.v1.data.get_output_classes(dataset). slash
W0831 04:36:03.545172 140353130383104 deprecation.py:323 slash ] From
/usr/local/lib/python3.7/site-packages/tensorflow/python/data/ops/iterator_ops.py:348:
Iterator.output_types (from tensorflow.python.data.ops.iterator_ops) is
deprecated and will be removed in a future version. slash
Instructions for updating: slash
Use tf.compat.v1.data.get_output_types(iterator). slash
W0831 04:36:03.545303 140353130383104 deprecation.py:323 slash ] From
/usr/local/lib/python3.7/site-packages/tensorflow/python/data/ops/iterator_ops.py:349:
Iterator.output_shapes (from tensorflow.python.data.ops.iterator_ops) is
deprecated and will be removed in a future version. slash
Instructions for updating: slash
Use tf.compat.v1.data.get_output_shapes(iterator). slash
W0831 04:36:03.545415 140353130383104 deprecation.py:323 slash ] From
/usr/local/lib/python3.7/site-packages/tensorflow/python/data/ops/iterator_ops.py:351:
Iterator.output_classes (from tensorflow.python.data.ops.iterator_ops)
is deprecated and will be removed in a future version. slash
Instructions for updating: slash
Use tf.compat.v1.data.get_output_classes(iterator). slash
W0831 04:36:03.546030 140353130383104 deprecation_wrapper.py:119 slash ] From
DeepSpeech_testing.py:396: The name tf.placeholder is deprecated. Please
use tf.compat.v1.placeholder instead.

W0831 04:36:03.548312 140353130383104 deprecation_wrapper.py:119 slash ] From
DeepSpeech_testing.py:215: The name tf.train.AdamOptimizer is
deprecated. Please use tf.compat.v1.train.AdamOptimizer instead.

W0831 04:36:03.548471 140353130383104 deprecation_wrapper.py:119 slash ] From
DeepSpeech_testing.py:251: The name tf.variable_scope is deprecated.
Please use tf.compat.v1.variable_scope instead.

W0831 04:36:03.631734 140353130383104 deprecation.py:506 slash ] From
/usr/local/lib/python3.7/site-packages/tensorflow/python/ops/init_ops.py:1251:
calling VarianceScaling.init (from tensorflow.python.ops.init_ops)
with dtype is deprecated and will be removed in a future version. slash
Instructions for updating: slash
Call initializer instance with the dtype argument instead of passing it
to the constructor slash
W0831 04:36:04.187909 140353130383104 deprecation_wrapper.py:119 slash ] From
DeepSpeech_testing.py:190: The name tf.nn.ctc_loss is deprecated. Please
use tf.compat.v1.nn.ctc_loss instead.

W0831 04:36:04.189303 140353130383104 deprecation.py:323 slash ] From
DeepSpeech_testing.py:194: add_dispatch_support..wrapper (from
tensorflow.python.ops.array_ops) is deprecated and will be removed in a
future version. slash
Instructions for updating: slash
Use tf.where in 2.0, which has the same broadcast rule as np.where slash
W0831 04:36:04.316739 140353130383104 deprecation_wrapper.py:119 slash ] From
DeepSpeech_testing.py:279: The name tf.summary.scalar is deprecated.
Please use tf.compat.v1.summary.scalar instead.

W0831 04:36:04.388463 140353130383104 deprecation_wrapper.py:119 slash ] From
DeepSpeech_testing.py:340: The name tf.summary.histogram is deprecated.
Please use tf.compat.v1.summary.histogram instead.

W0831 04:36:04.525462 140353130383104 deprecation_wrapper.py:119 slash ] From
DeepSpeech_testing.py:418: The name tf.train.get_or_create_global_step
is deprecated. Please use tf.compat.v1.train.get_or_create_global_step
instead.

W0831 04:36:04.632269 140353130383104 deprecation_wrapper.py:119 slash ] From
DeepSpeech_testing.py:422: The name tf.summary.merge_all is deprecated.
Please use tf.compat.v1.summary.merge_all instead.

W0831 04:36:04.632952 140353130383104 deprecation_wrapper.py:119 slash ] From
DeepSpeech_testing.py:424: The name tf.summary.FileWriter is deprecated.
Please use tf.compat.v1.summary.FileWriter instead.

W0831 04:36:04.649804 140353130383104 deprecation_wrapper.py:119 slash ] From
DeepSpeech_testing.py:429: The name tf.train.Saver is deprecated. Please
use tf.compat.v1.train.Saver instead.

W0831 04:36:04.757282 140353130383104 deprecation.py:323 slash ] From
/usr/local/lib/python3.7/site-packages/tensorflow/python/training/saver.py:1276:
checkpoint_exists (from
tensorflow.python.training.checkpoint_management) is deprecated and will
be removed in a future version. slash
Instructions for updating: slash
Use standard file APIs to check for files with this prefix. slash
I0831 04:36:04.759114 140353130383104 saver.py:1280 slash ] Restoring
parameters from ./deepspeech-0.5.1-checkpoint/train-555213 slash
W0831 04:36:17.228476 140353130383104 deprecation_wrapper.py:119 slash ] From
DeepSpeech_testing.py:365: The name tf.train.get_global_step is
deprecated. Please use tf.compat.v1.train.get_global_step instead.

I Restored variables from most recent checkpoint at
./deepspeech-0.5.1-checkpoint/train-555213, step 555214 slash
I STARTING Optimization slash
Epoch 0 slash | Training slash | Elapsed Time: 0:23:25 slash | Steps: 2044 slash | Loss:
9.845757

hi , the session get restart after the steps :87858 and again starting
the process from the begin but the total steps to be is 137000.

Epoch 0 slash | Training slash | Elapsed Time: 1 day, 9:20:19 slash | Steps: 87858 slash |
Loss: 6.972850 Killed slash
so it automatically killing the process

I have gtx 1050 4gb graphics card.

command : CUDA_VISIBLE_DEVICES=0 python3.7 DeepSpeech_testing.py
slash --lm_binary_path ./data/deepspeech_dataset/lm.binary slash --lm_trie_path
./data/deepspeech_dataset/trie slash --learning_rate 0.0001
slash --alphabet_config_path ./data/alphabet.txt slash --beam_width 1024
slash --use_seq_length false slash --remove_export true slash --notest
slash --train_batch_size 30 slash --train_files
./data/deepspeech_dataset/august_22_prune_dataset.csv slash --checkpoint_dir
./deepspeech-0.5.1-checkpoint slash --nodev slash --export_dir
./export_final_dataset slash --epochs 100 slash --dropout_rate4 0.05
slash --dropout_rate5 0.05

any suggestion

Thanks

[This is an archived TTS discussion thread from discourse.mozilla.org/t/automatically-session-get-restarted-in-between-the-steps]

[JRMeyer]

>>> haesustyle
[September 2, 2019, 3:53am]

From my experience, it's mostly a memory problem.

[Archived Post]