Training on common voice error error after importing

[JRMeyer]

>>> JohnWayne
[September 13, 2019, 2:30am]

Hi

Hope you are well

I am currently having an error of training the model based on common
voice.

Not sure why I cant pick up the extracted csv files after loading the
model.

Please can I ask as to where this is going wrong:

Loading Code slash
Loading TSV file: /users/chabani/Desktop/Deepspeech/en/train.tsv slash
Saving new DeepSpeech-formatted CSV file to:
/users/chabani/Desktop/Deepspeech/en/clips/train.csv slash
100% completed slash
Imported 12123 samples. slash
Skipped 103 samples that failed on transcript validation.

Loading TSV file: /users/chabani/Desktop/Deepspeech/en/test.tsv slash
Saving new DeepSpeech-formatted CSV file to:
/users/chabani/Desktop/Deepspeech/en/clips/test.csv slash
100% completed slash
Imported 6810 samples. slash
Skipped 360 samples that failed on transcript validation. slash
Skipped 206 samples that were longer than 10 seconds. slash
Final amount of imported audio: 10:21:17.

Loading TSV file: /users/chabani/Desktop/Deepspeech/en/dev.tsv slash
Saving new DeepSpeech-formatted CSV file to:
/users/chabani/Desktop/Deepspeech/en/clips/dev.csv slash
100% completed slash
Imported 6940 samples. slash
Skipped 3 samples that failed upon conversion. slash
Skipped 326 samples that failed on transcript validation. slash
Skipped 73 samples that were longer than 10 seconds. slash
Final amount of imported audio: 9:25:57.

(deepspeech-train-venv) Chabanis-MBP:DeepSpeech chabani slash $
./DeepSpeech.py slash --train_files
.../users/chabani/Desktop/Deepspeech/en/clips/train.csv slash --dev_files
.../users/chabani/Desktop/Deepspeech/en/clips/dev.csv slash --test_files
/users/chabani/Desktop/Deepspeech/en/clips/test.csv

Error: slash
FileNotFoundError: slash [Errno 2 slash ] File
b'.../users/chabani/Desktop/Deepspeech/en/clips/train.csv' does not
exist: b'.../users/chabani/Desktop/Deepspeech/en/clips/train.csv'

Kind Regards

[This is an archived TTS discussion thread from discourse.mozilla.org/t/training-on-common-voice-error-error-after-importing]

[JRMeyer]

>>> lissyx
[September 13, 2019, 6:51am]

[ slash JohnWayne](
seems your paths are wrong. Why do you have the leading ../ ? Also,
please note you won't be able to do any seriously intensive training on
macOS, since there is no TensorFlow CUDA support on that platform.

[Archived Post]

[JRMeyer]

>>> reyxuan
[September 13, 2019, 8:12am]

Try to make an os.path.exists to make sure that those paths are written
correctly.

[Archived Post]

[JRMeyer]

>>> JohnWayne
[September 13, 2019, 1:00pm]

and
, okay will get the paths
correct.

With regards to training, I got linuxOS on virtual machine. I currently
have an issue with hardware cause I have 8gb of RAM - slash
Wanted to ask whether to upgrade the RAM to 16 or get an external GPU?
Which is the better option.

Thanks

[Archived Post]

[JRMeyer]

>>> lissyx
[September 13, 2019, 1:27pm]



> Wanted to ask whether to upgrade the RAM to 16 or get an external GPU?
> Which is the better option.

I guess that if you are training a real model, neither are good options.
I fear external GPU might behave erratically, since we've got reports of
exotic hardware configuration (related PCIe) that impacts a lot
performances. Also, 16GB is not a lot as well, but the more RAM the
better.

[Archived Post]

[JRMeyer]

>>> reyxuan
[September 16, 2019, 6:12am]

Isn't an RTX for example
powerful enough for a real model?

[Archived Post]

[JRMeyer]

>>> JohnWayne
[September 16, 2019, 7:40am]

Okay, thanks for the suggestion. Will also take into consideration the
question posed if an RTX is also suitable.

Lastly, I have attempted to import the model for training on Ubuntu
19.04 however I have this error appear, been trying to correct it but to
no avail:

Code

(deepspeech-train-venv)
chabani slash chabani-VirtualBox: slash ~/DeepSpeech/DeepSpeech slash $ bin/import_cv2.py
slash --filter_alphabet alphabet.txt /media/sf_en/

Loading TSV file: /media/sf_en/train.tsv slash
Saving new DeepSpeech-formatted CSV file to:
/media/sf_en/clips/train.csv slash
Importing mp3 files...

Traceback (most recent call last): slash
File 'bin/import_cv2.py', line 166, in slash
slash _preprocess_data(PARAMS.tsv_dir, AUDIO_DIR, label_filter_fun,
PARAMS.space_after_every_character) slash
File 'bin/import_cv2.py', line 43, in slash _preprocess_data slash
slash _maybe_convert_set(input_tsv, audio_dir, label_filter,
space_after_every_character) slash
File 'bin/import_cv2.py', line 100, in slash _maybe_convert_set slash
bar = progressbar.ProgressBar(max_value=num_samples, widgets=SIMPLE_BAR)

[Archived Post]

[JRMeyer]

>>> lissyx
[September 16, 2019, 7:50am]



> RTX for example powerful enough for a real model?

An RTX is useless if it's not able to be fed at pace. We've got reports
from people using specific PCIe configurations with huge slowdown
depending on TensorFlow / CUDA / our model changes (we don't know what
impacts). So i'm just warning people.

Also, a real model really needs more than just one RTX GPU if you have a
serious amount of data. WIth slash ~250h of French, it takes slash ~4h to train a
model on 2x RTX 2080 Ti.

[Archived Post]

[JRMeyer]

>>> lissyx
[September 16, 2019, 7:51am]



> Lastly, I have attempted to import the model for training on Ubuntu
> 19.04 however I have this error appear, been trying to correct it but
> to no avail:

Please make an effort and use proper code formatting. Some important
informations might be mangled by the markdown parsing.

That's indeed the case, your error is incomplete, I cannot help you.

[Archived Post]

[JRMeyer]

>>> reyxuan
[September 16, 2019, 9:16am]

Thanks for the warning. I
think my PCIe configuration is fine. For the moment I have slash ~250h and an
RTX 2080 Super.

> WIth slash ~250h of French, it takes slash ~4h to train a model on 2x RTX 2080
> Ti.

Do you mean training a model or training an epoch?

[Archived Post]

[JRMeyer]

>>> lissyx
[September 16, 2019, 11:13am]



> Do you mean training a model or training an epoch?

One model. Please check
https://github.com/Common-Voice/commonvoice-fr/blob/master/DeepSpeech/Dockerfile.train.fr
for details.

[Archived Post]

[JRMeyer]

>>> JohnWayne
[September 17, 2019, 12:17am]

Apologies for that:

The code: slash
(deepspeech-train-venv)
chabani slash chabani-VirtualBox: slash ~/DeepSpeech/DeepSpeech slash $ bin/import_cv2.py
slash --filter_alphabet alphabet.txt /media/sf_en/

/home/chabani/tmp/deepspeech-train-venv/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:516:
FutureWarning: Passing (type, 1) or '1type' as a synonym of type is
deprecated; in a future version of numpy, it will be understood as
(type, (1,)) / '(1,)type'. slash
slash _np_qint8 = np.dtype( slash [('qint8', np.int8, 1) slash ]) slash
/home/chabani/tmp/deepspeech-train-venv/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:517:
FutureWarning: Passing (type, 1) or '1type' as a synonym of type is
deprecated; in a future version of numpy, it will be understood as
(type, (1,)) / '(1,)type'. slash
slash _np_quint8 = np.dtype( slash [('quint8', np.uint8, 1) slash ]) slash
/home/chabani/tmp/deepspeech-train-venv/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:518:
FutureWarning: Passing (type, 1) or '1type' as a synonym of type is
deprecated; in a future version of numpy, it will be understood as
(type, (1,)) / '(1,)type'. slash
slash _np_qint16 = np.dtype( slash [('qint16', np.int16, 1) slash ]) slash
/home/chabani/tmp/deepspeech-train-venv/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:519:
FutureWarning: Passing (type, 1) or '1type' as a synonym of type is
deprecated; in a future version of numpy, it will be understood as
(type, (1,)) / '(1,)type'. slash
slash _np_quint16 = np.dtype( slash [('quint16', np.uint16, 1) slash ]) slash
/home/chabani/tmp/deepspeech-train-venv/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:520:
FutureWarning: Passing (type, 1) or '1type' as a synonym of type is
deprecated; in a future version of numpy, it will be understood as
(type, (1,)) / '(1,)type'. slash
slash _np_qint32 = np.dtype( slash [('qint32', np.int32, 1) slash ]) slash
/home/chabani/tmp/deepspeech-train-venv/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:525:
FutureWarning: Passing (type, 1) or '1type' as a synonym of type is
deprecated; in a future version of numpy, it will be understood as
(type, (1,)) / '(1,)type'. slash
np_resource = np.dtype( slash [('resource', np.ubyte, 1) slash ]) slash
/home/chabani/tmp/deepspeech-train-venv/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:541:
FutureWarning: Passing (type, 1) or '1type' as a synonym of type is
deprecated; in a future version of numpy, it will be understood as
(type, (1,)) / '(1,)type'. slash
slash _np_qint8 = np.dtype( slash [('qint8', np.int8, 1) slash ]) slash
/home/chabani/tmp/deepspeech-train-venv/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:542:
FutureWarning: Passing (type, 1) or '1type' as a synonym of type is
deprecated; in a future version of numpy, it will be understood as
(type, (1,)) / '(1,)type'. slash
slash _np_quint8 = np.dtype( slash [('quint8', np.uint8, 1) slash ]) slash
/home/chabani/tmp/deepspeech-train-venv/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:543:
FutureWarning: Passing (type, 1) or '1type' as a synonym of type is
deprecated; in a future version of numpy, it will be understood as
(type, (1,)) / '(1,)type'. slash
slash _np_qint16 = np.dtype( slash [('qint16', np.int16, 1) slash ]) slash
/home/chabani/tmp/deepspeech-train-venv/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:544:
FutureWarning: Passing (type, 1) or '1type' as a synonym of type is
deprecated; in a future version of numpy, it will be understood as
(type, (1,)) / '(1,)type'. slash
slash _np_quint16 = np.dtype( slash [('quint16', np.uint16, 1) slash ]) slash
/home/chabani/tmp/deepspeech-train-venv/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:545:
FutureWarning: Passing (type, 1) or '1type' as a synonym of type is
deprecated; in a future version of numpy, it will be understood as
(type, (1,)) / '(1,)type'. slash
slash _np_qint32 = np.dtype( slash [('qint32', np.int32, 1) slash ]) slash
/home/chabani/tmp/deepspeech-train-venv/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:550:
FutureWarning: Passing (type, 1) or '1type' as a synonym of type is
deprecated; in a future version of numpy, it will be understood as
(type, (1,)) / '(1,)type'. slash
np_resource = np.dtype( slash [('resource', np.ubyte, 1) slash ]) slash
Loading TSV file: /media/sf_en/train.tsv slash
Saving new DeepSpeech-formatted CSV file to:
/media/sf_en/clips/train.csv slash
Importing mp3 files...

Traceback (most recent call last): slash
File 'bin/import_cv2.py', line 166, in slash
slash _preprocess_data(PARAMS.tsv_dir, AUDIO_DIR, label_filter_fun,
PARAMS.space_after_every_character) slash
File 'bin/import_cv2.py', line 43, in slash _preprocess_data slash
slash _maybe_convert_set(input_tsv, audio_dir, label_filter,
space_after_every_character) slash
File 'bin/import_cv2.py', line 100, in slash _maybe_convert_set slash
bar = progressbar.ProgressBar(max_value=num_samples,
widgets=SIMPLE_BAR) slash
TypeError: init() got an unexpected keyword argument 'max_value'

Hope the posting is appropriate. This is the output I receive from
terminal

[Archived Post]

[JRMeyer]

>>> lissyx
[September 17, 2019, 7:53am]



> Apologies for that:

Please use proper code formatting, this is unreadable and Markdown
parser is eating important Python informations.

[Archived Post]

[JRMeyer]

>>> reyxuan
[September 17, 2019, 8:20am]

[ slash JohnWayne](
Please use slash *slash* slash your code *slash* slash *slash*.

[Archived Post]

[JRMeyer]

>>> JohnWayne
[September 17, 2019, 3:04pm]

Okay, hope its readable now.

(deepspeech-train-venv) chabanichabani-VirtualBox:~/DeepSpeech/DeepSpeech$ bin/import_cv2.py --filter_alphabet alphabet.txt /media/sf_en/

_np_qint8 = np.dtype([('qint8', np.int8, 1)])
/home/chabani/tmp/deepspeech-train-venv/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:517: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_quint8 = np.dtype([('quint8', np.uint8, 1)])
/home/chabani/tmp/deepspeech-train-venv/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:518: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint16 = np.dtype([('qint16', np.int16, 1)])
/home/chabani/tmp/deepspeech-train-venv/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:519: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_quint16 = np.dtype([('quint16', np.uint16, 1)])
/home/chabani/tmp/deepspeech-train-venv/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:520: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint32 = np.dtype([('qint32', np.int32, 1)])
/home/chabani/tmp/deepspeech-train-venv/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:525: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
np_resource = np.dtype([('resource', np.ubyte, 1)])
/home/chabani/tmp/deepspeech-train-venv/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:541: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint8 = np.dtype([('qint8', np.int8, 1)])
/home/chabani/tmp/deepspeech-train-venv/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:542: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_quint8 = np.dtype([('quint8', np.uint8, 1)])
/home/chabani/tmp/deepspeech-train-venv/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:543: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint16 = np.dtype([('qint16', np.int16, 1)])
/home/chabani/tmp/deepspeech-train-venv/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:544: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_quint16 = np.dtype([('quint16', np.uint16, 1)])
/home/chabani/tmp/deepspeech-train-venv/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:545: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint32 = np.dtype([('qint32', np.int32, 1)])
/home/chabani/tmp/deepspeech-train-venv/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:550: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
np_resource = np.dtype([('resource', np.ubyte, 1)])
Loading TSV file: /media/sf_en/train.tsv
Saving new DeepSpeech-formatted CSV file to: /media/sf_en/clips/train.csv
Importing mp3 files...
Traceback (most recent call last):
File 'bin/import_cv2.py', line 166, in
_preprocess_data(PARAMS.tsv_dir, AUDIO_DIR, label_filter_fun, PARAMS.space_after_every_character)
File 'bin/import_cv2.py', line 43, in _preprocess_data
_maybe_convert_set(input_tsv, audio_dir, label_filter, space_after_every_character)
File 'bin/import_cv2.py', line 100, in _maybe_convert_set
bar = progressbar.ProgressBar(max_value=num_samples, widgets=SIMPLE_BAR)
TypeError: init() got an unexpected keyword argument 'max_value```

Placed the inverted commas as reyxuan suggested. Quite different on linux from mac. Hope it works now

[Archived Post]

[JRMeyer]

>>> lissyx
[September 17, 2019, 12:43pm]



> Placed the inverted commas as
suggested. Quite different
> on linux from mac. Hope it works now

No it's still the same, you have used the wrong ones.

[Archived Post]

[JRMeyer]

>>> nmstoker
[September 17, 2019, 1:30pm]

[ slash JohnWayne](
may have missed it, but you can also typically edit a post. slash
That way you avoid a whole repost; just need to go in and type the
correct character. My guess is that you've somehow got inverted commas
that are 'smart' (ie adjusted for opening and closing quotes) and those
are not the ones to use
![:slightly_smiling_face:](

[Archived Post]

[JRMeyer]

>>> JohnWayne
[September 17, 2019, 3:05pm]

.
Got the formatting done correctly

[Archived Post]

[JRMeyer]

>>> lissyx
[September 18, 2019, 7:52am]



> bar = progressbar.ProgressBar(max_value=num_samples,
> widgets=SIMPLE_BAR) TypeError: init() got an unexpected keyword
> argument slash 'max_value slash *slash* slash <br/> <br/> Have you properly setup your virtualenv? It looks like you have an<br/> incompatible progressbar` ...

[Archived Post]

[JRMeyer]

>>> JohnWayne
[September 18, 2019, 10:41pm]

I tried to set up my virtualenv again. Still had the same issue. Some
user on stackoverflow pointed out Progressbar2 deals with max_value
while Progressbar in GNU/Linux accepts maxval.

It works now after install Progressbar2.

Thank you for the help

[next page
→

[Archived Post]