Error in streaming

[JRMeyer]

>>> sayantangangs.91
[August 21, 2019, 5:41am]

I have been using deepspeech where the sound blob was recorded to the
full and then sent to the backend for inference. However, this is not
instantaneous. Instead, I planned to use real time streaming. However,
it seems the accuracy is reducing with streaming. slash
Could you please explain why is it so and how it can be improved?

[This is an archived TTS discussion thread from discourse.mozilla.org/t/error-in-streaming]

[JRMeyer]

>>> lissyx
[August 21, 2019, 6:10am]



> Could you please explain why is it so and how it can be improved?

That's not expected, please give more context.

[Archived Post]

[JRMeyer]

>>> reuben
[August 21, 2019, 7:15am]

One particular thing about streaming you should make sure to get right
is not to drop any samples during the transcription process. The
streaming process happens in batches of 16 time steps/320ms, so in those
bursty periods you can sometimes drop recorded samples, specially if
you're recording and transcribing in the same thread. Use threads and a
queue.

[Archived Post]

[JRMeyer]

>>> sayantangangs.91
[August 21, 2019, 9:56am]

Well, for eg:

'note down in diagnosis acute myeloid leukaemia'

works great if the whole file is given

In case it's streaming, it is like slash
'note down in diagnosis' slash
'take note in'

[Archived Post]

[JRMeyer]

>>> lissyx
[August 21, 2019, 9:58am]



> 'take note in'

Instead of 'acute myeloid leukaemia' ? What's your accents? It's very
much likely a case of domain-specfic words not well learnt, maybe with
non american english accent, and what
mentionned can also have a
play here.

[Archived Post]

[JRMeyer]

>>> sayantangangs.91
[August 21, 2019, 10:01am]

I get it. However, before prodution, I'm working with the example at
'mic_vad_streaming'. However, I understand using threads would help in
production so that the recorded files aren't lost. Although I'm not sure
DJango (I'm using django powered API) allows threading. Nonoetheless,
that's beyond the scope. Still, I shall try and use two threads, one for
recording the other for transcribing and any issues I get there, I shall
follow-up with you in this thread,

[Archived Post]

[JRMeyer]

>>> lissyx
[August 21, 2019, 10:03am]



> Still, I shall try and use two threads, one for recording the other
> for transcribing and any issues I get there, I shall follow-up with
> you in this thread,

This is the design that we used as well in several tools using Streaming
API, and it works quite well. With Python, you might want to use
multiprocess though, threads are not always the best solution in this
environment.

[Archived Post]

[JRMeyer]

>>> sayantangangs.91
[August 21, 2019, 10:07am]

Yes



> Instead of 'acute myeloid leukaemia' ?

Yes.



> What's your accents

My accent is Indian English and I've trained on about 80 hrs of data,
from the existing model (0.5.1). And the loss settles to around 12.3,
with WER around 8%. Of these 80 hrs, 65 hrs data is of general Indian
english and 15 hrs of domain specific voice in Indian English Accent.



> It's very much likely a case of domain-specfic words not well learnt,
> maybe with non american english accent

, could you please clarify if
the voice model is trained decently and then I add new words to the
language model, will that cause a big issue in the accuracy (For
example, if lets say 'alzheimer's disease' was not in original LM, but
added in updated LM, isn't it expected to work great too?



mentionned can also
> have a play here.

Yes, that's a legit point.

[Archived Post]

[JRMeyer]

>>> sayantangangs.91
[August 21, 2019, 10:09am]



> With Python, you might want to use multiprocess though, threads are
> not always the best solution in this environment.

Thanks a lot. Shall keep in mind and update accordingly. Thank you. slash
Looking forward to your reply on the previous point (about LM)

[Archived Post]

[JRMeyer]

>>> lissyx
[August 21, 2019, 10:12am]



> Of these 80 hrs, 65 hrs data is of general Indian english and 15 hrs
> of domain specific voice in Indian English Accent.

That might be enough, or not, I'm unsure. Be careful about noise, etc.,
as well.



> if the voice model is trained decently

At least you have some data, if it's enough or not, I don't know. That
depends on how you augmented / defined your test set, to know if the WER
figures you have are accurate.



> I add new words to the language model, will that cause a big issue in
> the accuracy (For example, if lets say 'alzheimer's disease' was not
> in original LM, but added in updated LM, isn't it expected to work
> great too?

That might be a good help. Unfortunately we don't yet have an API to add
domain-specific LM on top of generic LM.

[Archived Post]

[JRMeyer]

>>> sayantangangs.91
[August 21, 2019, 10:24am]

Thank you. I understand it'll not be possible to provide an answer on
whether the dataset is enough or not.



> That might be a good help. Unfortunately we don't yet have an API to
> add domain-specific LM on top of generic LM.

I think you got me wrong. I mean, I'm creating my own domain specific
LM, with sentences, lets say like: 'Note in diagnosis query acute
myeloid leukaemia' and the same is also having a voice model slash
Now, I'm recreating another LM (I have the existing .txt file, where I'm
adding new lines and creating LM from there) add adding some new lines,
one of them being: slash
'record in diagnosis alzheimer's disease' slash
Now, the AI model has never heard someone pronounce 'alzheimer's
disease'. Will it be a problem? I believe AI is expected to work with
unheard words as well. Right?

[Archived Post]

[JRMeyer]

>>> sayantangangs.91
[August 21, 2019, 11:05am]



> It's very much likely a case of domain-specfic words not well learnt

I just tested my model with the original LM (from Deepspeech) and
following is the output:

Original sentence: 'Can you hear me'

In case the whole sentence is translated, Deepspeech output: 'Can you
hear me' slash
In case the sentence is translated in parts via streaming, Deepspeech
output: slash
part 1: 'Can you' slash
part 2: 'here me'

I believe the LM is at play on selecting between 'here' and 'hear'. In
streaming its not getting the reference. Or am I missing something?

[Archived Post]

[JRMeyer]

>>> lissyx
[August 21, 2019, 11:12am]



> In streaming its not getting the reference

Streaming should pick up the context, as much as I remember. Maybe
can correct me on that ?

Also, what version are you running?



> Now, the AI model has never heard someone pronounce 'alzheimer's
> disease'. Will it be a problem? I believe AI is expected to work with
> unheard words as well. Right?

Up to the point that the model has been trained enough to be able to
generalize correctly.



> I think you got me wrong. I mean, I'm creating my own domain specific
> LM, with sentences, lets say like: 'Note in diagnosis query acute
> myeloid leukaemia' and the same is also having a voice model slash
> Now, I'm recreating another LM (I have the existing .txt file, where
> I'm adding new lines and creating LM from there) add adding some new
> lines, one of them being: slash
> 'record in diagnosis alzheimer's disease'

So you only have your sentences in the LM, right? You don't re-use
our LM and add your own data ?

[Archived Post]

[JRMeyer]

>>> sayantangangs.91
[August 21, 2019, 11:25am]

> Streaming should pick up the context, as much as I remember. Maybe
can correct me on that ?

That would be really helpful. In case streaming does pick-up reference,
I might've been digging the wrong hole...
if you can kindly enlighten

> Up to the point that the model has been trained enough to be able to
> generalize correctly.
>

Got it.

> So you only have your sentences in the LM, right? You don't re-use
> our LM and add your own data ?

I'm using my custom LM, with only my sentences in the LM and not
re-using your LM.

Although I feel I would have to reuse your LM for generalizing some
stuffs (so there, I'll use your .txt file and add lines from my.txt file
and generate a new LM, or use interpolation for LM_deepspeech &
LM_custom_generated_by_me).

[Archived Post]

[JRMeyer]

>>> sayantangangs.91
[August 22, 2019, 7:08am]



> That would be really helpful. In case streaming does pick-up
> reference, I might've been digging the wrong hole...
if you can kindly enlighten

looking forward to your
feedback on this

[Archived Post]

[JRMeyer]

>>> eggonlea
[August 23, 2019, 4:59pm]

I'm pretty sure streaming mode should give you the same result as legacy
mode. If it breaks the long sentence into pieces, maybe you have VAD
somewhere.

[Archived Post]