Prediction - Transcribe usage and documentation

[austinrausch]

Before opening, please confirm:

• I have searched for duplicate or closed issues and discussions.
• I have read the guide for submitting bug reports.
• I have done my best to include a minimal, self-contained set of instructions for consistently reproducing the issue.

JavaScript Framework

Angular

Amplify APIs

Predictions

Amplify Version

v6

Amplify Categories

predictions

Backend

Amplify Gen 2

Environment information

# Put output below this line
System:
    OS: macOS 14.6.1
    CPU: (12) x64 Intel(R) Core(TM) i7-8850H CPU @ 2.60GHz
    Memory: 82.43 MB / 16.00 GB
    Shell: 5.9 - /bin/zsh
  Binaries:
    Node: 22.14.0 - ~/.nvm/versions/node/v22.14.0/bin/node
    Yarn: 1.22.21 - /usr/local/bin/yarn
    npm: 10.9.2 - ~/.nvm/versions/node/v22.14.0/bin/npm
  Browsers:
    Chrome: 133.0.6943.127
    Safari: 17.6
  npmPackages:
    %name%:  0.1.0 
    @angular-devkit/build-angular: ^18.2.5 => 18.2.14 
    @angular/animations: ^18.2.5 => 18.2.13 
    @angular/cdk: ^18.2.5 => 18.2.14 
    @angular/cli: ~18.2.5 => 18.2.14 
    @angular/common: ^18.2.5 => 18.2.13 
    @angular/compiler: ^18.2.5 => 18.2.13 
    @angular/compiler-cli: ^18.2.5 => 18.2.13 
    @angular/core: ^18.2.5 => 18.2.13 
    @angular/forms: ^18.2.5 => 18.2.13 
    @angular/material: ^18.2.7 => 18.2.14 
    @angular/platform-browser: ^18.2.5 => 18.2.13 
    @angular/platform-browser-dynamic: ^18.2.5 => 18.2.13 
    @angular/router: ^18.2.5 => 18.2.13 
    @aws-amplify/backend: ^1.0.4 => 1.14.1 
    @aws-amplify/backend-cli: ^1.1.1 => 1.4.11 
    @aws-amplify/predictions: ^6.1.44 => 6.1.45 
    @aws-amplify/ui-angular: ^5.0.19 => 5.0.37 
    @dotenvx/dotenvx: ^1.14.1 => 1.36.0 
    audiomotion-analyzer: ^4.5.0 => 4.5.0 
    aws-amplify: ^6.4.0 => 6.13.1 
    aws-amplify/adapter-core:  undefined ()
    aws-amplify/adapter-core/internals:  undefined ()
    aws-amplify/analytics:  undefined ()
    aws-amplify/analytics/kinesis:  undefined ()
    aws-amplify/analytics/kinesis-firehose:  undefined ()
    aws-amplify/analytics/personalize:  undefined ()
    aws-amplify/analytics/pinpoint:  undefined ()
    aws-amplify/api:  undefined ()
    aws-amplify/api/internals:  undefined ()
    aws-amplify/api/server:  undefined ()
    aws-amplify/auth:  undefined ()
    aws-amplify/auth/cognito:  undefined ()
    aws-amplify/auth/cognito/server:  undefined ()
    aws-amplify/auth/enable-oauth-listener:  undefined ()
    aws-amplify/auth/server:  undefined ()
    aws-amplify/data:  undefined ()
    aws-amplify/data/server:  undefined ()
    aws-amplify/datastore:  undefined ()
    aws-amplify/in-app-messaging:  undefined ()
    aws-amplify/in-app-messaging/pinpoint:  undefined ()
    aws-amplify/push-notifications:  undefined ()
    aws-amplify/push-notifications/pinpoint:  undefined ()
    aws-amplify/storage:  undefined ()
    aws-amplify/storage/s3:  undefined ()
    aws-amplify/storage/s3/server:  undefined ()
    aws-amplify/storage/server:  undefined ()
    aws-amplify/utils:  undefined ()
    aws-cdk: ^2.149.0 => 2.1000.2 
    aws-cdk-lib: ^2.149.0 => 2.179.0 
    constructs: ^10.3.0 => 10.4.2 
    cookie: ^1.0.0 => 1.0.2 (0.7.2, 0.7.1)
    crypto-browserify: ^3.12.1 => 3.12.1 
    get-user-media-promise: ^1.1.4 => 1.1.4 
    jasmine: ^5.1.4 => 5.6.0 
    jasmine-core: ~5.3.0 => 5.3.0 (5.6.0, 4.6.1)
    karma: ~6.4.0 => 6.4.4 
    karma-chrome-launcher: ~3.2.0 => 3.2.0 
    karma-coverage: ~2.2.0 => 2.2.1 
    karma-coverage-coffee-example:  1.0.0 
    karma-jasmine: ~5.1.0 => 5.1.0 
    karma-jasmine-html-reporter: ~2.1.0 => 2.1.0 
    microphone-stream: ^5.0.1 => 5.3.1 
    ngx-skeleton-loader: ^10.0.0 => 10.0.0 
    node-polyfill-webpack-plugin: ^4.1.0 => 4.1.0 
    prettier: 3.3.3 => 3.3.3 (2.3.2, 2.8.8, 1.19.1)
    rxjs: ^7.8.1 => 7.8.1 
    rxjs/ajax:  undefined ()
    rxjs/fetch:  undefined ()
    rxjs/operators:  undefined ()
    rxjs/testing:  undefined ()
    rxjs/webSocket:  undefined ()
    sass: ^1.79.4 => 1.85.0 (1.77.6)
    sass-loader: ^16.0.2 => 16.0.5 (16.0.0)
    stream-browserify: ^3.0.0 => 3.0.0 
    tslib: ^2.3.0 => 2.8.1 (2.6.3, 2.0.3, 2.2.0, 2.3.1, 1.14.1, 2.4.1)
    typescript: ~5.5.0 => 5.5.4 (4.4.4, 4.9.5)
    zone.js: ~0.15.0 => 0.15.0 
  npmGlobalPackages:
    corepack: 0.31.0
    npm: 10.9.2

Describe the bug

Hello! I would like to request an update to documentation and to give a usable example for the Prediction calls that can be made with Amplify. I joined the Discord channel office hours to ask for clarification and I was asked to make a post here. Currently the documentation is a little thin, but I've found that the calls do work with a significant amount of custom changes around the given example.

I followed the instructions here: https://docs.amplify.aws/angular/build-a-backend/add-aws-services/predictions/set-up-predictions/

My backend.ts includes:

// Configure a policy for the required use case.
// The actions included below cover all supported ML capabilities
backend.auth.resources.unauthenticatedUserIamRole.addToPrincipalPolicy(
  new PolicyStatement({
    actions: ['transcribe:StartStreamTranscriptionWebSocket'],
    resources: ['*'],
  }),
);

backend.addOutput({
  custom: {
    Predictions: {
      convert: {
        transcription: {
          defaults: {
            language: 'en-US',
          },
          proxy: false,
          region: backend.auth.stack.region,
        },
      },
    },
  },
});

I also included in my main.ts the following:

import { Amplify } from "aws-amplify";
import outputs from '../amplify_outputs.json';

Amplify.configure(outputs);

My other Amplify backend features are working fine. Auth, api, data, storage, deploy all work. Predictions returns the error
Error processing chunk: NoRegion: Missing region.

It was suggested in my discord post that I update my project to include the following to resolve the error:

import { Amplify } from "aws-amplify";
import { parseAmplifyConfig } from "aws-amplify/utils";
import outputs from "../amplify_outputs.json";

const amplifyConfig = parseAmplifyConfig(outputs);

Amplify.configure({
  ...amplifyConfig,
  Predictions: outputs.custom.Predictions,
});

After this, the call succeeded, but I was only receiving empty string results when sending my payload after recording with the stock MediaRecorder approach. I found the following React project from a YouTuber which utilizes microphone-stream and get-user-media-promise packages: https://github.com/gitmurali/react-aws-sentiment-analysis/blob/b381790f893d4d13f4c16ee5890165d43544778e/src/components/AudioRecorder.js.

I was able to piece together parts of this to get it to work, but I feel like I'm missing something. Is there a simpler approach without installing multiple npm packages?

Thanks for all you do!

Expected behavior

Documentation provides a clear example of how to use the Transcribe call.

Reproduction steps

Using Amplify gen 2, attempt to use a transcription call by following the steps in this documentation:

https://docs.amplify.aws/angular/build-a-backend/add-aws-services/predictions/set-up-predictions/

https://docs.amplify.aws/angular/build-a-backend/add-aws-services/predictions/set-up-predictions/

Code Snippet

// Put your code below this line.

Log output

// Put your logs below this line

aws-exports.js

No response

Manual configuration

No response

Additional configuration

No response

Mobile Device

No response

Mobile Operating System

No response

Mobile Browser

No response

Mobile Browser Version

No response

Additional information and screenshots

No response

[HuiSF]

Hi @austinrausch thanks for reporting this documentation issue. The information about passing Prediction config into Amplify.configure() is indeed missing from the documentation.

Re:

After this, the call succeeded, but I was only receiving empty string results when sending my payload after recording with the stock MediaRecorder approach.

What was the payload you sent. According to this Prediction.convert API document and example (it's under Gen1 document, but Amplify JS library APIs remain the same across Gen1 and Gen2), the API expects to be passed with a PCM Audio byte buffer for transcription.

[jjarvisp]

Thanks @austinrausch, documentation has been updated with your suggestion.

[austinrausch]

Hi @austinrausch thanks for reporting this documentation issue. The information about passing Prediction config into Amplify.configure() is indeed missing from the documentation.

Re:

After this, the call succeeded, but I was only receiving empty string results when sending my payload after recording with the stock MediaRecorder approach.

What was the payload you sent. According to this Prediction.convert API document and example (it's under Gen1 document, but Amplify JS library APIs remain the same across Gen1 and Gen2), the API expects to be passed with a PCM Audio byte buffer for transcription.

Hi all, thanks for updating the documentation! I have a working setup now with the correct file type for the payload so I appreciate your help. Hopefully anybody else needing help with transcribe in the future can use this thread.

[hivemynds-dev]

Hi @austinrausch, I am also getting empty string for the output. Can you suggest what you did to fix the issue?

[austinrausch]

Hi @austinrausch, I am also getting empty string for the output. Can you suggest what you did to fix the issue?

@hivemynds-dev Sure. I found the following github page from a youtube video: https://github.com/gitmurali/react-aws-sentiment-analysis/blob/b381790f893d4d13f4c16ee5890165d43544778e/src/components/AudioRecorder.js

In this example he uses microphone-stream and get-user-media-promise npm packages to format the data correctly. After trying this I was able to get the transcription back.

[hivemynds-dev]

get-user-media-promise is a very old library and microphone-stream is really hard to get working on the latest react versions.

I feel this issue was closed prematurely.