Skip to content

SanskarG83/Dubwizz-Video-Dubbing-System

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

2 Commits
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Dubwiz : A Multilingual Video Dubbing System

fefe

Introducing an innovative solution designed to bridge the gap in disseminating government policies and schemes to rural India. Our Web-based Video Dubbing Platform harnesses advanced transformer models, offering a transformative way to convey vital information. It's as simple as uploading a video or YouTube link to our portal, where our cutting-edge technologies come into play. (Video Demonstration)

Overall Workflow of Project

FlowChart

Project Structure

project-root/
│
├── backend/
│   ├── server.py                 # Flask server script
│   ├── requirements/
│   │   └── requirements.txt      # Backend Python dependencies
│   ├── final/                    # Directory to store final dubbed video
│   ├── output/                   # Directory to store output files (subtitles, audio)
│   └── ...
│
└── frontend/
    ├── public/
        └── index.html
    ├── src/
    │   └── ...                    # ReactJS frontend source files
    ├── package.json               # Frontend Node.js dependencies
    └── ...

Frontend (ReactJS)

  • The frontend of this project is developed using ReactJS. You'll need Node.js and npm (Node Package Manager) to install and manage the frontend dependencies. Ensure you have Node.js installed, then navigate to the `frontend` directory and run:
npm install

This will install all the required frontend dependencies listed in `frontend/package.json`.

  • To run the frontend, use the following commands:
cd client
npm start

Backend Requirements

To run the backend of this project, you'll need the following Python libraries:

  • torch
  • torchaudio
  • transformers
  • datasets
  • moviepy
  • requests
  • Flask
  • gunicorn
  • python-dotenv
  • soundfile
  • librosa

These requirements are specified in the `requirements.txt` file provided in the `backend` folder. You can install these dependencies using pip:

pip install -r backend/requirements.txt

To run the backend server, use the following commands:

cd server
python server.py

Usage

  1. Setting up the Backend:

    • Install Python dependencies using `pip` as described above.
    • Start the Flask server by running `python backend/server.py` from the project root directory.
  2. Setting up the Frontend:

    • Install frontend dependencies using `npm install` from the `frontend` directory.
    • Start the React development server to run the frontend interface.
  3. Workflow:

    • Frontend:

      • User uploads a video file through the frontend interface.
    • Backend:

      • The backend receives the uploaded video file.

      • The video file is processed to detach the audio using MoviePy library.

      • The extracted audio is saved in the `backend/output` directory.

      • The extracted audio is transcribed into English text using OpenAI's Whisper model.

      • The English text is translated into the desired regional language using Facebook's mBART model (`facebook/mbart-large-50-one-to-many-mmt`).

      • The translated text is converted back into audio using a speech synthesis model like Silero.

      • The original video file is combined with the newly generated audio to create the final dubbed video.

      • The final dubbed video is saved in the `backend/final` directory.

Sample Input Video

You can use the following sample input video for testing the video dubbing application:

Sample Input Video

Sample Output Video

Here is an example of the output video generated by the video dubbing application:

Sample Output Video

ML Models used :

  • Open AI Whisper: Accurate audio-to-text transcription.
  • MBart: Seamless text translation into desired Indian languages.
  • Silero: Authentic audio conversion in multiple accents.
  • Wave2Lip: Lip Sync Audio over a video.

About

No description, website, or topics provided.

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published