Introducing an innovative solution designed to bridge the gap in disseminating government policies and schemes to rural India. Our Web-based Video Dubbing Platform harnesses advanced transformer models, offering a transformative way to convey vital information. It's as simple as uploading a video or YouTube link to our portal, where our cutting-edge technologies come into play. (Video Demonstration)
project-root/
│
├── backend/
│ ├── server.py # Flask server script
│ ├── requirements/
│ │ └── requirements.txt # Backend Python dependencies
│ ├── final/ # Directory to store final dubbed video
│ ├── output/ # Directory to store output files (subtitles, audio)
│ └── ...
│
└── frontend/
├── public/
└── index.html
├── src/
│ └── ... # ReactJS frontend source files
├── package.json # Frontend Node.js dependencies
└── ...
- The frontend of this project is developed using ReactJS. You'll need Node.js and npm (Node Package Manager) to install and manage the frontend dependencies. Ensure you have Node.js installed, then navigate to the `frontend` directory and run:
npm install
This will install all the required frontend dependencies listed in `frontend/package.json`.
- To run the frontend, use the following commands:
cd client
npm start
To run the backend of this project, you'll need the following Python libraries:
- torch
- torchaudio
- transformers
- datasets
- moviepy
- requests
- Flask
- gunicorn
- python-dotenv
- soundfile
- librosa
These requirements are specified in the `requirements.txt` file provided in the `backend` folder. You can install these dependencies using pip:
pip install -r backend/requirements.txt
To run the backend server, use the following commands:
cd server
python server.py
-
Setting up the Backend:
- Install Python dependencies using `pip` as described above.
- Start the Flask server by running `python backend/server.py` from the project root directory.
-
Setting up the Frontend:
- Install frontend dependencies using `npm install` from the `frontend` directory.
- Start the React development server to run the frontend interface.
-
Workflow:
-
Frontend:
- User uploads a video file through the frontend interface.
-
Backend:
-
The backend receives the uploaded video file.
-
The video file is processed to detach the audio using MoviePy library.
-
The extracted audio is saved in the `backend/output` directory.
-
The extracted audio is transcribed into English text using OpenAI's Whisper model.
-
The English text is translated into the desired regional language using Facebook's mBART model (`facebook/mbart-large-50-one-to-many-mmt`).
-
The translated text is converted back into audio using a speech synthesis model like Silero.
-
The original video file is combined with the newly generated audio to create the final dubbed video.
-
The final dubbed video is saved in the `backend/final` directory.
-
-
You can use the following sample input video for testing the video dubbing application:
Here is an example of the output video generated by the video dubbing application:
- Open AI Whisper: Accurate audio-to-text transcription.
- MBart: Seamless text translation into desired Indian languages.
- Silero: Authentic audio conversion in multiple accents.
- Wave2Lip: Lip Sync Audio over a video.