[Video] Video Component split audio from video

[ShihChun-H]

Issue Description

Current State

• The Video Component can only:

1. Subsample Video - Subsample video into a new video
2. Subsample Video Frames - Subsample video into frames

Why We Want to Change?

• Videos often come with audio but currently we cannot extract this for processing.

Proposed Change

• Add a task to extract the audio from video

Recipe proposal

"TASK_EXTRACT_AUDIOS": {
"instillShortDescription": "Extract audios from a video",
"input": {
"description": "Parameters to consider for extracting from the video",
"instillEditOnNodeFields": [c
"audio",
],
"instillUIOrder": 0,
"properties": {
"video": {
"description": "The input video",
"instillAcceptFormats": [
"video/",
"application/octet-stream"
],
"instillUIOrder": 0,
"instillUpstreamTypes": [
"reference"
],
"title": "Video",
"type": "binary"
},
"time-windows": {
"description": "Time windows for extracting videos in the format [[start time, end time], …], where the time is in the format hh:mm:ss",
"instillAcceptFormats": [
"array:array:string"
],
"instillUIOrder": 1,
"instillUpstreamTypes": [
"reference",
"value"
],
"title": "Time windows",
"type": "array"
},
},
"required": [
"video",
"time-windows"
],
"title": "Input",
"type": "object"
},
"output": {
"instillUIOrder": 1,
"properties": {
"audios": {
"description": "Extracted audios",
"instillFormat": "array:audio/",
"instillUIOrder": 0,
"items": {
"type": "binary",
"title": "Audio"
},
"title": "Audios",
"type": "array"
}
},
"required": [
"audios"
],
"title": "Output",
"type": "object"
}
}

---

Rules for the Component Hackathon

• Each issue will only be assigned to one person/team at a time.
• You can only work on one issue at a time.
• To express interest in an issue, please comment on it and tag @kuroxx, allowing the Instill AI team to assign it to you.
• Ensure you address all feedback and suggestions provided by the Instill AI team.
• If no commits are made within five days, the issue may be reassigned to another contributor.
• Join our Discord to engage in discussions and seek assistance in #hackathon channel. For technical queries, you can tag @chuang8511.

---

Component Contribution Guideline | Documentation | Official Go Tutorial

[Fustigate8933]

I'm interested in working on this one

[ShihChun-H]

Hi @Fustigate8933, the issue has been assigned to you 🙌🏻

[ShihChun-H]

Hi @Fustigate8933, I'm following up to check on any progress made regarding this issue. Could you please provide an update? Thanks 🙏

[kuroxx]

Hey @Fustigate8933 how's it going? If you have any progress update please let me know otherwise I will de-assign this issue from you soon so others have a chance to attempt it, thanks 🙏

[Anish0203]

Hi @kuroxx I would like to give a try. Can I please work on this issue? Thanks!

[kuroxx]

Hey @Anish0203 sounds great! I have assigned the issue to you 🤝 Happy Hacking!

[Anish0203]

Hi @kuroxx , as discussed over discord, I'm unable to set up the project locally due to resource constraints. Could you please unassign me? Sorry for any disturbance caused. Would love to contribute to Instill AI in future.
PS: I had tried a few pipelines in Instill Cloud and really loved the seamless feature of video summarizer. Here is the output of one of the videos that I had uploaded:
{
"key_insights": [
"Character Interaction: The scene involves two superheroes, indicating a moment of interaction or confrontation.",
"Urban Setting: The backdrop of skyscrapers suggests a major city, possibly New York.",
"Emotional Tone: The seated character appears contemplative or distressed, while the hovering character seems to be approaching or communicating."
],
"locations": [
"New York City",
"Metropolis-like urban areas"
],
"summary": "The images capture a dramatic encounter between two superheroes set against the backdrop of a bustling cityscape. One hero is seated on a ledge, seemingly deep in thought or distress, while the other hovers nearby, possibly engaging in a conversation or confrontation. The urban environment, characterized by towering skyscrapers, hints at a major city like New York, adding to the intensity and scale of the scene.",
"tags": [
"Superheroes",
"Cityscape",
"Interaction",
"Urban environment",
"Contemplation"
],
"title": "Superhero Encounter in the City"
}

[kuroxx]

@Anish0203 no worries at all and thank you for letting me know!! Love that you tried Instill Cloud and enjoyed the experience - can't wait to see what you build next! Happy hacking 🎃

The task has been unassigned now, thank you for trying though 🤝