[GSOC] Interest in Smart Meeting Notes Project and TIMM Performance Showcase Project

[sawradip]

Introduction

I am Sawradip Saha, a 4th year student at Bangladesh University of Engineering & Technology, pursuing my BSc in Mechanical Engineering. I have a passion for computer vision and have been working in this field for the past three years. I have previously contributed to Pytorch Official Repository as well as Skorch.

I have participated in several Deep Learning competitions, including the IEEE VIP Cup 2021(1st) and 2022(3rd). My projects and accomplishments in the field of computer vision and others my personal website.

OpenVino Contributions

As for the prerequisite of OpenVino GSoC'23 Participation, I have contributed this PR showing inference of YOLOX models using OpenVino.

Additionally I have submitted another PR adding a Text Spotting(Detection + Recognition) Notebook that is currenty under review. I have also presented some proposals regrading Google Colab support to Openvino-notebooks in this issue and this discussion with a demonstration of my proposed ideas in 219-knowledge-graphs-conve.

Interested Projects

For GSOC'23 I am specially interested in two projects.

2. Showcase performance of PyTorch Image Models (timm) with OpenVINO

Mentored by @AlexKoff88 and @l-bat
Reason of Interest: For most of my Computer Vision projects and competitions, I have worked with PyTorch. As a result I have done extensive use of TIMM models. It is an awesome project that is useful to Thus integration and performance improvement those models with OpenVino interests me very much.

9. Smart meeting notes

Mentored by @eaidova and @andrei-kochin
Reason of Interest: From the beginning, I have been exposed to a wide range of deep learning tasks due to participating in different competitions. As I have already worked in Speech-to-text(in DL Sprint), Pose/Action Recognition(in IEEE VIP Cup 2021) as well as other basic NLP tasks, the idea of integrating those different system using OpenVino optimization is very interesting to me. Thus this project interests me.

I express my gratitude to all the mentors who dedicated their valuable time from their busy schedules to mentor the participants of GSOC'23. Their efforts have led to significant skill improvements of the participants and contributed to the betterment of the open-source community as a whole.

[eaidova]

@sawradip thank you for your interest! Could you please briefly share your ideas how do you see the proposed tasks solution (which steps you can suggest to make these projects)?

[sawradip]

Let me share my ideas on Smart meeting notes project.

The project page mentions,

Short description: Artificial intelligence has incredible power for solving a lot of tasks, from creating works of art to helping to diagnose diseases. In this task, the candidate will explore how deep learning models can be used to process audio and video to create the meeting note-taking application using OpenVINO.

Expected outcomes: The demo application for audio and video analysis. The demo capabilities can be but are not limited to: speech recognition and separation, speaker and actions recognition, text summarization, and translation. The result should cover contributions to Open Model Zoo demos and can be represented as a notebook as well.

My thoughts:

At first I would like to start by researching and understanding the available OpenVINO models and frameworks for audio and video analysis, as well as the potential APIs and tools that could be used for speech recognition, speaker and action recognition, and text summarization and translation.

Then, I would develop a prototype or proof of concept to test and evaluate the different models and frameworks for audio and video analysis, as well as different approaches for combining and integrating them. Tho describe in steps,

• Take a sample meeting video clip
• Take a specific video as well as an audio analysis task. For example, lets take Action recognition (recognize and categorize different actions that occur during the meeting, such as applause or laughter) and Speech recognition and transcription(transcribe speech into text, and store the text in a database for later retrieval)
• Create a script/notebook to extract audio from the video clip and use those two models to analyze corresponding modalities.
• Create an extensible pipeline where I can easily add more models for further tasks, such as Speaker recognition , Text summarization and some additional ones to automatically create a short but organized document, the meeting note.
• Experiment with different features and functionalities, to determine the most useful and valuable features for the Smart Meeting Notes application.
• Test and evaluate the demo application with potential users, and use their feedback to further refine and improve the application.
• Finally, document the project, including the technical details and design decisions, and contribute any relevant code and models to the Open Model Zoo demos repository.

Like any complex AI projects. this one will also go through many iterations, review, correction and development cycles.

Would like to know you feedback, and what modifications you would suggest. @eaidova

[sawradip]

@eaidova @andrei-kochin I have submitted my proposal for Smart Meeting notes, on GSOC portal. Can you please review and share your feedback? Do I need to send somewhere else?

[sawradip]

@eaidova @andrei-kochin I have submitted my proposal, on GSOC portal. Can you please review and let me know feedback?

[sawradip]

@AlexKoff88 @l-bat I have submitted my proposal, on GSOC portal. Can you please review and let me know feedback?

[sawradip]

@eaidova Any chance you had any feedback for my proposal?