Working on a Live2D interactive character extension

[ye7iaserag]

So I got the idea to introduce some sort of intractability in chat mode and after a couple of tests, using emotion classification model with a Live2D character would give perfect results, also lip sync can be incorporated in the future.

The part where I need assistance is, I don't know where to add the character and it seems to look ugly where ever I add it

Vs.

What do you think?
Any other suggestion for character placement or the extension would be more than welcome

[Denshirenji-san]

[ye7iaserag]

You mean like a dedicated app?
From my perspective and experience, this would be like a dedicated game, game development takes loads of resources and time.

What I'm thinking about is just to add an extension here, also we can make the character placement movable on screen but not in a dedicated window.

Please elaborate if I misunderstood anything

[oobabooga]

Wow, that's really cool! What if the chat box itself was moved to the side instead of the middle of the screen, and the character occupied the left 30% and the chat the right 70% of the screen?

[ye7iaserag]

Here is how it looks:

[oobabooga]

I like it!

[sgsdxzy]

Just a side thought: what about using the LLM itself to do emotion classification? Is a separate model necessary?

[ye7iaserag]

I'm currently using emotion-english-distilroberta-base https://huggingface.co/j-hartmann/emotion-english-distilroberta-base
And might switch to https://huggingface.co/joeddav/distilbert-base-uncased-go-emotions-student but not very impressed with the results. the benefit we get here is that those classification models are extremely small and thus pretty fast to get a response.

I'm not sure, but I think we can even use an instruct based model for that and ask it about emotion and sentiment, but again this will be way slower and might also pollute context since I'm not sure how the webui is dealing with history.

[sgsdxzy]

I think batch size = 1 when inference never saturates gpus, especially when using int8. Usually I see around 20-30% usage of gpu on my 2080Ti. That means, running two concurrent inference tasks in the same batch would be almost as fast as one (it does require more VRAM though).
@oobabooga is it possible to support running batch size > 1 in webui? We could use one stream for text generation and the others for emotion, etc.

[ye7iaserag]

I'm also thinking about sentiment analysis models, cheap processing and high accuracy but not sure how those could be utilized here

[JustCaptcha]

Wow! awesome! That's what I were thinking about. 👍

[drkseo]

Any update on this ?
Great minds thinks alike I actually started working on a Web UI running Live2D to animate an avatar while chatting with llm (api or local). I'm at a point now where I see two clear paths :
1- Developping custom solution for businesses for enhanced chatbot experiences. Custom trained LLMs with internet/database access and narrowed to specific tasks (Q&A, after sales services, marketing stunts etc)
2-And a oobabooga addon, I'm surprise to still not see any WebGL addon to animate avatar in it after so much time and so many use case for RP.
(see attached for the advancement on my part)

[ye7iaserag]

I'd say go for a dedicated one, manipulating stuff around Gradio is a PAIN!

[DDXDB]

Man, you did what I always wanted to do but couldn't!
Is it possible to have Live2d output directly to an html or program source that OBS can capture?
Support Custom TTS?

[drkseo]

And more ! But alot happened in the space since then, I suggest you take a look at https://github.com/SillyTavern/SillyTavern for a local LLM inference front-end coupled with their https://github.com/SillyTavern/Extension-Live2d extension ! Have fun

[drkseo]

And if you still want to make your own check out https://github.com/RaSan147/pixi-live2d-display