🎆 Aura

Aligning AI Art with Emotion, in Real Time

Anger	Joy	Lust	Sadness

Aura is a live GAN-based video generator that supports the alignment of someone's emotion (or Aura) with its artistic representation. It uses StyleGAN-V's model with a CLIP-like representation of emotion-video pairs to perform interpolations between latents given a new input emotion and the current art state. We've also developed a WebRTC pipeline in Rust to stream model outputs and camera inputs in real time between your browser and your GPUs :)

How it Works

Emotion Feature Encoder Visualization with t-SNE

Face-Art Example

• Caption:

The artwork embodies a sense of vulnerability and innocence, expressed through soft, swirling curves that evoke warmth and comfort. Gentle shades of blue and muted grays intermingle with bursts of bright yellow and soft pastels, creating an interplay of solace and hope. The movement of the forms reflects subtle emotions, suggesting a narrative of resilience, and the texture of the piece gives a tactile quality, inviting viewers to connect with the underlying feelings of compassion and empathy.

Progress So Far

• Host UI on a website
• Create CLIP-based script for StyleGAN-V interpolations
• Enhance CLIP performance at Recall @ K
• Train StyleGAN-V
• Implement Queueing System in Python
• Create model endpoint to receive screen captures
• Generate Pairs of Embeddings and Videos with OpenSora and store on a large enough storage
• Create Ollama pipeline to generate video prompts to OpenSora
• Design an image processing pipeline to remove unfairness
• Find an emotion classification dataset
• Create Emotion Classifier Encoder CNN
• Allow screen-capture so that it can be sent to the model
• Create WebRTC renegotiation pipeline in Rust

How to Contribute

• The project supports rust-built Python modules as binaries if we need to do something fast i.e. a long Regex, camera I/O, etc :)
• To see details about running notebooks, visit /notebooks

Where you should be working:

• Python Modules: i.e. Camera input: aura/camera/ How to write Python modules
• Rust Packages for fast I/O and streaming: src/lib.rs (or another Rust module location inside src)
• Jupyter Notebooks for AI stuff: /notebooks

Development

Note: If you're developing on VSCode, download the necessary extensions to make your life easier (e.g. Jupyter Notebooks, Rust Analyzer, PyLint)

In your desired project location, run:

git clone https://github.com/ghubnerr/aura
cd aura

Initiate your conda environment. To download Anaconda and manage your Python environments, go here. Here we're using Python 3.10

conda create -n aura python=3.10
conda activate aura # Run this every time you open your project

We'll be using PyO3 + Maturin to integrate Rust and Python in the same package in case we need anything that executes fast. (Read more here)

Installing Python Dependencies

pip install -r requirements.txt

Using A100's for free to run our code in Colab (FIU Students Only)

• Follow instructions in /notebooks to connect to clusters and still run our repo.

Running Tests

Make sure you run this prior: maturin develop --extras tests

Testing Rust Source Code

Download Rust here

cargo test

If this doesn't work, try running:

For MacOS

export DYLD_LIBRARY_PATH="$CONDA_PREFIX/lib:$DYLD_LIBRARY_PATH"

For Linux (using tsch) -> onyx (FIU Server) uses tsch

setenv LD_LIBRARY_PATH "$CONDA_PREFIX/lib:$LD_LIBRARY_PATH"

• You will want to add this into your ~/.tcshrc file (use nano ~/.tcshrc):

source ~/miniconda3/etc/profile.d/conda.csh
setenv PATH "${PATH}:$HOME/.cargo/bin"
if ($?CONDA_PREFIX) then
    if ($?LD_LIBRARY_PATH) then
        setenv LD_LIBRARY_PATH ${CONDA_PREFIX}/lib:$LD_LIBRARY_PATH
    else
        setenv LD_LIBRARY_PATH ${CONDA_PREFIX}/lib
    endif
endif

• If it still doesn't work, do this:

conda activate aura
echo $CONDA_PREFIX # make sure this works
mkdir -p $CONDA_PREFIX/etc/conda/activate.d
touch $CONDA_PREFIX/etc/conda/activate.d/set_ld_library_path.csh
nano $CONDA_PREFIX/etc/conda/activate.d/set_ld_library_path.csh

• Write this:

#!/bin/tcsh
setenv LD_LIBRARY_PATH ${CONDA_PREFIX}/lib

• Save it and run this:

chmod +x $CONDA_PREFIX/etc/conda/activate.d/set_ld_library_path.csh

• Now run

conda deactivate
conda activate aura

This is an issue with a PATH variable that Maturin is using to access a Python file. I'm trying to see if there's a way to make this automatic.

Running Python Tests

conda install pytest # Only run this once

pytest
pytest aura/tests # (If the latter doesn't work)

Pushing to Source

1. Make sure you pull first

git pull
# If there are any conflicts, use `git stash`, then `git pull`, and then `git stash pop`. Address the conflicts there using VSCode Merge Editor

2. Commit your changes to a new branch

git add .
git commit -m "<your-message>"
git checkout -b <your_github_name>/<your_branch_feature>
git push -u origin <your_github_name>/<your_branch_feature>

3. Create a Pull Request for your branch on GitHub