Skip to content

Latest commit

 

History

History
76 lines (50 loc) · 2.52 KB

README.md

File metadata and controls

76 lines (50 loc) · 2.52 KB
Raw

Aurora -- An Open-sourced GAN-based Text-to-Image Generation Model

Exploring Sparse MoE in GANs for Text-conditioned Image Synthesis
Jiapeng Zhu*, Ceyuan Yang*, Kecheng Zheng, Yinghao Xu, Zifan Shi, Yujun Shen
arXiv preprint arXiv:2309.03904

[Paper]

TODO

  • Release inference code
  • Release text-to-image generator at 64x64 resolution
  • Release models at higher resolution
  • Release training code
  • Release plug-ins/efficient algorithms for more functionalities

Installation

This repository is developed based on Hammer, where you can find more detailed instructions on installation. Here, we summarize the necessary steps to facilitate reproduction.

  1. Environment: CUDA version == 11.3.

  2. Install package requirements with conda:

    conda create -n aurora python=3.8  # create virtual environment with Python 3.8
conda activate aurora
pip install -r requirements/minimal.txt -f https://download.pytorch.org/whl/cu113/torch_stable.html

Inference

First, please download the pre-trained model here.

To synthesize an image with given text prompt, you can use the following command

python run_synthesize.py aurora_v1.pth 'A photo of a tree with autumn leaves'

To make interpolation between two text prompts, you can use the following command

python run_interpolate.py aurora_v1.pth \
       --src_prompt 'A photo of a tree with autumn leaves' \
       --dst_prompt 'A photo of a victorian house'

Results

  • Text-conditioned image generation

image

  • Text prompt interpolation

image

LICENSE

The project is under MIT License, and is for research purpose ONLY.

Acknowledgements

We highly appreciate StyleGAN2, StyleGAN3, CLIP, and Hammer for their contributions to the community.

BibTeX

@article{zhu2023aurora,
  title   = {Exploring Sparse {MoE} in {GANs} for Text-conditioned Image Synthesis},
  author  = {Zhu, Jiapeng and Yang, Ceyuan and Zheng, Kecheng and Xu, Yinghao and Shi, Zifan and Shen, Yujun},
  journal = {arXiv preprint arXiv:2309.03904},
  year    = {2023}
}