• Paper Link: UnrealZoo: Enriching Photo-realistic Virtual Worlds for Embodied AI
• Comprehensive documentation is available in Doc
• Project website is available at UnrealZoo

1. What is UnrealZoo?
   • Key Features
   • Framework
2. Quick Installation
   • Dependencies
   • Install UnrealZoo-Gym
   • Prepare UE Binary
3. Run the Example Code
4. Build your agent
5. Documentation
6. TODO List
7. License and Acknowledgments
8. Citation

UnrealZoo is a rich collection of photo-realistic 3D virtual worlds built on Unreal Engine, designed to reflect the complexity and variability of the open worlds. There are various playable entities for embodied AI, including human characters, robots, vehicles, and animals. Integrated with UnrealCV, UnrealZoo provides a suite of easy-to-use Python APIs and tools for various potential applications, such as data annotation and collection, environment augmentation, distributed training, and benchmarking agents.

💡This repository provides the gym interface based on UnrealCV APIs for UE-based environments, which is compatible with OpenAI Gym and supports the high-level agent-environment interactions in UnrealZoo.

• Photorealistic: High-quality graphics rendering empowered by Unreal Engine (4.27/5.4).
• Large-scale: 100+ Scenes, the largest one covers 16km².
• Diverse Scenes: landscape🏜️🏞️🏝️, historic architecture⛩️🛕🏰, settlement🏘️🏙️, industrial zone🏭🏗️, facilities🤽🏻‍♂️🚉🏪...
• Diverse Bodies: human🚶🏻‍♂️️🚶🏻, robot🤖, vehicles🚗🏍️🛩️,animals🐘🐕🐎🐧🐢🐖🐏🐂...
• Diverse Actions: running🏃🏻‍♂️, climbing🧗🏻, sitting🧘🏻, jumping, squatting, pick-up...
• Easy-to-use: Pre-build UE binaries, integrated with optimized UnrealCV, are to be used without any prior knowledge of UE.
• Cross-platform: Runs on Linux, Windows, macOS
• Flexible Python APIs: Provide UnrealCV Python APIs and Gym Interfaces for various potential applications.
• Multi-agent Interactions: Real-time interaction of 10+ vision-based agents👩‍👩‍👧‍👧 in one scene.

• The Unreal Engine Environments (Binary) contains the scenes and playable entities.
• The UnrealCV+ Server is built in the UE binary as a plugin, including modules for rendering , data capture, object/agent control, and command parsing. We have optimized the rendering pipeline and command system in the server.
• The UnrealCV+ Client provides Python-based utility functions for launching the binary, connecting with the server, and interacting with UE environments. It uses IPC sockets and batch commands for optimized performance.
• The OpenAI Gym Interface provides agent-level interface for agent-environment interactions, which has been widely used in the community. Our gym interface supports customizing the task in a configuration file and contains a toolkit with a set of gym wrappers for environment augmentation, population control, etc.

• UnrealCV
• Gym
• CV2
• Matplotlib
• Numpy
• Docker(Optional)
• Nvidia-Docker(Optional)

We recommend you use anaconda to install and manage your Python environment. CV2 is used for image processing, like extracting object masks and bounding boxes. Matplotlib is used for visualization.

It is easy to install unrealzoo-gym, just run

git clone https://github.com/UnrealZoo/unrealzoo-gym.git
cd unrealzoo-gym
pip install -e . 

While installing gym-unrealcv, dependencies including OpenAI Gym, UnrealCV, numpy and matplotlib are installed.

Before running the environments, you need to prepare unreal binaries. To accommodate users' local memory limitations, we provide two different lightweight environment options based on UE4 and UE5. A larger preview-version environment package, which currently contain 50 scene maps (>60GB), is also available for download here. The full version of the UnrealZoo environment package will be released soon.

UE4 Example Scenes

Track Train	Greek Island
ContainerYard_Day	ContainerYard_Night
SuburbNeighborhood_Day	SuburbNeighborhood_Night

UE5 Example Scenes

ChemicalFactory	Modular Old Town
MiddleEast	Roof-City

Preview-version Environment Package
The preview-version environment package contains 60+ scene maps in total, the detailed map name can be found here EnvLists.

🗂️You can load the above binaries from the following link:

Then unzip and move the downloaded binary to the UnrealEnv folder, which is our default location for binaries, the folder structures are as follows:

gym-unrealcv/  
|-- docs/                  
|-- example/                
|-- gym_unrealcv/              
|   |-- envs/    
|   |   |-- agent/     
|   |   |-- UnrealEnv/                    # Binary default location
|   |   |   |-- Collection_WinNoEditor/   # Binary folder
|   |-- setting/
|   |   |-- env_config/                   # environment config json file location  
...
generate_env_config.py                    # generate environment config json file
...

If there is a permission issue, run the command chmod +x ./ under the UnrealEnv folder path to give the necessary permissions.

We have predefined a naming rule to launch different environment maps and their corresponding task interfaces.
Unreal{task}-{MapName}-{ActionSpace}{ObservationType}-v{version}

• {task}: the name of the task, we currently support: Track,Navigation,Rendezvous.
• {MapName}: the name of the map you want to run, track_train, Greek_Island, etc.
• {ActionSpace}: the action space of the agent, Discrete, Continuous, Mixed. (Only Mixed type support interactive actions)
• {ObservationType}: the observation type of the agent, Color, Depth, Rgbd, Gray, CG, Mask, Pose,MaskDepth,ColorMask.
• {version}: works on track_train map, 0-5 various the augmentation factor(light, obstacles, layout, textures).

• If your mouse cursor disappears after the game launches, press ` (the key above Tab) to release the mouse cursor from the game.

User could choose a map from the available map list, and run the random agent to interact with the environment.

python ./example/random_agent_multi.py -e UnrealTrack-track_train-ContinuousColor-v0

python ./example/tracking_demo.py -e UnrealTrack-Greek_Island-ContinuousColor-v0

Use the "I", "J", "K", and "L" keys to control the agent's movement.

python ./example/keyboard_agent.py -e UnrealTrack-Old_Town-MixedColor-v0

Use the "I", "J", "K", and "L" keys to control the agent's movement.
Space( ␣ ): Jump.
Up Arrow(⬆️): Look up.
Down Arrow(⬇️): Look down.
(Double "Jump" will trigger the agent to climb)

python ./example/keyboard_Navigationagent.py -e UnrealNavigation-Demo_Roof-MixedColor-v0

Control the agent to navigate to the target location by using the keyboard.

Map Reference(Demo_Roof)	Target Reference(Demo_Roof)
Map Reference(SuburbNeighborhood_Day)	Target Reference(SuburbNeighborhood_Day)

You can build your agent based on the latest method in reinforcement learning, evaluating it in UnrealZoo.

• Reinforcement Learning

  • Online RL

    • Visual Navigation Agent comming soon..

  • Offline RL

    • Embodied Visual Tracking Agent(ECCV24)
• Large Vision-Language Model

• We provide a set of wrappers for FPS control, population randomization, etc. Please refer to the wrapper for more details.
• We provide a document for adding a new environment in unrealzoo-gym. Please refer to the addEnv for more details.

• Release an all-in-one package of the collected environments
• Add gym interface for heterogeneous mutli-agent co-operation.
• Expand the list of supported interactive actions.
• Add more detailed examples for reinforcement learning agents.
• Add more detailed examples for large vision-language models.

The UnrealZoo project is licensed under the Apache 2.0. We acknowledge the following projects for their contributions to the UnrealZoo project:

• UnrealCV
• OpenAI Gym
• Unreal Engine
• The fantastic Plugins/Content from UE Marketplace
  • Smart Locomotion
  • Animal Pack
  • Drivable Car
  • ...

If you use UnrealZoo in your research, please consider citing:

@misc{zhong2024unrealzooenrichingphotorealisticvirtual,
      title={UnrealZoo: Enriching Photo-realistic Virtual Worlds for Embodied AI}, 
      author={Fangwei Zhong and Kui Wu and Churan Wang and Hao Chen and Hai Ci and Zhoujun Li and Yizhou Wang},
      year={2024},
      eprint={2412.20977},
      archivePrefix={arXiv},
      primaryClass={cs.AI},
      url={https://arxiv.org/abs/2412.20977}, 
}