GretaUnity

Table of Contents GretaUnity: Integration of Greta/VIB in Unity3D Pre-requisites Systems Setup Java Environment Setup Unity3D Project Setup VIB/Greta Run the Demo Controls Demo FMLs Current Limitations

GretaUnity: Integration of Greta/VIB in Unity3D

This page contains the instructions to setup and run the Unity3D project GretaUnity-Simple for development.

The GretaUnity-Simple project in Unity is connected to Greta/VIB via thrift.

The BAPs and FAPs animation parameters produced in VIB are sent to Unity and a script named CharacterAnimation.cs attached to the character animates the joints accordingly.

This is a simple starting point for future developments of the Greta-Unity integration. See here a list of limitations that needs to be addressed.

Pre-requisites

• Unity3D (version 4.6, currently not working with 5): Unity Download
• Tortoise SVN: Tortoise Download
• OpenMaryTTS
• VIB

Systems Setup

Java Environment

Make sure that you can run Java programs (needed for running VIB) and that your system "PATH" variable has the Java paths all set (to run Java from console)

Setup Unity3D Project

• Install Unity3D
• Check-out the unity project from our repository at the following link: GretaUnity-Simple SVN Checkout
• With TortoiseSVN on Windows, right-click in a folder for the desired project location on your hard drive->SVN Checkout...->Insert the svn link provided above and your credentials when asked
• Open the checked out project with Unity (in the Unity dialog select the folder where you have checked out the project)
• Using TortoiseSVN, unversion (i.e. add to the ignore list) the Library folder in the unity project folder

Setup VIB/Greta

You only need to run Modular and open the configuration file located at: /bin/GretaUnity-Simple.xml

Run the Demo

In order to run the demo you need to follow, in this exact order, the following steps:

1. Run OpenMaryTTS: \MARYTTS431\bin\maryserver.bat (if it is the first time you used Open MaryTTS look the wiki page Installing the speech syntetizer)
2. Lunch Modular (with the given configuration for Unity)
3. Open and Play, in Unity3D.
   1. You can Gadd the character in the unity scene drugging and dropping the character in the folder Asset/Prefabs/Character.
   2. Once added the character look at right, to the inspector (click on the character on the scene if you don't see anything) and click on Add_component/Script/CharacterAnimation_Single_Autodesk. In this way a new window will be added in the inspector allowing you to check the thrift port numbers.
   3. Look to thrift port numbers and check that, in the modular configuration you opened before, the thrift port numbers for commandReceiver, audio, FAP and BAP are the same in both Modular and Unity. If not, you can change them in Unity or modular.
   4. Once you added the character in the unity scene and checked the thrift port number, you can click play in Unity, send fml or bml file to greta and see the agent move.

Controls

• WASD/Arrows: Movement
• Mouse: look around
• Q or E: Climb or Drop Camera
• Shift / CTRL: Move slower / faster
• End: Toggle cursor locking to screen
• ESC: quit (only when running the compiled version)
• R: reset camera position

Demo FMLs

Using the FMLSender in modular you can play the following demo FML files located at /bin/Examples/DemoEN/:

• Welcome.xml
• SeeYou.xml

Current Limitations

• Idle movements generated in VIB are not blended with gestures, currently the character is using the Unity/Mechanim animation for this, but the legs are freezed (i.e. BAPs received by VIB are ignored and idle animation is displayed instead)
• Speech is synchronized with lips movements, but the produced audio need to be pre-created using Modular and then the corrensponding audio file must be stored in the Unity3D project folder at: ..\GretaUnity-Simple\Assets\Resources, this means that the character's voice is not streamed in real-time via thrift (as for the animation parameters), instead it needs to be pre-recorded and when an FML file is requested/sent from/to unity a name matched file needs to be found
• The only available characters are Camille and Alice. Currently only Camille is included in the assets. If a new character is needed the rig needs to be done in order to work with the CharacterAnimation.cs script and correctly perform the animation parameters received via thrift.