Basic Configuration
- FML File Reader
- Behavior Planner
- Behavior Realizer
- Keyframe Performers
- MPEG-4
- Environment
- Editors
- Character Manager
- INI Manager
We can explain the role of each module by following the workflow that FML and BML files take to be processed and to be translated into real verbal and non-verbal behaviours. The workflow can start from a very high level, like the FML file, or from lower level, BML file. The FML represents what an agent wants to achieve: its intentions, goals and plans. The BML language allows us to specify the signals that can be expressed through the agent communication modalities (head, torso, face, gaze, body, legs, gesture, speech, lips).
It takes as input the XMLtree file in FML format then translate it in a list of intentions. The FML file is divided in two part (see Figure 3): the first is the BML that can contain the speech signal, boundary signals and pitch accents; the second part is the FML that contains the intentions (emotion, certainty, laugh, performative, deictic, etcetera). The list of intentions is sent to the following module, the BehaviorPlanner.
Once the module receives the list of intentions, they are temporized and sorted by importance, starting time and duration. For each intention, the corresponding behavior-set in extracted from the default or character specific lexicon. The behavior-set can contain different signals (HeadSignal, Gesture, FaceSignal, TorsoSignal, etc.) that are extracted and added to the list of signals sent to the next module, the BehaviourRealizer.
The BehaviorPlanner module allows to update the Agent lexicon without restarting the platform. Once selected the BehaviorPlanner module, a window will appear containing a button to update the character lexicon.
This module receives the list of signals sent by the BehaviorPlanner or the “BMLFileReader” module as input. The BMLFileReader takes as input, the BML file that contains the text to be spoken and/or a set of nonverbal signals to be displayed. All signals like facial expressions, gaze, gestures, torso movements are described symbolically in repository files (libraries). The agent's speech, which is also part of the BML input, is synthesized by an external TTS system (TTS Module). The TTS system provides the list of phonemes and their respective duration. This information is used to compute the lip movements. Once received the signals’ list, the BehaviorRealizer follows some steps to translate the signal in keyframe:
- To calculate start and end times for each signal;
- To sort the signals;
- To identify which phase is realized in each signal and when it is realized. In this way the signals can be scheduled independently;
- The phases are translated into keyframes;
- Sort the keyframe and store them in a list;
- Sendeach keyframe to the correspondent KeyframePerformer (i.e. HeadKeyframe are sent to HeadkeyframePerformer, TorsoKeyFrame to TorsoKeyframePerformer, etcetera).
The Behavior Realizer solves eventual conflicts between the signals that are scheduled to happen on the same modality at the same time. It uses repository files of predefined facial expressions, gestures, torso movements and so on.
The BehaviorRealizer module allows to update the Agent libraries without restarting the platform. Once selected the BehaviorRealizer module, a window will appear containing buttons to update the face, gestures, head and torso libraries.
The keyframes’ list contains different kind of keyframe: AUKeyframe (for face expression and gaze), BodyKeyframe (for Head, Torso, Shoulder and Arms gestures), SpeechKeyframe. There are sent to the correspondent Keyframe performer (i.e. BodyAnimationPerformer, FaceKeyframePerformer, LipModel, AudioPerformer). Each keyframe is translated as FAP ( Face Animation Parameter) or BAP (Body Animation Parameter) keyframe or audio and then sent to the MPEG-4 module.
The SimpleAUKeyframePerformer module allows to update the AUlibraries without restarting the platform. Once selected the module, a window will appear containing a button to update the character AU library.
This module implements the MPEG-4 Face and Body Animation (MPEG-4 FBA) [ISO14496] International Standard for the users of Visage Technologies software. It provides the details on the Face Animation Parameters (FAPs) and Body Animation Parameters (BAPs). For the face, the MPEG-4 specification defines 66 low-level Face Animation Parameters (FAPs) and two high-level FAPs. The low-level FAPs are based on the study of minimal facial actions and are closely related to muscle actions. They represent a complete set of basic facial actions, and therefore allow the representation of most natural facial expressions. Exaggerated values permit the definition of actions that are normally not possible for humans, but could be desirable for cartoon-like characters. For the body, there are 196 Body Animation Parameters (BAPs). BAP parameters are the angles of rotation of body joints connecting different body parts. These joints include: toe, ankle, knee, hip, spine (C1-C7, T1-T12, L1-L5), shoulder, clavicle, elbow, wrist, and the hand fingers. The hands are capable of performing complicated motions and are included in the body hierarchy. There are totally 29 degrees of freedom on each hand, assuming that the hand has a standard structure with five fingers. The MPEG-4 module is the skeleton of the agent. Once the BAP or FAP keyframes are received, MPEG-4 module allows to actually move the character.
The Environment module is used to add characters and objects into a scene. This module creates a tree of the scene, where characters and objects are leaves. Each leaf contains 3 nodes for direction, rotation and scale. Because this tree is compatible with computer graphics standards, this environment can be used in different graphic engines like Ogre3D or Unity3D.
In the Greta System there are also modules that allow to edit gesture, libraries or the environment, like:
- Gesture Editor: it allows to create the own proper gesture choosing:
- Arm position: defines the gestures arm position by symbolic value gestures (see MecNeill sectors)
- hand shape: defines the hand shape key frame which is already defines in library
- hand orientation: defines the hand orientation by using different combination of directions
- Trajectory: defines the trajectory for arm gesture by adding new key frames on the path
- Openness: changes the gesture form (elbow space) in IK
- Expressive parameters: defines the expressiveness that can influence the variation of a whole gesture;
- Hand shape editor: it allows to chose the proper hand shape by editing the joints’ orientation of the fingers;
- Face Library Viewer: it allows to edit the all 62 face Action Unit (AU) and so create the own facial expression;
- AU Library editor: it allows to edit the position of the main face AU (just 16/62) by using a graphic panel where it is drawn the face and the small point indicating the position of AU;
- Environment editor: it allows to select all the object or agents displayed in the scene and change their position, scale and orientation. It is also possible to add new objects.
This module contains information (see Figure 5) about the characters that can be used by the system. The data provided is about the path where the lexicons, libraries or the voice, etc. are stored. All data contained in CharacterManager module are accessible by any of the other modules. There are two characters (Camille and DefaultCharacter) available on the GitHub version of the Greta platform.
This module contains information stored in “Greta.ini” file (see Figure 8). It displayed the info about all characters’ path file, external TTS software (MaryTTS and CereProc) and Ogre3D engine, etc. All data contained in INIManager module are accessible by any of the other modules.
