Generating New Gestures

Content

• Editing gestures
• Add Gesture to Your Output
• Adjust Gesture Expressivity

Editing gestures

In Greta, gestures are the movements executed with the arms.

To create a new gesture, use the Gesture editor module in the 'Tools' category (Add->Tools->Gesture Editor). If this doesn’t work for you, click on the large purple block (CharacterManager module) beneath most of the blocks in the Greta window and try again.

Connect the GestureEditor to the BehaviorRealizer and AnimatioKeyframePerformer. To do this, move the mouse over the block until you see a finger, then drag from this block to the block you want to connect to.

Click on the block "GestureAnimatorEditor". At the bottom of the main window, you'll see an embedded title bar. Click on the icon at the right and it will open up the Behavior editor dialog in its own window, as shown below.

Gestures are stored in either default file called BehaviorRealizer/gestuary.xml or character specific files such as gestuary_camille.xml. You can edit gestures directly by changing the entries in these files or edit them with the GestureAnimationEditor. You can also browse the animations using the GestureAnimationEditor.

Browsing gestures:

There are a large set of predefined gestures. To see these, click on the drop down in the upper right. Select a gesture that you’re interested in and then click “Send Gesture” (upper left) to play it.

See also: https://github.com/gretaproject/greta/wiki/Gesture

You can start editing the existing gesture:

• Select the gesture you want to edit in the upper right dropdown
• To play the gesture, click “Send Gesture”
• To edit it, select the tab for either the start or the end of the stroke (or other keyframes if defined). Change any parameters that you like. There are more details in the instructions for creating a new gesture below.

You can also create new gestures:

• File -> New Gesture
  • The resulting dialog will ask you for two fields:
    • Category: The category represents the type of gesture. It should be a single word, all lower case. Current categories are: adjectival, beat (small movements that accompany the rhythm of the speech), certainty, deictic (pointing), emotion, empty, iconic (representations of concrete forms), performative and rest. It is possible to define new categories
    • Name:
      • Enter a unique name for the gesture, following the naming convention:
        • nameofgesture_Pos/Ges_Hand_Adjective
        • nameofgesture is a name that encapsulates the gesture form
        • Pos are positions that include a single keyframe. Ges are gestures that contain two or more keyframes.
        • Hand is one or both hands ( R= Right, L= Left, B= Both hands)
        • Adjective is an (optional) adjective to differentiate similar gestures ( eg: Strong, Medium, Low, etc)

<gesture category="<categoryName>" id="<actualName>_<Ges/Pos>_<B/R/L>%_<adjective>%">
</gesture>

• The newly created gesture will be saved in <GRETA_DIR>/bin/BehaviorRealizer/gestuary.xml file. New gestures can be defined in this file using an XML description of the gestures.

• Navigation

  • To adjust your 3D view:
  • Rotate: left click on Greta and drag
  • Pan: right click on Greta and drag
  • Zoom: alt + right click and drag

Add a first keyframe (or phase)

Once you create your gesture, click a check box to enable the right and/or left arm. You will have three options:

• Uniform: This allows you to specify hand coordinates in Cartesian space, relative to the center of mass. (Use this for the exercise.)
• Symbolic: This allows you to use a symbolic representation of gesture space developed by McNeill (https://github.com/gretaproject/greta/wiki/Gesture-Editor)
• TouchPoint: This allows you to specify positions on the body that the gesture should touch

Adjust the wrist position and orientation as needed for the first frame of your gesture. For orientation, Y is the axial rotation of the forearm, X is left-right bend of the wrist and Z is up-down bend of the wrist

• You can see the effect of the parameters by clicking on “Send Current Keyframe”
• You can play the entire gesture by clicking on “Send Gesture”
• The fixed and invariant options are for more complex functionality related to how gestures are chained together. Don’t worry about them for this tutorial.

The Openness slider allows you to adjust the swivel angle of the arm, rotating the elbow in towards the body, or out from the body.

You can change the shape of the hand with the HandShape drop down. You can also create new handshapes if needed: https://github.com/gretaproject/greta/wiki/Generating-new-Hand-configurations

To save your gesture, go File->Save Gesture You can save the gesture to the default gestuary, the gestuary for the current character or both. You will need the data in the file for the current character (at least).

N.B. when you save your gesture, it will no longer be the active gesture in the editor. You need to reselect it from the dropdown.

If you want a two handed gesture, go ahead and add in the other arm by checking the other Enable <Left/Right> Arm box

• If you want to mirror the two arms, you can copy and paste the values for the one arm into the other.
• The “Switch Arms” buttons flips the left and right arms.

Add a second keyframe

Gesture -> Add Keyframe

• This will create a second keyframe and has the nice feature of setting the values for this to be the same as the first, so you only need to adjust the values you want to vary. You will see “Stroke Start” and Stroke End” tabs. Pick the one you want to change and adjust the values as above.

To create more complicated gesture forms, you can add additional keyframes (optional)

The Expressivity Parameters are discussed in the “Adjust Gesture Expressivity” section below. They are not saved with a gesture, but you can modify them in the gesture editor to get an idea of what the gesture will look like if you later apply these values. You’ll want to experiment with them in the editor to determine how to set them below.

Save your gesture: when you are satisfied, File->Save Current Gesture

Examples :

<gesture category="iconic" id="count_Ges_R">
	<phase type="STROKE-START"></phase>
	<phase type="STROKE"></phase>
	<phase type="STROKE"></phase>
	<phase type="STROKE"></phase>
	<phase type="STROKE-END"></phase>
</gesture>

Add Gesture to Your Output

You can consider the gesture to be a way to communicate an intention of your agent. Intentions are represented in FML.

The first step is to create a new intention that uses your gesture. Go to BehaviorPlanner/IntentionLexicon and open the XML file for your character (e.g. lexicon_Camille.xml). Add an XML block like:

<behaviorset name="metaphoric-frame">
	<signals>
		<signal id="1" name="metaphoric=frame_Ges_B" modality="gesture"/>
	</signals>
	<constraints>
		<core>
			<item id="1"/>
		</core>
	</constraints>
</behaviorset>

The the example above is for a new gesture created and where the category is “metaphoric”. The name of the gesture is “frame_Ges_B”. In the top line, intention name frame (the “frame” part of the name “metaphoric_frame”).

To invoke this intention in your script, you can add an FML command like:

<fml>
	<metaphoric id="p1" type="frame" start="s1:tm2" end="s1:tm3" importance="1.0"/>
</fml>

It can be easiest to start with an existing XML file, like 1-Welcome.xml, and edit this to do what you want.

When you update the lexicon, you must reload it. Select the BehaviorPlanner widget, open its interface and click on update. To update the Gestuary, click on the BehaviorRealizer, open its interface and click on the appropriate update button.

Adjust Gesture Expressivity

The expressivity parameters (for more info go to the page Expressivity Parameters) allow you to adjust how a gesture sequence is performed in Greta to alter its style. We will only look at gesture here, but they can be applied to gesture, torso, facial expressions, etc. I

The parameters operate at two levels:

• There are parameters that are applied to all motion that is generated. These are called baseline ( see page Baseline).
• There are also parameters that are applied only to a particular intention (convey a concept, appear sad, etc.). These are the dynamaline ( see page Dynamicline) .

The final appearance of a gesture will be the result of applying the sum of the baseline and dynamaline expressivity parameters to it.

To modify these parameters, create a block for your intention in behaviorqualifiers.xml, for example (You only need to include the parameters you want to modify):

<qualifier name="metaphoric-frame">
		<modulation>		<destination>gesture</destination><parameter>SPC.value</parameter><operator>ADD</operator>
	<op1_name>gesture</op1_name><op1_attribute>SPC.value</op1_attribute><op2_value>1</op2_value>
		</modulation>
		<modulation>			<destination>gesture</destination><parameter>TMP.value</parameter><operator>ADD</operator>
	<op1_name>gesture</op1_name><op1_attribute>TMP.value</op1_attribute><op2_value>-1</op2_value>
		</modulation>
		<modulation>		<destination>gesture</destination><parameter>PWR.value</parameter><operator>ADD</operator>
	<op1_name>gesture</op1_name><op1_attribute>PWR.value</op1_attribute><op2_value>1</op2_value>
		</modulation>
		<modulation>			<destination>gesture</destination><parameter>OAC.value</parameter><operator>ADD</operator>
	<op1_name>gesture</op1_name><op1_attribute>OAC.value</op1_attribute><op2_value>1</op2_value>
		</modulation>
		<modulation>
	<destination>gesture</destination><parameter>FLD.value</parameter><operator>ADD</operator>		<op1_name>gesture</op1_name><op1_attribute>FLD.value</op1_attribute><op2_value>1.5</op2_value>
		</modulation>
		<modulation>
	<destination>gesture</destination><parameter>OAC.value</parameter><operator>ADD</operator>
			<op1_name>gesture</op1_name><op1_attribute>OAC.value</op1_attribute><op2_value>-.4</op2_value>
		</modulation>
		<modulation>
	<destination>gesture</destination><parameter>SPC.value</parameter><operator>LIM</operator>
	<op1_name>gesture</op1_name><op1_attribute>SPC.min</op1_attribute><op2_name>gesture</op2_name>
			<op2_attribute>SPC.max</op2_attribute>
		</modulation>
</qualifier>

These are the current baseline configuration values at the time of writing. You can find and alter them in baseline.xml :

<parameter-set name="gesture">
	<Parameter name="preference" value="0.9"/>
	<Parameter name="OAC" value="0.95"/>
	<Parameter name="SPC" value="0.00" min="-1.0" max="1.0"/>
	<Parameter name="TMP" value="0.05" min="-1.0" max="1.0"/>
	<Parameter name="FLD" value="-0.5" min="-1.0" max="1.0"/>
	<Parameter name="PWR" value="-0.5" min="-1.0" max="1.0"/>
	<Parameter name="REP" value="-0.1" min="-1.0" max="1.0"/>
</parameter-set>