Create a new virtual character

Virtual Agents

The virtual agents used in Greta, are embodied in a humanoid 3D model. These models are created using the Autodesk Character Generator online application. The 3D models in * .fbx format obtained, are then integrated into GretaUnity. The original models present unrealistic/unaesthetic physical characteristics, therefore new visual aspects using realistic rendering methods are needed to give a more credible look determined by its quality, realism and accuracy. In Addition, a methodical procedure is created to be suitable for the creation of new characters.

Virtual Agent Components in Unity

Before enumerating the steps to create a new virtual character, we should understand the 3D character’s components in Unity. A 3D animated character is composed of a polygon mesh that determines the shape of the object and the virtual skeleton that determines the movements. The mesh should be attached to the bones and form a ‘skin’ so it can be animated.

1- Rigged/Skinned Mesh

The format *.fbx of the character contains the following mesh elements:

• h_DDS_HighRes → the body mesh

• h_L_eye → left eye mesh

• h_L_gland → left gland mesh

• h_L_trans → left transparent eyecoat mesh

• h_R_eye → right eye mesh

• h_R_Gland → right gland mesh

• h_Wig → wig mesh ( only for some characters that have a wig mesh detached from the body mesh)

• h_TeethDown → Lower part of the teeth mesh

• h_TeethUp → Upper part of the mesh

• Master → skinned mesh. It contains the hierarchical bone structure → all the bones of the character’s mesh that can be used for the animation of the character In each character of Autodesk Character Generator, the annotation of the components depends on the resolution exported (h and HighRes for the high resolution, m and medRes for the medium resolution)

2- Materials

Each polygon of a mesh has a material assigned to it, and each material uses several textures. A texture is a bitmap implemented on the surface of an unpainted model, and sets information on its color, smoothness, roughness, illusion of relief.. ‘Texture mapping’ is the technique used for texturing a model. The "texture mapping", also known as "UV mapping" is a method, in the field of 3D modeling, which consists of applying a plane image (2D texture) on the surface of a volume model (3D). The 3D model is cut into several parts which are assembled flat, on a 2D plane texture. In Autodesk Character Generator, the characters have the same texture mapping, but each character has its own bitmaps.

Three texture maps are generated with the character:

• Color Map: texture to determine the color information
• Normal Map: texture to determine the illusion of relief
• Specular Map: texture to determine the shininess

New character materials in Unity

Originally, all the mesh polygons have one material. Since not all the elements of a 3D virtual agent have the same surface properties ( example: clothes are not smooth as the skin), the polygons will be assigned to different materials according to the surface identity using Blender or any other 3D modeling software (see the steps). The polygons that correspond to the eye will have a different material that the polygons of the skin or of the clothes. The new character in Unity has five main materials: Since the main mesh contains 3 different surface properties ( skin, shirt, pants) in the same element ( body element , H_DDS_Highres) , 3 different materials will be stuck in a materials list for this element.

1- Body Material

this material should be associated to the following components (H_DDS_Highres, H_L_gland, h_R_gland, h_teethDown,H_teethUP) The material is an HDRP lit shader material edited using shadergraph. It uses subsurface scattering as a material type. Eighth textures are included:

• Base Map: color map that defines the color of the texture without any diffuse reflection

• Smoothness map/ Mask Map/or Roughness Map: the map that defines the shininess of the surface, it describes the surface irregularities that cause light diffusion

• Normal Map: Common map that includes the wrinkles on the face, scratches on the mouth...

• Second Normal Map : Normal Map that is generated from Autodesk Character Generator

• Thickness Map : it is used as a mask for subsurface scattering

• Detail Map : normal map for the skin details pores (the texture is taken from the digital human tech package)

• Smoothness Detail Map: smoothness of the detail map

• BlendColorMap: face tonality colors that are blended with the main color map (one for the female character, one for the male character)

  The skin has different color zones, an optimized visual result is perceptible after adding three colors: - Yellow: on the zones closest to the bone, areas devoid of muscles and red blood cells, like the forehead, collarbones - Blue: On places where microscopic hairs grow and where the veins are more prominent, such as under the eyes and mustaches for men. - Red: On places, where cells, arteries and muscles can be seen, like the cheeks, lips, nose and eyelids

  Since all the characters have the same UVmaps, we painted a new texture that corresponds to the zolor tones of the facial skin, and added a transparency so it will be blended on top of the main bitmap texture as seen in the image below

The maps are found in the (Assets/Resources/Textures) except for the color base map and the first normal map that have to be integrated depending on each character.

In HDRP, there are materials that use subsurface scattering, a mechanism in which the light that penetrates a translucent surface, is scattered and leaves the surface at a different point. It is used especially to improve the quality of skin shading. An asset, called the diffusion profile, is used to store most of the subsurface scattering settings and it is included in the shader ("Skin_diffusionProfile" qui se trouve dans (Assets/Resources/DiffusionProfiles)

2- Clothes Materials

Two materials are generated using a clothes shader that we created using Unity's ShaderGraph system, one for the shirt and the other for the pants.

The shader consists of the following elements :

• ColorMap: the texture that defines the color information and the pattern of the fabric.

  • Tiling Color Map values → to tile the UV map used for scrolling textures to set the scale of the color map
  • Offset Color Map values → to offset the uv map to set the appropriate position
  • Color Map Rotation values → to rotate the uv map if it is necessary

You can change manually the texture of the fabric by using the texture samples in 'Assets/Resources/Textures/Clothes' and you can change the tiling and offset property accordingly.

• The color of the map : to add a color to the base color map. It is recommended to use a whit colormap so the final color will be the same as the color input value and not affected by another color .

• First Normal Map: normal map for the fabric texture

  • NormalStrength: value of the strength of the normal map
  • Tiling First Normal Map
  • Offset First Normal Map

• Second Normal Map : normal map for the clothes wrinkles

  • NormalStrength
  • Tiling First Normal Map
  • Offset First Normal Map
  • Rotation

• Smoothness

• Smoothness map

3- Eyes Material

Two Materials are generated, one for each eye. A sample for each material "M_LeftEyes" for the left eye, and "M_RightEyes" for the right eye, can be found in'Assets/Resouces/Materials'. They both have the same textures and properties but different offsetting values.

• The color map determines the color of the eyes. We have included in the 'Assets/Resources/Textures/Eyes', five different colored textures ( blue, brown, gray, green, hazel). You can change the textures manually if you're creating a new character, the eyes are created automatically with all the color options.

4- Cornea Material

This material, named 'M_TransEyes' is common for all the characters. It should be assigned to ‘h_L_Trans’ and ‘h_R_trans’ elements which are the outer coat of each eye. It is situated ‘Assets/Resources/Materials’.

The only property of this material is its smoothness which represents the reflection. If you’re not satisfied with the reflection effect, you can change the smoothness value in the materials properties.

Steps to create new virtual characters

To create new virtual characters more realistic than the ones used in Greta platform that uses Ogre3D player as a rendering engine, we will be using The High Definition Render Pipeline (HDRP) with Unity , that creates high fidelity visuals. Skin setups for the characters are recreated to be able to adapt to all the characters of Autodesk Character Generator. Some textures are retouched and new ones are generated to give a more realistic look.

Content

• 1. Create the character
• 2. Add new material IDs using Blender
• 3. Add the new character in Unity

1. Create the character

Follow the first step of creating a new character using Autodesk Character Generator, Create a new virtual character.

You can customize your character’s body, face, clothes and hair.

**Note : ** It is recommended to choose clothes that are not tight on the body, to be able to assign new materials to them later on. When you export the model, make sure to export all the following textures (clothes, normal). As a result, a customized rigged 3D character will be generated.

2. Add new material IDs using Blender

The textures applied to the Autodesk character generator 3D model, use fake reflections and lack some realism. The clothes for example that are tight on the body, need to have a small thickness in order to be more believable. The same material is applied on the skin and the clothes, however they have different material properties, so it is better if we created new materials for each element that we can later customize in Unity. We will be using [Blender] as an open source software to create the new material IDs.

2.1. Import into Blender

Import the .fbx into Blender and delete the box that shows up in the startup scene. **Note : ** The bones are usually imported in the wrong direction for an unknown reason. Don't change anything related to the bones as they will get back to the right position once you export the *.fbx back to unity. Also for some models, the bones of the right hand are twisted. You have the option to change the rotation in Blender, but in this documentation we will be using an editor script that fixes the bone rotations, so it is okay if the bone orientations are wrong. User Tip in Blender: You have the option to hide the armature on the top right of the screen in order to visualize the 3D model more clearly.

2.2. Add the materials

We will add two new materials , "_shirt" and "_Pants".

• object mode --> select the model’s mesh
• Material properties on the right :
  • click on the plus button to add a new material
  • click on "+ new" to create the new material and rename it "_shirt"
• Follow the same steps for the second material and name it "_pants"

2.3. Assign the new materials to the appropriate polygons

• If you’re in the object mode , select the character's 3D mesh then select edit mode to be able to select the polygons of the mesh
• Edit mode → select faces → start selecting the faces that correspond to the character’s shirt. The color of the face will turn orange when you select it.
• If you want to have an extrusion effect for the clothes: select the polygons that you want to be extruded by having the face selection turned on --> face --> extrude face and choose visually the amount of the extrusion. Then to have a smooth effet (not edgy), select the new edges of the border and bevel them : ctrlB to bevel and with the mouse wheel to add segments.
• In Materials Properties , select the shirt material and click assign to assign the polygons selected to the shirt material newly created **Note : ** To visualize more clearly the effects: change the base color of the material in the materials property and select the viewport shading as a material preview. If some of the polygons were not selected right, select the missed ones and reassign the material.
• Follow the same steps for the pants

3. Add the new character in Unity

If you are not in the HDRP (High Definition Render Pipeline) mode, you will need to switch to it. Open the Graphics Settings window (menu: Edit > Project Settings > Graphics)

3.1. Create your new character's folder

• Create a new folder in the characters folder after your character's name in the Assets/Models/Characters.
• Import the new mesh's .fbx to the folder
• Create a new "Textures" folder inside the new folder
• Select the .fbx --> inspector --> materials --> extract textures and choose the textures folder newly created

Your new character’s folder must now contain the .fbx , and two folders : Textures, Materials

3.2. Generate the new character

An editor script called 'charactermanager' is used to generate the new character. It has options like :

• generating the new character: to create new materials and associate them to the right meshes
• Rescale Hands: to rescale the hands that are disproportionate
• Fix the right hand's bones : to fix the incorrect rotation of the bones that are imported from blender
• Add wig : add a new wig to the female characters
• Create Prefab: set the finished model in a prefab asset so it will be used for other scenes.

The following image presents the different functions of the script and the inspector of the 'characterGenerator'

Steps:

use the prefab FemaleCharGeneratorSample in your scene.

OR

Alternatively, open the 'TestNewCharacter' Scene in Unity. In the hierarchy, there's an empty game object called "Generators" and several children with a "CharacterGenerator" component, go to one of them.

In its inspector and edit the following properties of the script:

• add the character's name
• drag the .fbx to the character's folder
• drag the _color map from the textures folder to the color map
• drag the _normal map from the textures folder to the normal map
• choose the pants's color
• choose the shirt's color
• choose the color of the eyes
• choose the model gender
• click on 'generate the new character'

New materials (body , left and right eyes, pants and shirt) will be added in the materials folder and the transparent materials will be automatically assigned to the eyes. You can modify some of the properties values of the materials if you're not satisfied with the result, like changing the clothes wrinkles rotation , changing the clothes texture .. etc.

• click on rescale hands if you find that the hands are not proportional
• click on fix the right hand's bones if you see that the right hand's bones are rotated
• click on add wig for female characters if you want to change the hair model and add a new one. The new hair model might not be positioned in the right place, it has to be repositioned manually according to the character's head position and scale.

Once you're happy with the results , click on create the prefab and your new character will be in the prefabs folder so that it can be used in any Unity project.