How to correctly get the 3D Joints position

[vittorione94]

Hello,

I'm trying to extract the 3D joints positions using PHALP and 3D Human Body Prior (link), to later display them in a physics engine.

This is a sample code of how I'm doing this for just a single frame.

from human_body_prior.tools.rotation_tools import matrot2aa
from human_body_prior.body_model.body_model import BodyModel
from human_body_prior.tools.omni_tools import copy2cpu as c2c


results = joblib.load(file_path)
bm_fname = "body_models/smplh/neutral/model.npz"
k = 0 
body_pose = results[k]['smpl'][0]['body_pose']
global_orient_euler = matrot2aa(torch.Tensor(results[k]['smpl'][0]['global_orient']).to(comp_device))
eulers = matrot2aa(torch.Tensor(body_pose[:-2,::]).to(comp_device))

num_betas = 10  # number of body parameters
body_parms = {
        'pose_body': 
            torch.unsqueeze(eulers.flatten(), axis=0), # controls the body
        'betas': 
            torch.Tensor(np.repeat(results[k]['smpl'][0]['betas'][:num_betas][np.newaxis], repeats=1, axis=0)).to(comp_device), # controls the body shape. Body shape is static
        'root_orient': 
            torch.unsqueeze(global_orient_euler.flatten(),axis=0)
        }

body_pose = BodyModel(bm_fname)(**{
                k: torch.Tensor(v)
                for k, v in body_parms.items()
                if k in ['pose_body', 'root_orient', "betas"]
            }
        )
SMPLX_JointPos = c2c(body_pose.Jtr)

If I use the code as is the joints location is correctly displayed but it's not aligned with the engine I'm using, and no matter what kind of transformations I'm using I can't align it :(

A few things to note:

1. I need to transform body_pose and global_orient to euler angles otherwise human body prior won't work.
2. I have no idea what's the frame used to express the rotations in PHALP. Meaning what are the up/down, forward/backward and right/left axis and their direction. In the physics engine I'm the reference frame I'm using is +x right, -x is left, +y is up, -y is down, +z is forward and -z is backward.
3. Also human body prior needs a 'trans' attribute for the root translation otherwise the joints locations move in place, do you have any insights on how to get this value?

Any help would be very appreciated!

Cheers,
-Vittorio

[wusar]

You can follow the code of the visualizer:

PHALP/phalp/visualize/py_renderer.py

Lines 108 to 122 in 677074a

	baseColorFactors = np.hstack([color[:, [2,1,0]], np.ones((color.shape[0], 1))])
	fl = self.focal_length
	verts = vertices.copy()
	cam_trans = camera_translation.copy()
	verts = verts + cam_trans
	color = self.__call__(verts, focal_length=fl, baseColorFactors=baseColorFactors)
	valid_mask = color[:,:,3:4]
	if(use_image):
	output_img = color[:, :, :3] * valid_mask + (1 - valid_mask) * images
	else:
	output_img = color[:, :, :3]
	return output_img, valid_mask

In Phalp, the body is centered at the origin, so the camera translation is the same as the global translation, the ’trans‘.