The ReprGesture entry to the GENEA Challenge 2022

For each of the modality, the pipeline of extracting features is as follows:

• Text: We first use FastText (Bojanowski et al., 2017) to get the word embeddings. Padding tokens are inserted to make the words temporally match the gestures by following (Yoon et al., 2020). One-dimensional (1D) convolutional layers are then adopted to generate 32-D text feature sequence $U_t$ (‘$t$’ for ‘text’) from the 300-D word embeddings.

• Audio: All audio recordings are downsampled to 16kHz, and features are generated from the pre-trained models of WavLM Large (Chen et al., 2021). We further adjust sizes, strides and padding in the 1D convolutional layers to reduce the dimension of features from 1024 to 128 forming the final audio feature sequence $U_a$ (‘$a$’ for ‘audio’).

• Gesture: Due to the poor quality of hand motion-capture, we only use 18 joints corresponding to the upper body without hands or fingers. Root normalization is used to make objects face the same direction. We apply standard normalization (zero mean and unit variant) to all joints. Seed gestures for the first few frames are utilized for better continuity between consecutive syntheses, as in (Yoon et al., 2020). On top of these, position and 3 × 3 rotation matrix features are computed, and the size of final gesture sequence $U_g$ (‘$g$’ for ‘gesture’) feature is 216.

First, for each modality $m\in \{t,a,g\}$, we use a linear layer with leaky ReLU activation and layer normalization to map its feature sequence $U_m$ into a new feature sequence $u_m\in R^{T\times d_h}$ with the same feature dimension $d_h$. Then, we project each sequence $u_m$ to two distinct representations: modality-invariant $h_m^c$ and modality-specific $h_m^p$. Afterwards, $h_m^c$ learns a shared representation in a common subspace with distributional similarity constraints (Guo et al., 2019). $h_m^p$ captures the unique characteristics of that modality. We derive the representations using the simple feed-forward neural encoding functions:

Encoder $E_c$ shares the parameters ${\theta }^c$ across all three modalities, whereas $E_p$ assigns separate parameters ${\theta }_m^p$ for each modality.

Domain learning can improve a model’s ability to extract domain-invariant features (Bousmalis et al., 2016). We use an adversarial classifier to minimize domain loss that reduces the discrepancy among shared representations of each modality.

The domain loss can be formulated as:

where $D_{repr}$ represents feed-forward neural discriminator, $d_m$ represents the result after gradient reversal of $h_m^p$.

The modality reconstruction loss $L_{\text{recon}}$ is computed on the reconstructed modality and the original input $u_m$. The $L_{\text{recon}}$ is used to ensure the hidden representations to capture the details of their respective modality.

Specifically, a modality decoder $D$ is proposed to reconstruct $u_m$:

where ${\theta }^d$ are the parameters of the modality decoder. The modality reconstruction loss can then be computed as:

where $\parallel \cdot {\parallel }_2^2$ is the squared $L_2$-norm.

We use generative adversarial network (GAN) based gesture decoder for generating gestures. Gestures are directly related to rhythm and beat, thus we concatenate audio rhythm related features (pitch, energy and volume) and the output of six stacked modality representations together and send them to Transformer encoders with multi-head self-attention as the generator, as shown in Figure 2. The generator part is trained using $L_{gesture}$ consisting of the Huber loss and the MSE loss, and the discriminator part is trained with $L_{GAN}$.

where $D_{gesture}$ represents gesture discriminator using multilayered bidirectional gated recurrent unit (GRU) (Cho et al., 2014) that outputs binary output for each time step, $t$ is the length of the gesture sequence, $g_i$ represents the $i$th human gesture, $\widehat{g_i}$ represents the $i$th generated gesture.

The loss of the proposed system can be computed as:

In the challenge, the Talking With Hands 16.2M (Lee et al., 2019) is used as the standard dataset. Each video is separated into two independent sides with one speaker each. The audio and text in the dataset have been aligned. For more details please refer to the challenge paper (Yoon et al., 2022). We note that the data in the training, validation and test sets are extremely unbalanced, so we only use the data from the speaker with identity ”1” for training. And we believe that if speech and gesture data are trained on the same person, the gesture behavior would match the speech.

The proposed system is trained on training data only, using the ADAM (Kingma and Ba, 2014) optimizer (learning rate is e-4, ${\beta }_1$ = 0.5, ${\beta }_2$ = 0.98) with a batch size of 128 for 100 steps. We set $\alpha =300$, $\beta =50$ for Equation (5) and $\gamma =5,\delta =0.1,ϵ=0.1$ (we noticed in our experiments that too large $\delta$ and $ϵ$ will lead to non-convergence) for Equation (7). There is a warm-up period of 10 epochs in which the $L_{GAN}$ is not used ($\gamma$ = 0). The feature dimension $d_h$ of sequence ${\text{u}}_m$ is 48. During training, each training sample having 100 frames is sampled with a stride of 10 from the valid motion sections; the initial 10 frames are used as seed gesture poses and the model is trained to generate the remaining 90 poses (3 seconds).

In this evaluations, study participants are asked to rate “How human-like does the gesture motion appear?” on a scale from 0 (worst) to 100 (best). Bar plots and significance comparisons are shown in Figure 3(b). Our system (USN) receives a median score of 44 and a mean score of 44.2, and is ranked fourth among the participating systems.

In this evaluation, participants are asked to choose the character on the left, on the right, or indicate that the two are equally well matched to response “Please indicate which character’s motion best matches the speech, both in terms of rhythm and intonation and in terms of meaning.” Bar plots are shown in Figure 4. Our system (USN) receives a “Percent matched” 54.6, which identifies how often participants preferred matched over mismatched motion in terms of appropriateness. Our system is rated seventh in appropriateness among the participants’ submissions. It should be noted that the difference of our system to the five higher-ranked systems (USL, UBA, USO, USK and USJ) is not significant. Furthermore, if we only consider the ratio of matched motion, i.e., the blue bar in Figure 4, our system is ranked fifth among the participating systems.