Towards Practical Single-shot Motion Synthesis

We present the background of GANimator [28] and formalize the notations to set the stage for our improvements in the GAN training process.

Data representation. Li et al. [28] form a motion representation $\mathcal{M}_{T}\equiv\mathbb{R}^{T\times(JQ+C+3)}$, where $T$ indicates the number of frames, $J$ is the number of skeleton joints, $Q=6$ corresponds to the 6D rotation representation of the joints, and $C$ indicates the foot contact labels followed by the 3D representation of the root joint including the $x$- and $z$-axis velocity and the $y$-axis position.

GAN architecture. GANimator follows a coarse-to-fine motion feature learning approach, with $S$ stages of generators $G(\cdot)$ and discriminators $D(\cdot)$ pairs. The GAN model does not train all stages in an end-to-end manner, but follows a gradual learning approach, as shown in Fig. 2. The stages are groups of pairs of $\{G(\cdot),D(\cdot)\}$ forming $4$ pyramid levels $L_{*}$ - except for $L_{4}$ that contains only $S_{7}$. Each level’s stage learns to generate the motion features of increasing temporal resolution. For the rest of the paper we use the subscript to denote stage levels and the superscript to denote the pyramid levels (e.g. $\hat{T}^{1}_{2}$ corresponds to the generated motion features from the second $G(\cdot)$ of the first pyramid level). Each generator $G(\cdot)$ and discriminator $D(\cdot)$ consists of 4 skeleton-aware convolutions [1] followed by leaky ReLU activations (except for the final convolution layer). As depicted in Fig. 3, $G^{1}_{1}$ is responsible for learning the mapping between the sampled noise and the motion representation denoted as $\hat{T}+{1}$, i.e. $\hat{T}^{1}_{1}=G^{1}_{1}(z_{1})$, while the rest $G^{*}_{\{2,\dots,S\}}(\cdot)$ form a hierarchical auto-regressive process that progressively upsamples the generated sequence:

where $z_{i}$ is sampled from an i.i.d. normal distribution $\mathcal{N}(0,I)$ and multiplied with an decreasing amplitude $\sigma_{i}$.

Losses. The model is trained with multiple losses for securing an equilibrium in the adversarial game between generators and discriminators, having the constraint that there is only a single sample to train the model. Both $G(\cdot)$ and $D(\cdot)$ are supervised with the Wasserstein variant from [13]:

where $\mathbb{P}$ denotes a learned distribution and $\tilde{T}_{i}=\alpha\hat{T}_{i}+(1-\alpha)T_{i}$ is a linear combination of the generated and ground truth motion features, respectively. The last term of the equation, i.e. the gradient penalty regularization, enforces Lipschitz continuity and stabilizes the training. To prevent mode collapse due to the single-sample training the $G(\cdot)$ are additionally supervised by an $L1$ reconstruction loss:

where $z^{*}_{i}$ denotes a predefined noise code (at different levels of amplitude $i\in\{1,\dots,S\}$) that is set to approximate the reconstruction of $T$. Lastly, a regularization loss is added to supervise the contact of the predefined set of joints, denoted as $\mathcal{C}$, with the ground:

where S denotes the skewed Sigmoid function that forces the output to be almost binary, and $\mathcal{V}$ is the velocity computed for the joints of $\mathcal{C}$ using forward kinematics.

Although each loss operates on a different part of the GAN training process, we can summarize them as:

where $\lambda_{adv},\lambda_{rec},\lambda_{con}$, and $\lambda_{GP}$ are responsible for weighting the contribution of each loss to the adversarial game of the single sample learning. Li et al. [28] discuss the contribution of each loss to the final result, while our focus is concentrated in the balancing between the losses.

One major advantage of the single-shot diffusion models vs single-shot GANs is the exploitation of mini-batch training. In fact, finding the equilibrium in the adversarial game of a single-shot GAN is challenging and the mode collapse is a common result when trying to set the batch size larger than $1$. As discussed in [35], in a data-driven adversarial game mini-batching helps the discriminator to understand when the generator produces samples of very low variation and avoid mode collapse due to this side information. In single-shot generation, the outputs of the generator are by definition minor variations of the single input sample, while the narrow receptive field of the patch-based discriminator and the reconstruction loss $\mathcal{L}_{rec}$ are responsible for preventing the mode collapse. Naturally, increasing the batch size counters the intuition that the narrow receptive field of the discriminator will prevent mode collapse, while the reconstruction loss tends to reduce the coverage by forcing the average of the batch to be similar to the input sequence. Since mini-batching critically improves the training time, we performed an ablation study on the weights of $\mathcal{L}_{rec}$ and $\mathcal{L}_{adv}$ in a quest for finding the correct combination that preserves the equilibrium in the adversarial game. However, note that this is not a straight-forward weight tuning process since each loss operates on different parts of the training process (i.e. different optimizer). Starting from the original weight values $\lambda_{adv}=1$ and $\lambda_{rec}=50$ we choose a stage-based linear annealing of: a) $\lambda_{adv}$, b) $\lambda_{rec}$, c) both $\lambda_{adv}$ and $\lambda_{rec}$. As shown in Table 1, we achieve the best results with (c) when we boost $\mathcal{L}_{adv}$ in the early stages where mapping from sampled noise to motion representation takes place, while the later stages are dominated by the reconstruction loss to ensure that no rare mini-motions are omitted.

Apart from increasing convergence through larger mini-batch training, the hierarchical structure of single sample GANs can be exploited to improve convergence. ConSinGAN [15] trained multiple stages in parallel and re-used the discriminator from the previous stage to improve performance by transferring its weights to initialize the next stage discriminator. GANimator [28] already trains two stages in the same pyramid level but does not perform discriminator transfer across stages or pyramid levels, even though the same discriminator is used – in terms of architecture and capacity - as in ConSinGAN and in contrast to SinGAN that increases discriminator capacity at each stage.

Curiously, re-using the generator weights does not lead to improved performance. To study this we turn to studying the neural network representations using finer grained layer-wise analysis. It has been shown that the similarity of representations across layers can be measured despite the higher dimensionality of the representations [23]. We perform linear centered kernel alignment (CKA) for all stage combinations across all pyramid levels.

Our results, depicted in Fig. 4, show that despite the hierarchical approach of using the same model and capacity for each stage, most layers exhibit low similarity scores. Yet we find that the early layer representations across the stages of each pyramid level, exhibit higher levels of similarity. Contrary to ConSinGAN that outputs features at each level apart from the last, GANimator reconstructs the output motion at each level. Therefore the early convolutional layers operate on similar features and, as indicated by their CKA scores, extract similar representations, whereas the latter layers apply the level’s motion motifs, style and details. Based on this analysis, we design a generator transfer learning scheme across levels and stages where each generator’s ($G^{i}_{j}$) early layer weights are initialized from the early layer weights of the previous level generator ($G^{i-1}_{j-2}$). This ensures that each next training stage is closer to the converged state allowing us to reduce the number of iterations significantly, further boosting training time.

Fist, we briefly report the metrics used in the literature for assessing the quality and the diversity of single-sample generation, present our implementation details, and discuss our results compared to similar models [28, 33].

Metrics. For a fair comparison we use the metrics presented in [28, 33], which try to measure the local and global diversity of the generated sequences, as well as their quality in terms of plausibility and coverage. We give a brief description for each metric, commenting on its usefulness.

Quality: The combination of the coverage and the plausibility expressed as distance from a distribution form a robust pair for understanding how realistic is the generated motion. Li et al. [28] consider a temporal window $T_{w}$ of the input sequence covered if its distance from its nearest neighbor $Q_{w}$ in the generated sequence is less than a predefined threshold $\epsilon$; on the other hand, SinMDM adopts the Fréchet Inception Distance (FID) variant from [34], which uses the deep features of an earlier convolutional layer of the Inception Network [41] in order to compute the FID statistics between the input and the generated sequences. As noted in [33], coverage has been experimentally shown to be sensitive, thus we choose to interpret it jointly with the single sample FID (SiFID) to truly describe quality.

Diversity: [28] and [33] compute the local and global diversity of the generated sequence following different approaches; Li et al. [28] use high-level features (i.e. rotation angles) to compute distances either from the nearest neighbors of the input sequence, while Raab et al. [33] use embeddings of motion features from a pre-trained motion encoder for computing the corresponding distances. From our experiments, we confirm the superiority of deep features over raw input features [46] in interpreting the inter-diversity and the intra-diversity of the generated sequences. However, we run our evaluation using all presented metrics, as well as the harmonic mean from [33] that attempts to describe both quality and diversity with one value.

Data. We use the Mixamo sequences presented in [33] to evaluate our improved GAN against the GANimator and the SinMDM in terms of training time, inference time, quality and diversity using the aforementioned metrics.

Implementation details. We use the provided PyTorch [31] implementations for GANimator and SinMDM and adopt the pretrained motion encoder from the SinMDM codebase to extract the motion embeddings for SiFID and inter/intra diversity metrics computation. All experiments are conducted on a NVIDIA RTX 3060 GPU.

Results. Table 1 details the performed ablation study for improving the training performance of GANimator by validating the two presented techniques, i.e. mini-batch training with equilibrium preservation and transfer learning between pyramid levels. As shown in the results, increasing the batch size significantly improves the training time, but should be accompanied with an increase of training iterations, $\lambda_{adv}$, and $\lambda_{rec}$. However, larger batch sizes seem to hurt the $G(\cdot)-D(\cdot)$ adversarial game and leads to inferior performance despite decreasing the training time. After having improved training time and demonstrated similar performance with the baseline, we move to exploit the correlations between pyramid levels. From Table 1, we can see that the combination of cross-stage transfer for the generators $G(\cdot)$, and annealing $\lambda_{adv}$ and $\lambda_{rec}$ leads to the best results, while exhibiting the best improvement in training time. As a next step, we compare our best model with GANimator [28] and SinMDM [33]. From the results in Table 2, we conclude that our improved GAN achieves slightly better results that its baseline [28] when tested on the Mixamo dataset, while also approaching SinMDM [33]. However, the results showcase that our GAN exhibits a significant improvement in training time compared to both SoTA (almost $\sim\!\times 7$ and $\sim\!\times 2$ respectively), while being extremely faster in inference time compared to the latter ($\sim\!\times 1000$).

Apart from the training time performance increase, we introduce new applications that can be performed with a single-shot GAN-based model, such as the body-part and full-body motion composition, in addition to showcasing results on applications introduced by GANimator, like motion re-styling and crowd generation. Note that contrary to [28] we focus solely on applications that do not need re-training, as they pose the main challenge and exploit the superiority of GANs over other approaches in terms of performance.

Body-part motion composition. As detailed in Sec. 3.1, the motion feature $\mathcal{M}_{T}$ of each input sequence $T$ includes a 6D representation about each skeleton joint. This allow us to define body binary masks $M$ for the upper and the lower body, which can be applied on the motion features during inference and force the masked area to retain its original values, while the rest of the body’s movement will be generated based on randomly sampled noise. As depicted in Fig. 5 (top) we use the Mixamo motion “swing dancing” as input sequence $T$ and we choose to keep the lower-body unaltered, while generating alternative - but natural - versions of the upper-body. To achieve that, we use the $G(\cdot)$ trained with this sequence; the hierarchical structure of the model allows us to choose at which level $L_{*}$ to apply the predefined mask $M^{lb}$ that will keep the lower-body (lb) unaltered. We choose to apply $M^{lb}$ at $L_{2}$ as it leads to the smoothest blending of the body parts. Fig. 5 (bottom) demonstrates the application of $M^{ub}$ to the upper-body (ub) of the same motion sample, which preserves its pose despite the generated global rotation and lower body pose.

Full-body motion composition. Assuming a reference motion $T$ of arbitrary length, we consider the following options: a) remove mini-clips of $T$ and use the GAN to “inpaint” them with generated content, and b) select one (or more) mini-clip(s) from $T$ and compose a new motion with the mini-clip(s) placed at the temporal spot(s) of interest. To perform this options, we use a binary mask $M\in\mathbb{B}^{T\times(JQ+C+3)}$ applied to the whole motion feature and not on specific joint-related features. As depicted in Fig. 6, we use $M$ to select the region of interest (ROI) of the “salsa dance” sequence, i.e. $M$ values are ones for the frames that correspond to a clock-wise spin. Then, for option (a) we remove the ROI and inpaint the missing part of the motion as:

where $\tilde{M}$ is the inverse of binary mask $M$ and $\hat{T^{1}_{2}}=G^{1}_{2}(G^{1}_{1}(z_{1}),z_{2})$ is the generated motion that is “inpainted” in the downsampled ($\downarrow$) salsa sequence. The result of the “inpainting” is presented in the middle of Fig. 6. For option (b) we use the ROI as a standalone mini-clip $T^{ROI}$ which we downsample to the $L_{2}$ input level and concatenate a predefined numbers of ROIs with generated motion features from $L_{1}$ level. For example, as depicted in Fig. 6(bottom), two $T^{ROI}$ - each representing a spin - are concatenated with $\hat{T}^{1}_{2}$ generated by generators trained with the “salsa dance” sample. The two spins are smoothly blended into the generated salsa dance sequence in the desired time steps.

Crowd generation & motion expansion. Single-shot learning enables the generation of motions with common low-frequency features, i.e. same motion base, and small variations in the high-level features. This means that by sampling multiple codes from a Gaussian distribution $\mathcal{N}(0,I)$, we can generate a crowd performing similar motions. An example of crowd generation is depicted in Fig. 7. Another straight-forward application is the motion expansion. Since the skeleton-aware convolutions can be applied to a motion feature of arbitrary size, we can concatenate generated features $\hat{T}^{2}_{4}$ to the downsampled original motion features $T^{2}_{4}$ at the temporal dimension and use them as input to the corresponding $G_{\{2,\dots,S\}}$.

Re-styling. This has been the most discussed application in single-shot generation as it is relatively trivial for the image domain, however applying style on a motion is challenging. In the image domain, transferring style is the process of applying texture encoded information on image content (e.g. applying a style of another artist on a painting as in [10]). In the motion domain, style transfer is realized as applying high-frequency details on low-frequency features that correspond to a certain motion. This means that one cannot apply a dancing style from a stationary (i.e. minor translation) motion to a walking one with single-shot generation. However, even with some restrictions, re-styling a motion in real-time is still valuable and leads to interesting results, as the example in Fig. 8. To re-style motion $T_{x}$ with style from motion $T_{y}$, we use the generators $G^{y}_{i}$ with $i\in{2,\dots S}$, i.e. the stages that learn the high-level features (style) of $T_{y}$ with a downsampled version of $T_{x}$ that corresponds to the content ($T^{C}_{x}$). Note that the temporal downsampling process operates as low-pass filter, encoding the coarsest features of $T_{x}$ as its content.