Noise-in, Bias-out: Balanced and Real-time MoCap Solving

Solving the joints’ positions or transforms from marker data is a cascade of numerous (sometimes optional) steps. The markers need to be labeled, ghost markers need to be removed, occluded markers should be predicted and then an articulated body structure needs to be fit to the observed marker data. Various works address errors at different stages of MoCap solving, with contemporary ones relying on smoothness and bone-related (angles, offsets and lengths) constraints [27, 66, 31, 6, 18, 53, 73]. Recent approaches started resorting to existing data for initialization [69] or marker cleaning [5]. MoSh [44] moved one step ahead and instead of relying on plain structures employed a parametric human body to solve labeled marker data and estimate pose articulation and joint positions, even accounting for marker layout inconsistencies and/or soft tissue motion.

Nonetheless the advent of modern – deep – data-driven technologies have stimulated new approaches for MoCap solving. A label-via-regression approach was employed in [25] where a deep model was used to regress marker positions and then perform maximum assignment matching for labeling the input. Labeling was also formulated as permutation learning problem [21], albeit with constraints on the input, which were then relaxed in [20] by adding a ghost category. However, labeling assumes that the raw data are of a certain quality as the raw measurements are then used to solve for the joints’ transforms or extra processing steps are required to denoise the input.

Consequently, end-to-end data-driven approaches that can simultaneously denoise and solve have been a parallel line of research. While end-to-end cleaning and solving is possible using solely a single feed-forward network [29], the process naturally benefits from using two cascaded autoencoders [62], the first operating on marker data and cleaning them for the subsequent joint regressor. The staging from markers to joints was also shown to be important from a performance perspective in [13] which trained a convolutional network with coupled noisy and clean data captures to address noisy inputs. Recently, graph convolutional models were employed in [14] allowing for the explicit encoding of marker layout and skeleton hierarchy, two crucial factors of variation that were only implicitly handled in prior end-to-end solvers.

Learning to solve MoCap marker data requires supervision provided by collecting data using professional high-end MoCap systems [29, 20, 14, 13]. SOMA [20] standardized the representation using the AMASS dataset [49] which, in turn, relied on an extension of MoSh [44] to fit a parametric human body model to markers. All other works suffer from inconsistent marker layouts which is a problem that was either implicitly addressed [29, 13] or quasi-explicitly [14] using the layouts as inputs. Marker data can be (re-)synthesized in different layouts when higher-level information is available (e.g. marker-to-joint offsets, meshes) [29, 20]. Yet, it has been also shown that fitting a synthetic hand model to depth data acquired by consumer-grade sensors can also produce usable training data [25] for deploying a model to a high-end marker capturing system for data-driven MoCap. Compared to [25], we experimentally demonstrate this feasibility and even extend it to noisy inputs at run-time, something not considered in [25] as it relied on a high-end system for live capture.

Statistical parametric models [45, 61, 58, 59, 85, 87, 4, 88] are more expressive alternatives than the skinned mesh [83] used in [25] as, apart from realistic shape variations, deformation corrective factors can also be employed. They have been used to synthesize standardized training data before [82, 28, 38] but crucially rely on preceeding high-end MoCap acquisition. We also explore this path using multi-view markerless capture [33, 15, 92] to produce parametric model fits and synthesize marker positions as a solution to the cold-start problem of data-driven MoCap solving. Even though such data can be fit to marker data as done in AMASS [49] and Fit3D [19], the potential of acquiring them using less expensive capture solutions is very important, as long as it is feasible to train high quality models.

Still, one also needs to take into account the nature of human performance data and their collection processes. As seen in AMASS [49] and Fit3D [19], both contain significant redundancies and suffer from the long-tail distribution effect. Rare poses are challenging for regression models to predict, mainly stemming from the combined effect of the selected estimators and stochastic optimization with mini-batches. Various solutions have been surfacing in the literature, some tailored to the nature of the problem [67], leveraging a prototype classifier branch to initialize the learned iterative refinement, and others adapting works from imbalanced classification to the regression domain. Traditional approaches fall into either the re-sampling or re-weighting category, with the former focusing on balancing the frequency of samples and the latter on properly adjusting the parameter optimization process. Re-sampling strategies involve common sample under-sampling [79], rare sample over-sampling by synthesizing new samples via interpolation [81], re-sampling after perturbing with noise [9], and hybrid approaches that simultaneously under- and over-sample [10]. Yet interpolating high-dimensional samples like human pose is non-trivial or even defining the rare samples that need to be re-sampled.

Utility-based [80] – or otherwise, cost-sensitive – regression assigns different weights – or relevance – to different samples. Defining a utility function is also essential to re-sampling strategies for regression [79]. Recent approaches employ kernel density estimation [74], adapt evaluation metrics as losses [72], or resort to label/feature smoothing and binning [89]. Another family of methods that are now explored can be categorized as contrastive, with [22] regularizing training to enforce feature and output space proximity. BalancedMSE [64] is also a contrastive-like objective that employs intra-batch minimum error sample classification using a cross-entropy term that corresponds to an L2 error from a likelihood perspective. However, most approaches rely on stratified binning of the output space using distance measures that lose significance in higher dimensions. Further, binning can only be used with specific networks/architectures (proper feature representations for classifying bins or feature-based losses). It has not been shown to be applicable in high-performing dense networks relying on heatmap representations. Instead, we introduce a novel technique that can jointly over-sample and assign higher relevance to rare samples by leveraging representation learning and its synthesis and auto-encoding traits.

Relevance functions drive utility regression and guide the re-/over-/inter-sample selection/generation [10, 79, 80, 81]. Instead of defining relevance or sample selection based on an explicit formula or set of rules, we employ representation learning to learn it from the data. Autoencoding synthesis models [41, 65] jointly learn a reconstruction model as well as a generative sampler:

with varying constraints on the input $\boldsymbol{\theta}$ and latent $\mathbf{z}=\mathcal{E}(\boldsymbol{\theta}),\mathbf{z}\in\mathbb{R}^{Z}$ spaces. An encoder $\mathcal{E}(\boldsymbol{\theta})$ maps input $\boldsymbol{\theta}$ to a latent space $\mathbf{z}$ which gets reconstructed to $\boldsymbol{\theta}^{\ddagger}$ by a generator $\mathcal{G}(\mathbf{z})$. Using a sampling function $\mathcal{S}$ to sample the latent space it is also possible to generate novel output samples $\boldsymbol{\theta}^{\star}$. We exploit the hybrid nature of such models to design a novel imbalanced regression solution that simultaneously over-samples the distribution at the tail and adjusts the optimization by re-weighting rarer samples. Our solution is based on a deep Variational AutoEncoder (VAE) [41].

Relevance via Reconstructability. Autoencoding models are expected to reflect the bias of their training data, with redundant/rare samples being easier/harder to properly reconstruct respectively. This bias in reconstructability can be used to assign relevance to each sample as those more challenging to reconstruct properly are more likely to be tail samples. We define a relevance function $\rho$ (see Fig. 2 re-weighting) using a reconstruction error $\epsilon$:

with $(\bar{\cdot})$ denoting unit normalization using the input joints’ bounding box diagonal, $\epsilon$ the normalized-RMSE over the reconstructed and original joints, and $\sigma$ a scaling factor controlling the relevance $\rho$. Using landmark positions we can preserve interpretable semantics in $\rho$ and $\sigma$ as they are unidirectionally interchangeable (linear mapping) with the pose $\boldsymbol{\theta}$ given fixed shape $\boldsymbol{\beta}$. Fig. 3 shows exemplary poses as scored by our relevance function.

Balance via Controlled Synthesis. Even though the tail samples are not reconstructed faithfully, the generative and disentangling nature of modern synthesis models shape manifolds that map inputs to the underlying factors of data variation, effectively mapping similar poses to nearby latent codes which can be traversed across the latent space dimensions. Based on this, we define a controlled sampling scheme for synthesizing new tail samples (see Fig. 2 oversampling). Using the relevance function from Eq. 2, it is possible to identify tail samples $\boldsymbol{\theta}^{\dagger}$ via statistical thresholding that serve as anchor latent codes $\mathcal{A}=\{\,\mathbf{z}^{\dagger}\mid\mathbf{z}^{\dagger}=\mathcal{E}(\boldsymbol{\theta}^{\dagger})\,\}$. This process adapts to the training data distribution instead of risking a mismatch via empiric manual picking when using a purely generative model (e.g. [78]). We then sample using the following function:

Specifically, we sample from a normal distribution centered around two random anchors $i$ and $j,i\neq j,$ from $\mathcal{A}$ using a standard deviation $\mathbf{s}$, and blend them using spherical linear interpolation [70] $\varsigma$ with a uniformly sampled blending factor $b\in\mathcal{U}(0,B),B\in[0,1]$. Non-linear interpolation between samples avoids dead manifold regions as not all directions lead to meaningful samples [35, 37] and increases our samples’ plausibility [86], as illustrated in Fig. 4.

Compared to pure labeling [20, 21] or pure solving approaches [29, 14] we design our model around simultaneous denoising, solving and hallucination.

While some approaches use the raw marker positions as input, we opt to leverage the maturity of structured heatmap representations and employ a convolutional model, similar to [25, 13] instead of relying on unstructured regression [14, 29] using MLPs. This improves the convergence of the model and by using multi-view fusion we can also improve accuracy via robust regression. First, we augment and corrupt the input markers $\boldsymbol{\ell}_{gt}$ into $\tilde{\boldsymbol{\ell}}_{in}^{\star}$. Then, we normalize and render $\tilde{\boldsymbol{\ell}}_{in}^{\star}$ from two orthographic viewpoints as in [13], but with a notable difference when processing the model’s output; instead of predicting the $3^{rd}$ dimension, we manage to predict normalized 3D coordinates by learning to solve a single 2D task. To achieve that, we use the two rendered views as input to the model, predict the corresponding view’s heatmaps, and fuse them with a variant of marginal heatmap regression [56, 90] (see Fig. 2 fusion). We assume the gravity direction along the $y$ axis and use the orthogonal and orthographic views denoted as $xy$ and $yz$ which share the $y$ axis. To estimate the landmarks’ normalized positions $\tilde{\boldsymbol{\ell}}_{est}$, we employ center-of-mass regression [48, 75, 55, 77] taking the average expectation [56, 90] for $y$ from the two views.

The model is supervised by:

where $\mathcal{L}_{JS}$ is the $\lambda-$weighted Jensen-Shannon divergence [52] between the normalized ground truth and soft-max normalized predicted heatmaps, while $\mathcal{L}_{w}^{\nu}$ is the robust Welsch penalty function [30, 17], with the support parameter $\nu$, between the normalized landmark ground-truth $\tilde{\boldsymbol{\ell}}_{gt}$ and estimated $\tilde{\boldsymbol{\ell}}_{est}$ coordinates. Overall, $\mathcal{L}_{JS}$ accelerates training while $\mathcal{L}_{w}^{\nu}$ facilitates higher levels of sub-pixel accuracy since even though we reconstruct the heatmaps $\mathbf{H}$ using the normalized – un-quantized – coordinates [93], discretization artifacts can never be removed entirely.

Note that the fusion outcome $\tilde{\boldsymbol{\ell}}_{est}$ comprises both marker and joint estimations, essentially estimating a complete, labeled, and denoized marker set, as well as solving for the joints’ positions.

Finally, we use U-Net [68] as a regression backbone for its runtime performance and its efficiency in high-resolution regression.

Given the denoised and complete set of landmarks $\tilde{\boldsymbol{\ell}}_{est}\in\mathbb{R}^{L\times 3}$, we can fit the body to these estimates and obtain the pose $\boldsymbol{\theta}$ and shape $\boldsymbol{\beta}$ which implies an articulated skeleton and mesh surface. This is a non-linear optimization problem with the standard solution being MoSh [44] and its successor MoSh++ [49]. However, MoSh(++) also solves for the marker layout which in our case is known apriori as the model was trained with a standard $53$ marker configuration. Compared to prior works that assume the estimates are of high-quality or low signal-to-noise ratios, we seek to relax this assumption to support additional sensing options. The solution to this is robust optimization but typical approaches that involve robust kernels/estimators require confident knowledge about the underlying data distribution. This is not easily available in practice, and moreover, it varies with different sensing options but more importantly, when involving a data-driven model, it is skewed by another challenging-to-model distribution. The Barron loss [7] is a robust variant that also adapts to the underlying distribution and interpolates/generalizes many known variants by adjusting their shape and scale jointly.

Following likelihood-based formulations [39, 24] that have been presented for multi-task/robust stochastic optimization, we formulate a noise-aware fitting objective that is adaptive and optimizes the Gaussian uncertainty region $\boldsymbol{\sigma}\in\mathbb{R}^{L}$ jointly with the data and prior terms:

We use standard prior terms [44, 49, 61] $\mathcal{E}_{prior}=\lambda_{\boldsymbol{\beta}}\sum||\boldsymbol{\beta}||_{2}+\lambda_{\mathbf{z}}\sum||\mathbf{z}||_{2}$, and a data term formulated as:

As in MoSh(++) we perform staged annealed optimization but with only 2 stages as there is no marker layout optimization. The first stage optimizes over $\boldsymbol{\beta}^{\ast},\boldsymbol{\theta}^{\ast},\mathbf{T}^{\ast}$, while the second stage fixes $\boldsymbol{\beta}$ and $\mathbf{T}$ and optimizes $\boldsymbol{\theta}^{\ast},\boldsymbol{\sigma}^{\ast}$.