BundleMoCap: Efficient, Robust and Smooth Motion Capture from Sparse Multiview Videos

In addition to digitizing human motion, capturing the pose and/or motion of humans is critical for dataset creation (Cheng et al., 2022; Zhang et al., 2023; Huang et al., 2022), training data collection (Bhatnagar et al., 2020), bootstrapping dynamic radiance fields (Zhao et al., 2022), and has even found use in physicalizing human motion (Zhang et al., 2018). The standard approach is to rely on 2D keypoint detections (Cao et al., 2019) on calibrated images/videos and fit an articulated skeleton to these observations (Huang et al., 2017). While there are approaches that only estimate 3D joints (Iskakov et al., 2019; Ye et al., 2022; Bartol et al., 2022), these do not provide a consistent structure, suffering from varying bone lengths, and neither offer articulation parameters. Fitting to 2D keypoint detections is challenging even in the multi-view case, especially when the sensors are sparse, as 2D detectors suffer from high amounts of jitter, inversion and missing detections (Ruggero Ronchi and Perona, 2017) due to occlusions. While direct (Jiang et al., 2022) or progressive (Gong et al., 2023) data-driven regression approaches exist, extra progress is required to improve their performance and generalization.

Capturing motion on a per-frame basis suffers from noisy estimates or fitting to noisy observations. While temporal filtering is one option to resolve the temporal jitter and inconsistency of per-frame captures (Ingwersen et al., 2023), it is very challenging to design filters to handle occlusions and outliers. This led to the integration of temporal constraints and priors when solving for the pose and motion. One of the earlier attempts (Kanazawa et al., 2019) used a bundle of images as input to regressors predicting a coherent bundle of outputs, and specifically the parameters of the SMPL body model. Follow-up approaches (Li et al., 2021) relied on recurrent architectures which are a better architectural fit to the problem at hand, as each prediction directly depends on the previous frame estimates. Regarding temporal constraints within optimization loops, the most common objective is a joint smoothness one (Arnab et al., 2019; Peng et al., 2018; Zhang et al., 2018; Ye et al., 2023) that essentially enforces a position constancy. A DCT smoothness term has also been used when fitting the SMPL body to multiple images (Huang et al., 2017). Another variant involves higher order smoothness terms like velocity constancy (Zanfir et al., 2018; Loper et al., 2014; Mahmood et al., 2019), either on joint/marker positions or on joints’ angles. With the advent of data-driven priors, lots of works have added higher level smoothness constraints to supplement the joint position one. These include different representations, like the feature space of a velocity autoencoder (Zhang et al., 2021; Huang et al., 2022), the latent space of poses (Huang et al., 2021), motion (Saini et al., 2023), or the latent space and transition state of an autoregressive motion prior (Jin and Liu, 2023; Ye et al., 2023).

A common theme to all prior approaches is that motion smoothness is achieved by the addition of one or more additional objectives to the optimization problem. This incurs the cost of additional parameters that need tuning, a case that is especially complex when considering the high number of objectives that are involved when fitting a parametric body model to images. Instead, our approach achieves motion smoothness implicitly by smoothly interpolation a pose manifold, requiring no additional hyper-parameters.

Adding such temporal constraints is more effective when solving for a bundle of frames simultaneously. The DCT prior used in (Huang et al., 2017) was added the second stage of optimization when a group of $30$ frames was solved simultaneously. Similarly, joint smoothness was imposed on a group of frames solved simultaneously in MoSculp (Zhang et al., 2018) as well as other works (Peng et al., 2018; Saini et al., 2023; Ye et al., 2023). Bundle solving with joint smoothness is also possible in the latent space of regressor models (Peng et al., 2018), albeit costly given its excessive parameter count. Apart from solving a small bundle, it has also been shown that it is possible to solve entire videos simultaneously (Arnab et al., 2019), exploiting the entire temporal context, even with very complex objective functions (Huang et al., 2022) involving human-object interactions. Still this requires good initialization, which can be achieved using a regressor model (Arnab et al., 2019) or a per-frame body fitting stage (Huang et al., 2022). This type of staged optimization is standard in the ill-posed monocular case (Bogo et al., 2016; Pavlakos et al., 2019), but also finds widespread use in multiview settings (Huang et al., 2017; Saini et al., 2023). When considering videos, multiple stages are used to initialize estimates for each separate frame, and then solve bundles using smoothness constraints (Zhang et al., 2021; Ye et al., 2023; Saini et al., 2023; Arnab et al., 2019; Peng et al., 2018), be it either via direct regression models or solving an optimization problem for each frame.

Nonetheless, this is rather costly, as multiple passes are required over the videos. In addition, the temporal nature of the data are only considered at the constraints level, but not at the parameters level, with each frame’s state being solved separately in the bundle. Our approach, BundleMoCap, only solves for a single keyframe for each bundle, and only requires a single – first – frame initialization, essentially solving for smooth motion robustly in a single stage.

Earlier body fitting works solved for the pose articulation parameters directly and regularized the solution to plausible poses using Gaussian mixture priors (Bogo et al., 2016; Huang et al., 2017). Modern representation learning offered a more compact latent space to solve instead of the rotation parameter space and simultaneously allowed for penalizing implausible poses directly on the solved parameters (Pavlakos et al., 2019). As a result, while earlier bundle solving MoCap works solved for the articulation (Arnab et al., 2019; Huang et al., 2017; Zanfir et al., 2018), solving for latent representations is now established (Pavlakos et al., 2019; Ye et al., 2023; Saini et al., 2023). Even though initial attempts used the latent space of regressors that is not compact (Peng et al., 2018), or pure autoencoder embeddings (Zhang et al., 2021), variational autoencoders (VAE) (Kingma and Welling, 2015) have proven to be more effective as a prior over their latent space enforces the solution to be directed towards more plausible poses. These have been used to model poses, as in the case of VPoser (Pavlakos et al., 2019), as well as motion (Saini et al., 2023; Huang et al., 2021), with the latter typically employing higher dimensional latent spaces. Another category of solved latent spaces are autoregressive models, represented by HuMoR (Jin and Liu, 2023), which has been used when in video-based MoCap (Ye et al., 2023) during the third solving stage to offer motion smoothness. Still, a VPoser VAE was used in the second stage for pose plausibility. On the other hand, GAN-based methods(Davydov et al., 2022) discriminate the generated poses from the real ones, essentially learning adversarial priors, while Pose-NDF (Tiwari et al., 2022) employs a neural implicit representation for modelling the plausible pose manifolds with a neural implicit function.

In BundleMoCap we show that a compact pose prior can be used to solve for smooth motion using a single latent code, instead of relying on motion or autoregressive priors. This carries the advantage of being less reliant on data offering a high variety of motion transitions, and more reliant on learning a high quality smooth pose manifold, reducing data pressure.

To capture human motion, we use a parametric human body model $\mathcal{B}$ that acts as a function over a group of parameters to generate a human body geometry $(\mathbf{V},\mathbf{F})=\mathcal{B}(\boldsymbol{\beta},\boldsymbol{\theta},\mathbf{T})$. A triangular mesh surface $(\mathbf{V},\mathbf{F})$ is defined by the vertices $\mathbf{V}\in\mathbb{R}^{V\times 3}$ and faces $\mathbf{F}\in\mathbb{N}^{F\times 3}$. It is reconstructed by $\mathcal{B}$ using $S$ blendshape coefficients $\boldsymbol{\beta}\in\mathbb{R}^{S}$, articulated by $P$ pose parameters $\boldsymbol{\theta}\in\mathbb{SO}(3)^{P}$, and globally positioned by the transform $\mathbf{T}=\left[\begin{smallmatrix}\mathbf{R}&\mathbf{t}\\ \mathbf{0}&1\end{smallmatrix}\right]\in\mathbb{SE}(3)$. Using linear operations expressed as a matrix $\mathbf{R}\in\mathbb{R}^{J\times V}$ it is possible to extract $J$ different body joints $\mathbf{j}\in\mathbb{R}^{J\times 3}$ via matrix multiplication $\mathbf{j}=\mathbf{R}\times\mathbf{V}$. The joints $\mathbf{j}$ are projected to an image domain $\Omega:=\mathbb{R}^{W\times H}$ as keypoints $\mathbf{k}=\boldsymbol{\mathcal{\pi}}(\mathbf{j})$ using a projection function $\boldsymbol{\mathcal{\pi}}$ parameterized by the intrinsic properties of $\Omega$.

To estimate the human pose and shape at a single time instance $t$ a minimization problem is formulated using an objective comprising weighted data $\mathcal{E}_{data}$ and prior $\mathcal{E}_{prior}$ terms (Bogo et al., 2016; Pavlakos et al., 2019):

with a notable difference being the optimization over a latent code $\mathbf{z}\in\mathbb{R}^{L}$ instead of the pose parameters $\boldsymbol{\theta}$. While earlier works (Bogo et al., 2016) relied on Gaussian mixture pose priors using the joints’ angles $\boldsymbol{\theta}$ as input, recent works are now using data-driven priors (Pavlakos et al., 2019) to instead optimize lower dimensional latent codes. These reconstruct the pose parameters $\boldsymbol{\theta}=\mathcal{G}(\mathbf{z})$, with the generator function $\mathcal{G}$ being a fixed, pre-trained neural network. Apart from the benefit of optimizing a lower dimensional parameter space, it is also possible to impose a prior term on the latent code itself (Pavlakos et al., 2019; Davydov et al., 2022; Tiwari et al., 2022). This is important as it helps prevent degenerate solutions and provides additional constraints to alleviate the ill-posedness of the problem. Another use of such priors is to initially improve the convexity of the problem when weighted higher, and stage the solve by progressively lowering the prior weights as the solution is refined. This type of annealing is typical and used in both challenging monocular (Bogo et al., 2016; Pavlakos et al., 2019) or multi-view cases (Huang et al., 2017), and even when fitting body models to marker sets (Loper et al., 2014; Mahmood et al., 2019). Still, this multi-staged solving process incurs higher run-times that are costly as each time instance $t$ is separately solved and is usually improved by fixing the shape $\boldsymbol{\beta}$ using a single estimate across the entire sequence.

Our proposition is to solve a temporal window $\mathcal{T}:=[0,\dots,T]$ simultaneously using two latent keyframes at time instances $0$ and $T$:

using an initial and fixed shape estimate $\boldsymbol{\beta}$, with the prior term imposed only on the keyframes, and the data term defined over the entire window $\mathcal{T}$:

where:

Here, $\rho$ is the Geman-McClure penalty function which we favour over a traditional L2 as it can deal better with noisy estimates.

To solve over the entire window and fit our solution to all available data constraints, we reconstruct the intermediate frames via interpolation:

with $\mathcal{S}^{t}$ and $\mathcal{L}^{t}$ being spherical and linear interpolation functions that map $t$ to a closed unit interval ($[0,1]$) within $\mathcal{T}$ to blend the starting $t=0$ and end $t=T$ points, as depicted below:

with $\theta$ representing the angular distance between the two latent codes $\mathbf{z}^{0}$ and $\mathbf{z}^{T}$ (Davydov et al., 2022).

Specifically for the pose $\boldsymbol{\theta}$, this process crucially relies on a well-trained generator $\mathcal{G}$ that captures an expressive manifold $\mathcal{M}$ that can be smoothly interpolated to reconstruct correspondingly smooth pose space transitions.

We use SMPL-X (Pavlakos et al., 2019) as the parametric human body model $\mathcal{B}$. Even though we only optimize for the body joints and ignore the expressive parts, i.e. the hands and face, we use SMPL-X as its corresponding pose prior, VPoser (Pavlakos et al., 2019), offers a well-trained decoder/generator $\mathcal{G}$ over the manifold $\mathcal{M}$ of plausible poses. To obtain the 2D keypoint constraints, $\mathbf{k}_{det}$ and the confidence score $w_{i}$ for the $i^{th}$ joint, we use OpenPose (Cao et al., 2019), an established 2D keypoint estimator model that predicts human joint positions in the 2D image space. We solve the optimization problem using the limited memory BFGS (L-BFGS) optimizer (Wright et al., 1999), with a strong Wolfe line search strategy to determine suitable step sizes during optimization. Optimization iterations are performed using a budget of $30$ iterations, striking a balance between convergence speed and computation resources. All experiments are implemented with a custom PyTorch-based (Paszke et al., 2017) framework (moai: PyTorch Model Development Kit, 2021) using the same PyTorch version, specifically 1.12. We set the weight for the data term and prior term to $1.0$ and $10.76$ respectively and optimize only for a single stage, using a temporal window of length $|\mathcal{T}|=10$.

Our formulation allows for an efficient sliding window optimization implementation. Instead of optimizing Eq. (2) over the two latent keyframes, we can only optimize for the next keyframe, i.e. $(\mathbf{z}^{T},\mathbf{T}^{T})$, while keeping the first keyframe, i.e. $(\mathbf{z}^{0},\mathbf{T}^{0})$, fixed. We initialize the fitting process with a single-frame fit to acquire the first frame ($t=0$) solution for the first temporal window $\mathcal{T}^{1}$ and then solve for the first window’s next latent keyframe $t=T^{1}$. The optimization process then slides to the next temporal window $\mathcal{T}^{2}$ using the latent keyframe $t=T^{1}$ as the first keyframe of the second temporal window $\mathcal{T}^{2}$, keeping it fixed, while solving for the next latent keyframe $t=T^{2}$. This process repeats until all temporal segments are solved, effectively splitting an $F$ frames sequence into $N=F/T$ temporal windows $\mathcal{T}^{i},i\in[0,N]$ of length $|\mathcal{T}^{i}|=T+1$ represented as $N+1$ latent keyframes $(\mathbf{z}^{i},\mathbf{T}^{i})$. For the first frame we solve Eq. (1) with the same process and weights using $2$ stages of optimization. This initial single frame fit is initialized using an HMR (Kanazawa et al., 2018) prediction, and also initializes the shape parameters $\boldsymbol{\beta}$. The latter are kept constant for the entire sequence. This process is consistent for all methods that we will compare our method to, with any method relying on regressor estimates using HMR, and the same shape parameters used for all methods.