🤖AI-generated documentation☐ curatedAI Generated

This page was drafted by an AI assistant and may contain inaccuracies.

About content generation types

🤖

AI Generated — Page drafted entirely by AI from codebase or prompt instructions.
(e.g., docs generated from codebase analysis)

← this page

✋→🤖

AI Transformatted — Human provided raw material; AI restructured it into a different format.
(e.g., livestream → blog post, meeting notes → docs)

✋

Human Generated — Page written entirely by a human author.
(e.g., hand-written tutorial)

More info about content generation types ↗

Mocap & Skeleton Processing

Mocap processing takes 2D pose detections from multiple camera views, triangulates them into 3D, filters the resulting skeleton, and exports the data. It runs in two contexts: realtime (live, camera-bound) and posthoc (offline, video-bound).

Pose Detection

RTMPose (Primary Detector)

RTMPose is a top-down pose estimation model from MMPose, running via ONNX Runtime. It's the default skeleton detector for both realtime and posthoc pipelines.

Configuration (RTMPoseDetectorConfig):

• model_name: RTMPose model variant (e.g. "rtmw-x-l_256x192") — trades off inference time vs detection quality
• confidence_threshold: minimum confidence for a keypoint to be kept (a float, e.g. 0.004)

Two deployment modes:

Mode	Where it runs	Tradeoff
Centralized GPU (realtime default)	RealtimeSkeletonInferenceNode batches all cameras	One CUDA context, one OnnxSession, batched process_batch, lower latency
Inline (per-camera fallback / posthoc)	Each CameraNode/VideoNode runs its own Tracker	Simpler, no inter-camera dependency, higher total GPU memory

In centralized GPU mode (the realtime default):

• CameraNodes skip skeleton detection entirely (they still run ChArUco on CPU)
• RealtimeSkeletonInferenceNode reads raw frames from all camera ring buffers
• Calls tracker.process_batch(images_dict) — one batched SkellyTracker call (one ONNX session.run per model) for all cameras simultaneously
• TensorRT engine compilation on first run (1–3 minutes, cached)
• GPU OOM recovery: catches MemoryError, rebuilds session (up to 3 retries)

CharUco Detection

Runs alongside skeleton detection to find ChArUco board markers. Used for calibration (validating camera models) and can optionally provide additional 3D reference points during mocap.

Detector Config Flexibility

Posthoc pipelines build a SkellyTracker Tracker from a TrackerConfig / DetectionStageConfig, so they can work with ChArUco (for calibration), MediaPipe, or RTMPose — MediaPipe and RTMPose are both first-class skeleton detectors. The realtime pipeline defaults to RTMPose but can also run MediaPipe (in per-camera mode).

Skeleton Filtering Pipeline (Realtime)

The realtime filtering + fitting pipeline runs in RealtimeAggregatorNode after triangulation. Processing order:

Raw 3D keypoints (from triangulation; RTMPose names, mm)
    │
    ▼
RealtimeKeypointFilter (One Euro smoothing + velocity-decay gap fill)
    │ Smooths jitter while preserving fast motion; briefly extrapolates
    │ keypoints that drop out for a few frames
    │
    ▼
RealtimePointGate (velocity check)
    │ Rejects points that "teleport" (velocity exceeds max), so a single
    │ bad triangulation can't corrupt the skeleton
    │
    ▼
RealtimeSkeletonRigidifier (rigid-body correction)
    │ Maps tracker keypoints → canonical landmarks, estimates each bone's
    │ length online (best-K-by-reprojection-error median, seeded from
    │ anthropometry), then enforces it with one closed-form forward pass
    │ for body + both hands. Output: a rigid, bone-length-consistent skeleton.
    │
    ▼
Center of mass + XCoM (per-frame, on the rigidified skeleton; Winter 2009 / Hof 2008)

OneEuroFilter

A low-pass filter with adaptive cutoff frequency. It smooths jitter in slow movements while preserving fast motion. Parameters:

• min_cutoff: minimum cutoff frequency (Hz)
• beta: speed coefficient (higher = more responsive to fast motion)
• d_cutoff: cutoff for velocity estimation

RealtimeKeypointFilter also gap-fills: a keypoint missing for a few frames is extrapolated from its last velocity (decaying to a stop) so the stream doesn't blink during brief dropouts.

RealtimePointGate

Detects "teleportation" artifacts — points that jump impossibly far in a single frame. If a point's velocity exceeds the max, it's rejected, preventing single-frame detection errors from corrupting downstream fitting.

RealtimeSkeletonRigidifier (rigid-body correction)

The rigidifier is the single realtime skeleton constraint — a streaming version of the posthoc rigid-bones step (skellyforge enforce_rigid_bones). Per frame it:

• Maps RTMPose keypoints onto the canonical anatomical model (canonical_body.yaml / canonical_hand.yaml in skellyforge) via the skellytracker tracker→canonical mapping.
• Updates each bone's online length estimate: a per-bone buffer keeps the best-K measurements ranked by reprojection error (with an age decay, so it never sticks on stale data) and reports their median. Each buffer is seeded from an anthropometric prior (bone-length ratio × subject height) until real observations accumulate.
• Runs a single closed-form forward pass from the root (hips_center for the body, wrist for each hand): for each bone it keeps the observed direction but sets the length to the current estimate, placing the child off the already-corrected parent. No iteration, no convergence loop — O(bones) per frame, so it can't get stuck.

Because the enforced length is the estimate (not the noisy per-frame observation), segment lengths are rigid over time and converge to the same medians the posthoc pipeline produces. The center of mass is computed on this rigidified skeleton, matching the posthoc rigid_xyz → CoM flow.

The canonical model is the single source of truth for body topology, bone-length seeds, and — with its segment/COM tables — the center-of-mass calculation.

Triangulation

The Triangulator (core/tasks/triangulation/triangulator.py, ~578 lines) is a pure DLT implementation with no anipose dependency:

1. Precomputation: On initialization, precomputes extrinsics matrices, rodrigues vectors, camera matrices, and distortion coefficients from the CalibrationResult
2. Undistort: Corrects 2D observations using camera intrinsics + distortion coefficients
3. DLT: Direct Linear Transform triangulation from 2+ camera views
4. Outlier rejection: Subset-ensemble method — tries combinations of camera subsets, rejects points with high reprojection error across all subsets
5. Reprojection error gating: Projects 3D points back to each camera, rejects if error exceeds threshold

Performance: All operations are vectorized over P points simultaneously. _batch_outlier_rejection() runs O(combinations) SVD calls, not O(P × combinations).

Input formats: Accepts dict[camera_id, ndarray] (single frame), 3D numpy arrays (n_cameras, n_points, 2) for single-frame batches, or 4D numpy arrays (n_cameras, n_frames, n_points, 2) for multi-frame batches.

Posthoc Mocap Task

Orchestrated by run_posthoc_mocap_aggregator_task() in posthoc_mocap_task.py:

1. Receives frame_observations — a per-frame list of {camera_id: Observation} collected from all VideoNodes (unified SkellyTracker Observations)
2. Builds an ObservationBuffer per camera and accumulates each camera's observations
3. Resolves the calibration TOML (explicit path wins; otherwise the most-recent successful calibration) and copies it into the recording folder
4. Calls skeleton_from_mediapipe_observation_recorders(detector=…, observation_recorders=…, …) to triangulate the 3D skeleton into output_data/
5. Writes tracker_schema.json from the active TrackerDefinition (e.g. RTMPOSE_WHOLEBODY_DEFINITION)
6. Optionally triggers Blender export

Output Artifacts

After mocap processing, results are written under the recording folder:

recording_folder/
├── synchronized_videos/      Input camera videos
├── annotated_videos/         Videos with detected 2D keypoints drawn on
├── output_data/              Reconstructed + post-processed 3D data
│   ├── *.npy                 3D keypoint arrays (frames × keypoints × 3), CoM, etc.
│   ├── *.csv
│   ├── *.parquet             Primary tabular mocap data store
│   └── per_camera_weights.npy
├── tracker_schema.json       Keypoint names + connections (from the TrackerDefinition)
├── <calibration>.toml        Calibration used for this recording (copied in)
└── <name>.blend              Blender export (if enabled)

For the end-user walkthrough that produces these, see the Post-Hoc Motion Capture guide.

Blender Export

Triggered after mocap processing (or manually via REST API). The export flow:

1. Checks required .npy files exist in output/
2. Detects Blender executable (auto-detect or user-specified path)
3. Runs Blender in --background mode with --python run_blender_export.py
4. Injects freemocap_blender_addon site-packages into Blender's sys.path (no addon installation into Blender's user preferences needed)
5. Waits for subprocess to complete
6. Verifies the .blend file was created and is non-empty
7. Optionally opens the .blend in Blender GUI via subprocess.Popen

The Blender addon reads the .npy files and builds a complete 3D scene: skeleton mesh, camera frustums, ground plane, lighting.

Realtime vs Posthoc: Known Differences (V2 Alpha)

Since the posthoc pipeline was ported from V1 while the realtime pipeline was built fresh for V2, there are implementation differences:

Aspect	Realtime	Posthoc
Filtering	One Euro + velocity gate + rigid-body correction	V1 filtering pipeline (being updated)
Triangulation	DLT with subset-ensemble outlier rejection	DLT with configurable outlier rejection
Detector	RTMPose (default); MediaPipe in per-camera mode	RTMPose or MediaPipe (configurable)
Output format	WebSocket streaming	.npy + .parquet files

These gaps are known and will be closed as the posthoc pipeline is updated to match the realtime pipeline's filtering and triangulation approach. See the Post-Hoc Motion Capture guide for the current user-facing limitations (e.g. the 3D Reconstruction filter settings — Point Gate, One Euro, FABRIK — are not yet passed into posthoc processing).