Glossary
Capture volume
3-dimensional area (volume) with sufficient camera coverage to support 3D tracking.
Calibration
Calibration is the process of measuring information about the cameras used for a recording. It measures the camera "intrinsics," like the focal length and lens distortion, as well as "extrinsics," like where the cameras are in space and where they are pointed. Having the cameras calibrated is necessary to triangulate the 2-dimensional data from each camera into 3-dimensional data.
Link to a section of the 'braindump' video discussing capture volume calibration
ChArUco board
A ChArUco board is a combination of a chessboard and ArUco markers, two common tools for calibrating cameras. The ChArUco is a known object that is easily detected in images, and allows the software to figure out where the camera is in relation to the board and correct for distortions from the camera.
Link to a section of the 'braindump' video discussing the ChArUco board
MediaPipe
An open source framework for machine learning perception pipelines from Google. FreeMoCap supports MediaPipe's pose model (which also tracks hands and face) as one of its two skeleton detectors; RTMPose is the default.
YOLOX
An open source object detection model family, originally developed by Megvii. FreeMoCap uses YOLOX to detect the person in each frame so the region around them can be cropped out before pose estimation, see use YOLO cropping. The specific YOLOX checkpoints used are downloaded from OpenMMLab's MMPose ONNX SDK.
Reprojection error
"reprojection error" is the distance (in pixels) between the originally measured point (that is the 2D skeleton) and the reconstructed 3D point reprojected back onto the image plane.
The intuition is that if the 3D reconstruction and original 2D track are perfect, then reprojection error is zero. If it isn't, then there is some inaccuracy in either:
- the original 2D tracks (that is bad skeleton detection in one or more cameras),
- in the 3D reconstruction (that is bad camera calibration),
- a combination of the two
Click here to follow a conversation about reprojection error on discord