🤖AI-generated documentation☐ curatedAI Generated

This page was drafted by an AI assistant and may contain inaccuracies.

About content generation types

🤖

AI Generated — Page drafted entirely by AI from codebase or prompt instructions.
(e.g., docs generated from codebase analysis)

← this page

✋→🤖

AI Transformatted — Human provided raw material; AI restructured it into a different format.
(e.g., livestream → blog post, meeting notes → docs)

✋

Human Generated — Page written entirely by a human author.
(e.g., hand-written tutorial)

More info about content generation types ↗

API Boundary

This page documents the complete contract between the React frontend and the Python backend. Everything here is observable from the frontend side — the backend internals that implement these endpoints are documented in the Backend Architecture pages.

REST API Reference

Base URL: http://localhost:53117 (configurable via ServerUrls singleton).

Camera Management (SkellyCam)

note

TODO - add typehints to the models!

POST    /skellycam/camera/detect
        Detect connected USB cameras.
        Response: DetectedCamerasResponse { cameras: CameraDeviceInfo[] }
        CameraDeviceInfo: { camera_id, index, name, vendor_id, product_id, path, backend_id, backend_name }

POST    /skellycam/camera/group/apply
        Connect to cameras with desired configurations.
        Body: CameraGroupCreateRequest { camera_configs: dict[CameraIdString, CameraConfig] }
        Note: field is camera_configs (not configs), and it's a dict keyed by camera ID, not an array.

DELETE  /skellycam/camera/group/close/all
        Disconnect all cameras.

GET     /skellycam/camera/group/all/pause_unpause
        Toggle pause state of all camera streams.

GET     /skellycam/camera/microphone/detect
        Detect available microphones.
        Response: DetectedMicrophonesResponse { microphones: dict[int, str] }
        Note: response is a dict mapping index→name, not a list of {id, name} objects.

Recording

POST    /skellycam/camera/group/all/record/start
        Start recording on all connected cameras.
        Body: StartRecordingRequest { recording_name, recording_directory, mic_device_index }

GET     /skellycam/camera/group/all/record/stop
        Stop recording on all cameras.
        Response: list[StopRecordingResponse]
        Each entry: { recording_name, recording_path, number_of_cameras, number_of_frames,
                      total_duration_sec, mean_framerate, mean_inter_camera_sync_ms,
                      framerate_stats, frame_duration_stats, inter_camera_grab_range_ms_stats }

Playback

GET     /freemocap/playback/recordings
        List all available recordings.
        Response: list[RecordingListEntry]
        Each entry: { name, path, video_count, total_size_bytes, created_timestamp,
                      total_frames, duration_seconds, fps, status_summary, status, layout_validation }

GET     /freemocap/playback/{recording_id}/videos
        List videos for a recording.
        Response: VideoSourcesResponse { preferred_source, sources: { "annotated": VideoSourceInfo, "synchronized": VideoSourceInfo } }
        Each VideoSourceInfo: { video_id, filename, size_bytes, stream_url }

GET     /freemocap/playback/{recording_id}/videos/{video_id}
        Stream a video file (supports HTTP Range for seeking).
        Response: video/mp4 binary stream

GET     /freemocap/playback/{recording_id}/timestamps
        Get all frame timestamps for a recording.

GET     /freemocap/playback/{recording_id}/videos/{video_id}/timestamps
        Get timestamps for a specific video.

GET     /freemocap/playback/{recording_id}/status
        Get playback availability status (full RecordingStatus model).

GET     /freemocap/playback/{recording_id}/bundle
        Get full playback bundle.
        Response: RecordingBundle { recording_id, recording_fps, total_frames, duration_seconds,
                                    videos: VideoSourcesResponse, timestamps, calibration,
                                    tracker_schema, status_summary }

GET     /freemocap/playback/{recording_id}/parquet
        Serve the recording's freemocap_data_by_frame.parquet file.

GET     /freemocap/playback/{recording_id}/calibration
        Return parsed camera calibration/pose data from the recording's calibration TOML.

GET     /freemocap/playback/{recording_id}/tracker-schema
        Return the tracker schema (tracked points + connections) for a recording.

GET     /freemocap/playback/viewer (include_in_schema=False)
        Serve the parquet skeleton viewer HTML page.

GET     /freemocap/playback/parquet (include_in_schema=False)
        Serve a parquet file by absolute path query parameter.

Calibration

POST    /freemocap/calibration/recording/start
        Start a calibration recording.
        Body: CalibrateRecordingRequest { calibrationTaskConfig, calibrationRecordingDirectory }

POST    /freemocap/calibration/recording/stop
        Stop the calibration recording.

POST    /freemocap/calibration/recording/calibrate
        Run the calibration solver on the recording.
        Response: streams progress over WebSocket; final response with calibration data

Mocap

POST    /freemocap/mocap/recording/start
        Start a mocap recording.
        Body: StartMocapRecordingRequest { mocapRecordingDirectory, mocapTaskConfig }

POST    /freemocap/mocap/recording/stop
        Stop the mocap recording.

POST    /freemocap/mocap/recording/process
        Run post-processing on the recording.
        Response: streams progress over WebSocket

Realtime Pipeline

POST    /freemocap/realtime/apply
        Create or update the realtime pipeline.
        Body: RealtimePipelineConnectRequest { cameraConfigs, realtimeCameraIds, realtimeConfig }

DELETE  /freemocap/realtime/all/close
        Close the realtime pipeline.

Posthoc Pipeline Control

DELETE  /freemocap/posthoc/pipeline/{pipeline_id}
        Stop a specific running pipeline.

DELETE  /freemocap/posthoc/pipeline
        Stop all running pipelines.

Blender

GET     /freemocap/blender/detect
        Find installed Blender executables.
        Response: DetectBlenderResponse { blender_exe_path, found, message }

POST    /freemocap/blender/addon/install
        Install the freemocap_blender_addon into detected Blender.
        Body: InstallAddonRequest

POST    /freemocap/blender/export
        Export recording to .blend file.
        Body: ExportToBlenderRequest { recordingFolderPath, blenderExePath, autoOpenBlendFile }
        Response: { blend_file_path }

POST    /freemocap/blender/open
        Open a recording's .blend file in Blender.
        Body: OpenInBlenderRequest { recordingFolderPath, blenderExePath }

System

GET     /health
        Health check. Returns "Hello" with 200.

GET     /shutdown
        Gracefully shut down the server.

SkellyCam Playback Router (undocumented in frontend)

The skellycam package also mounts its own playback router under /skellycam/playback/:

GET     /skellycam/playback/recordings
GET     /skellycam/playback/{recording_id}/videos
GET     /skellycam/playback/{recording_id}/videos/{video_id}
GET     /skellycam/playback/{recording_id}/timestamps
GET     /skellycam/playback/{recording_id}/videos/{video_id}/timestamps

WebSocket Message Reference

WebSocket URL: ws://localhost:53117/websocket/connect

Server → Client Messages

Binary Messages

Frame Message (msgType 0x00)

Payload Header (16 bytes):
  Byte 0:     message_type = 0 (PAYLOAD_HEADER)
  Bytes 1-8:  frame_number (int64, little-endian)
  Bytes 9-12: number_of_cameras (int32)

Per camera:
  Frame Header (44 bytes):
    Byte 0:     message_type = 1 (FRAME_HEADER)
    Bytes 1-8:  frame_number (int64)
    Bytes 9-24: camera_id (string, 16 bytes)
    Bytes 25-28: camera_index (int32)
    Bytes 29-32: image_width (int32)
    Bytes 33-36: image_height (int32)
    Bytes 37-40: color_channels (int32)
    Bytes 41-44: jpeg_string_length (int32)
  JPEG data (jpeg_string_length bytes)

Payload Footer (16 bytes): mirrors header with message_type = 2

Keypoints Message (msgType 0x03)

Payload Header (24 bytes):
  Byte 0:     message_type = 3 (KEYPOINTS_HEADER)
  Bytes 1-8:  frame_number (int64)
  Bytes 9-12: num_blocks (uint32)

Per block:
  Block Header (44 bytes):
    message_type = 4, block_kind, dtype_code, dims,
    camera_id (string, 16 bytes), tracker_id (string, 32 bytes),
    num_points (uint32), data_byte_length (uint32)
  Block Data: interleaved x/y(/z)/visibility per point, row-major

Payload Footer (24 bytes): message_type = 5

JSON Messages

Log Message

{
    "message_type": "log_record",
    "levelname": "INFO",
    "message": "Camera 0 connected",
    "timestamp": 1623456789.123
}

Log levels: TRACE, DEBUG, INFO, SUCCESS, API, WARNING, ERROR, CRITICAL.

Framerate Message

{
    "message_type": "framerate_update",
    "camera_group_id": "group_1",
    "backend_framerate": { "fps": 29.97, "frame_count": 120, ... },
    "frontend_framerate": { "fps": 29.95, "frame_count": 119, ... }
}

Frontend Payload

{
    "message_type": "frontend_payload",
    "payload_type": "keypoints_raw | keypoints_filtered | rigid_body_poses | charuco_overlays | skeleton_overlays | pupil_data",
    "data": { ... }
}

Posthoc Progress

{
    "message_type": "posthoc_progress",
    "pipeline_id": "cal_abc123",
    "pipeline_type": "calibration | mocap",
    "phase": "processing_videos",
    "progress_fraction": 0.455,
    "detail": "Processing video 3/6",
    "recording_name": "my_recording"
}

Note: progress_fraction is a float from 0.0 to 1.0 (not a percentage).

Pipeline phases: queued → setting_up → processing_videos → aggregating → finalizing → complete (or failed).

Tracker Schemas

{
    "message_type": "tracker_schemas",
    "schemas": {
        "rtmpose_wholebody": {
            "name": "rtmpose_wholebody",
            "tracked_points": ["nose", "left_eye", ...],
            "connections": [["left_shoulder", "left_elbow"], ["left_elbow", "left_wrist"], ...]
        }
    }
}

Note: message type is "tracker_schemas" (plural). The schemas field is a dict of tracker name → TrackerDefinition — each with name, tracked_points, and name-pair connections (from core/tracking/tracker_definitions.py, e.g. RTMPOSE_WHOLEBODY_DEFINITION). Multiple schemas are sent in one message. See Tracking Integration.

Client → Server Messages

The client sends JSON messages to the server over the WebSocket. The critical message for backpressure is:

{ "frameNumber": 42 }

This acknowledges receipt of frame N, allowing the server to send the next batch. If the client never sends frame acknowledgments, the image relay stalls after a backpressure threshold.

Other messages include ping/pong for heartbeat, and displayImageSizes for overlay scaling hints. The websocket.disconnect event is handled at the Starlette WebSocket message layer, not as a JSON message type.

Client-to-Server Frame Acknowledgment (Backpressure)

The backpressure protocol is critical for realtime performance:

Backend produces frame N → sends to frontend
                         → waits for ack (frame number N)
Frontend receives frame N → processes in rAF loop
                          → sends ack {"frameNumber": N}
Backend receives ack → allowed to send frame N+1

This prevents the backend from flooding the frontend with frames faster than it can render them. The frontend acks the frame number immediately at the top of its rAF loop (before decoding), so the backend can pipeline the next batch while the frontend decodes the current one. Warnings are logged at multiples of the backpressure threshold (300 frames), and a reset clears internal state if the frontend falls too far behind.

Error Handling Across the Boundary

REST Errors

Non-2xx responses include a JSON body with { error: string, detail?: string }
Network errors (server unreachable) are caught in Redux thunks and surfaced as slice-level error state
Timeouts use the default fetch timeout (browser default, typically 30s)

WebSocket Errors

Disconnect: Auto-reconnect with exponential backoff (1s, 2s, 4s, 8s, 16s), max 5 attempts
Heartbeat failure: If 3 consecutive pings go unanswered, the connection is considered dead and a reconnect cycle starts
Message parse errors: Malformed messages are dropped and logged. The rAF loop continues processing subsequent messages
Binary frame decode errors: Corrupted JPEG frames are dropped. The canvas for that camera shows its last valid frame

How the Frontend Surfaces Errors

Connection status: ServerContext.isConnected boolean, reflected in the ServerConnectionStatus UI component
Camera errors: cameras.error in Redux, surfaced in the camera config tree
Pipeline failures: PipelineProgress.phase === 'failed' with error detail, shown in PipelineProgressSnackbar
REST errors: Individual slice error fields, surfaced inline in the relevant panel

REST API Reference​

Camera Management (SkellyCam)​

Recording​

Playback​

Calibration​

Mocap​

Realtime Pipeline​

Posthoc Pipeline Control​

Blender​

System​

SkellyCam Playback Router (undocumented in frontend)​

WebSocket Message Reference​

Server → Client Messages​

Binary Messages​

JSON Messages​

Client → Server Messages​

Client-to-Server Frame Acknowledgment (Backpressure)​

Error Handling Across the Boundary​

REST Errors​

WebSocket Errors​

How the Frontend Surfaces Errors​