Skip to content

Add fmpose3d functionality#3208

Open
deruyter92 wants to merge 5 commits intomainfrom
feat/add_fmpose3d
Open

Add fmpose3d functionality#3208
deruyter92 wants to merge 5 commits intomainfrom
feat/add_fmpose3d

Conversation

@deruyter92
Copy link
Collaborator

@deruyter92 deruyter92 commented Feb 12, 2026

Add FMPose3D integration for monocular 3D pose estimation

Add functionality for using https://github.com/AdaptiveMotorControlLab/FMPose3D to lift 2D predictions to 3D keypoints. (see paper)

Summary

  • Integrate FMPose3D into the DeepLabCut model zoo, enabling monocular 3D pose estimation via flow matching for both human (17 H36M joints) and animal (26 Animal3D joints) pipelines
  • Add a convenience wrapper get_fmpose3d_inference_api() that creates a ready-to-use inference object with automatic weight downloading from HuggingFace Hub
  • Add integration tests covering API initialization (lightweight) and full end-to-end 2D-to-3D inference (network-dependent)
  • Add a Jupyter notebook recipe to the docs cookbook showing end-to-end usage, two-stage workflows, and how to lift existing DLC 2D predictions to 3D

Details

Core integration (fmpose3d.py): A thin wrapper that constructs an FMPose3DConfig and returns an FMPose3DInference object. Supports both fmpose3d_humans and fmpose3d_animals model types, optional custom snapshot paths, device selection, and extra config kwargs.

Tests (test_fmpose_integration.py):

  • test_api_init — parametrized over both model types; verifies the API object has the expected interface (prepare_2d, pose_3d, predict). Runs on CPU with no network required.
  • test_prepare_2d_and_pose_3d / test_predict_end_to_end — full integration tests that download weights and run inference on the included example image. Skipped automatically when no network is available.

Recipe notebook (fmpose3d.ipynb): Covers three workflows:

  1. End-to-end human pose estimation (single predict call)
  2. End-to-end animal pose estimation
  3. Two-stage pipeline (separate 2D detection + 3D lifting)
  4. Lifting existing DLC 2D predictions (from analyze_images or analyze_videos) to 3D

@deruyter92 deruyter92 marked this pull request as ready for review February 13, 2026 17:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant