Initial v2 release: 512-256-128 LayerNorm+GELU MLP distilled from img2pose on CelebV-HQ

README.md

@@ -0,0 +1,194 @@
+---
+tags:
+- pytorch
+- safetensors
+- pose-estimation
+- head-pose
+- landmark-to-pose
+- distillation
+- py-feat
+library_name: py-feat
+pipeline_tag: image-feature-extraction
+license: mit
+---
+
+# Py-Feat Pose-MLP v2 — Landmark-to-6DoF Head Pose
+
+A small distilled MLP that takes 68 face landmarks (the dlib-68 / OpenFace
+layout produced by `mobilefacenet`, OpenFace, etc.) and emits 6DoF head
+pose calibrated to img2pose's coordinate frame. Designed for `py-feat`
+pipelines that use a face detector without a built-in pose head (e.g.
+RetinaFace in `py-feat ≥ 0.7`).
+
+## Model Description
+
+`py-feat`'s v0.6 production pipeline used `img2pose` as its face detector,
+which multi-tasks face localization with 6DoF head pose regression — so
+pose came "for free" from the detector. In v0.7 the default face detector
+became `RetinaFace` (much higher WIDERFACE Hard AP) which only detects
+faces. To preserve the `Fex` schema (`pitch`, `roll`, `yaw`, `x`, `y`,
+`z` columns), `py-feat` distills img2pose's pose regression into a small
+MLP that operates entirely on already-computed landmarks.
+
+The MLP is bbox-free: it normalizes incoming landmarks by their centroid
+and inter-eye distance, so the same model works regardless of whether
+the upstream detector produced loose (img2pose) or tight (RetinaFace)
+face crops.
+
+## Model Details
+
+- **Model type**: Multi-layer perceptron (MLP)
+- **Architecture**: `Linear(136→512) → LayerNorm → GELU → Dropout(0.15)
+  → Linear(512→256) → LayerNorm → GELU → Dropout → Linear(256→128) →
+  LayerNorm → GELU → Dropout → Linear(128→6)`
+- **Parameter count**: 236,934 (~0.9 MB safetensors)
+- **Input**: 68 2D landmarks, normalized by landmark centroid and
+  inter-eye distance (`feat.utils.face_pose_mlp.normalize_landmarks`).
+- **Output**: 6 values — `[Pitch, Roll, Yaw, X, Y, Z]`. The MLP emits
+  z-scored values; the loader de-normalizes using `mean`/`std` stored in
+  the sidecar `pose_mlp_v2.json`. Angles are radians, calibrated to
+  img2pose's coordinate frame.
+- **Framework**: PyTorch (safetensors weight file, no pickle).
+- **Inference cost**: ~10 µs / face on CPU (batched), negligible vs.
+  the upstream face/landmark stages.
+
+## Training Details
+
+- **Teacher**: `img2pose` (Albiero et al., 2021). The MLP is trained to
+  match img2pose's regressed `[Pitch, Roll, Yaw, X, Y, Z]` outputs.
+- **Training corpus**: CelebV-HQ — `n_clips = 35,445`,
+  `n_train_frames = 2,783,134`, `n_val_frames = 154,619`. Frames with
+  `FaceScore < 0.8` or `|pose| > 75°` are dropped (filters bad teacher
+  signal on degenerate poses).
+- **Loss**: MSE on z-scored 6D output.
+- **Optimizer**: Adam, `lr=1e-3`, `batch_size=1024`.
+- **Epochs**: 40 (best val loss at last epoch — see `pose_mlp_v2.json`
+  for per-epoch history).
+- **Hardware**: single GPU (training takes ~2 hr).
+- **Seed**: 42.
+
+### Held-out validation MAE on CelebV-HQ (clip-disjoint split)
+
+| Axis | MAE (°) |
+|---|---|
+| Pitch | 2.66 |
+| Roll | 2.34 |
+| Yaw | 1.58 |
+
+For reference, img2pose's reported MAE on the AFLW2000-3D / BIWI test
+sets is ~4° average. The MLP cannot exceed its teacher; values here are
+the gap between the MLP and the teacher's predictions, not against a
+ground-truth motion-capture rig.
+
+### v1 → v2 changelog
+
+| Aspect | v1 | v2 |
+|---|---|---|
+| Hidden | 256→128→64 | 512→256→128 |
+| Activation | Linear → ReLU → Dropout | Linear → LayerNorm → GELU → Dropout |
+| Dropout | 0.10 | 0.15 |
+| Training frames | 569,678 | 2,783,134 |
+| Epochs | 30 | 40 |
+| Best val loss | 0.0809 | 0.0777 |
+| Roll MAE (°) | 2.530 | 2.335 |
+
+## Intended Use
+
+- **Primary**: Drop-in replacement for img2pose's pose head when using
+  `py-feat` with a face detector that doesn't predict pose
+  (`face_model='retinaface'` in `feat.Detector`, MediaPipe in
+  `feat.MPDetector`).
+- **Secondary**: Any pipeline that produces 68 dlib-style face landmarks
+  and wants img2pose-compatible head pose without re-running img2pose.
+
+### Out of scope
+
+- Eye / gaze direction — use `L2CS-Net` for gaze.
+- Mediapipe-478 landmarks — translate to 68 dlib landmarks first.
+- Static head-pose inference from a single landmark (less than 68 pts).
+
+## Usage
+
+The MLP is loaded automatically by `feat.Detector` when
+`face_model != 'img2pose'`. To call it directly:
+
+```python
+import torch
+from feat.utils.face_pose_mlp import pose_from_landmarks_mlp
+
+# 68 (x, y) landmarks in image-pixel coordinates, e.g. from mobilefacenet.
+landmarks = torch.tensor([
+    # ... [68, 2] ...
+], dtype=torch.float32).unsqueeze(0)  # [1, 68, 2]
+
+pose = pose_from_landmarks_mlp(landmarks)  # [1, 6]: (Pitch, Roll, Yaw, X, Y, Z)
+print(pose)
+```
+
+Weights resolve from (in order):
+1. `FEAT_POSE_MLP_PATH` environment variable
+2. `models/pose_mlp_v2.safetensors` in the repo
+3. This HuggingFace repo (`py-feat/pose_mlp_v2`)
+
+## Limitations
+
+- The MLP cannot improve on img2pose's accuracy — it only matches it
+  more efficiently with bbox-free input. Use img2pose directly if you
+  need img2pose's exact behavior (a tiny ~1° distillation gap may remain).
+- Trained on CelebV-HQ — performance on non-frontal, occluded, or
+  heavily-rotated faces (>75°) is degraded by both the teacher and the
+  data filter.
+- Output coordinates are img2pose's frame, not a standard FACS / BIWI
+  frame. Pose values are interpretable across the `py-feat` pipeline
+  but may need recalibration to compare with other tools.
+
+## Citation
+
+If you use `py-feat` and this pose-MLP, please cite both `py-feat` and
+img2pose:
+
+```bibtex
+@article{cheong2023pyfeat,
+  title={Py-Feat: Python Facial Expression Analysis Toolbox},
+  author={Cheong, Jin Hyun and Jolly, Eshin and Xie, Tiankang and Byrne, Sophie and Kenney, Matthew and Chang, Luke J.},
+  journal={Affective Science},
+  volume={4},
+  pages={781--796},
+  year={2023}
+}
+
+@inproceedings{albiero2021img2pose,
+  title={img2pose: Face Alignment and Detection via 6DoF, Face Pose Estimation},
+  author={Albiero, Vítor and Chen, Xingyu and Yin, Xi and Pang, Guan and Hassner, Tal},
+  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
+  pages={7617--7627},
+  year={2021}
+}
+
+@inproceedings{zhu2022celebvhq,
+  title={CelebV-HQ: A Large-Scale Video Facial Attributes Dataset},
+  author={Zhu, Hao and Wu, Wayne and Zhu, Wentao and Jiang, Liming and Tang, Siwei and Zhang, Li and Liu, Ziwei and Loy, Chen Change},
+  booktitle={Proceedings of the European Conference on Computer Vision (ECCV)},
+  year={2022}
+}
+```
+
+## License
+
+MIT (this distillation). The teacher (`img2pose`) is BSD-3, and the
+training corpus (CelebV-HQ) is released for non-commercial research
+use — please honor each upstream license if you re-train or
+re-distribute.
+
+## Files
+
+- `pose_mlp_v2.safetensors` — model weights (1 MB)
+- `pose_mlp_v2.json` — architecture, output-normalization stats, training
+  history, validation MAE per epoch
+- `README.md` — this card
+
+## Acknowledgments
+
+Distilled from img2pose by Vítor Albiero et al. (Meta AI / NVIDIA),
+trained on CelebV-HQ by Hao Zhu et al. (CUHK / S-Lab NTU). Built and
+maintained by [Cosanlab](https://cosanlab.com) at Dartmouth.

pose_mlp_v2.json

@@ -0,0 +1,368 @@
+{
+  "architecture": {
+    "type": "PoseMLP",
+    "hidden": [
+      512,
+      256,
+      128
+    ],
+    "dropout": 0.15,
+    "input_dim": 136,
+    "output_dim": 6,
+    "input_description": "68 (x, y) face landmarks normalized to face bbox [0,1]^2",
+    "output_description": "Pitch, Roll, Yaw (rad), X, Y, Z (units of img2pose template)",
+    "output_normalization": {
+      "mean": [
+        0.047110091894865036,
+        1.6245392531200054e-10,
+        -1.0606995282103071e-09,
+        -4.56669946302668e-09,
+        0.17980565130710602,
+        7.046011924743652
+      ],
+      "std": [
+        0.19027948379516602,
+        0.3928057849407196,
+        0.17927488684654236,
+        0.5255976915359497,
+        0.40686145424842834,
+        2.5513007640838623
+      ]
+    }
+  },
+  "training": {
+    "data": "/Storage/Projects/mp_blendshapes/data/bbox_pose/*_img2pose.csv (CelebV-HQ extraction)",
+    "n_clips": 35445,
+    "n_train_frames": 2783134,
+    "n_val_frames": 154619,
+    "face_score_threshold": 0.8,
+    "pose_clip_deg": 75.0,
+    "epochs": 40,
+    "batch_size": 1024,
+    "lr": 0.001,
+    "seed": 42
+  },
+  "best_val_loss": 0.0777415705746726,
+  "history": [
+    {
+      "epoch": 1,
+      "train": 0.11315019997803054,
+      "val": 0.104828848552547,
+      "pitch_mae_deg": 3.152665853500366,
+      "roll_mae_deg": 2.792881488800049,
+      "yaw_mae_deg": 2.13679838180542
+    },
+    {
+      "epoch": 2,
+      "train": 0.1065590488611336,
+      "val": 0.09961274236832794,
+      "pitch_mae_deg": 2.9992449283599854,
+      "roll_mae_deg": 2.6430251598358154,
+      "yaw_mae_deg": 1.9946171045303345
+    },
+    {
+      "epoch": 3,
+      "train": 0.10402416926071958,
+      "val": 0.09787551097963985,
+      "pitch_mae_deg": 2.9984378814697266,
+      "roll_mae_deg": 2.6911206245422363,
+      "yaw_mae_deg": 1.8750346899032593
+    },
+    {
+      "epoch": 4,
+      "train": 0.10253555926057764,
+      "val": 0.09612164048379973,
+      "pitch_mae_deg": 3.0127367973327637,
+      "roll_mae_deg": 2.6305713653564453,
+      "yaw_mae_deg": 1.8552767038345337
+    },
+    {
+      "epoch": 5,
+      "train": 0.10148297401859183,
+      "val": 0.09552087193649066,
+      "pitch_mae_deg": 2.9490880966186523,
+      "roll_mae_deg": 2.6785519123077393,
+      "yaw_mae_deg": 1.8824330568313599
+    },
+    {
+      "epoch": 6,
+      "train": 0.10051884218458308,
+      "val": 0.09247205514264734,
+      "pitch_mae_deg": 2.8985040187835693,
+      "roll_mae_deg": 2.477334976196289,
+      "yaw_mae_deg": 1.7141964435577393
+    },
+    {
+      "epoch": 7,
+      "train": 0.0995787868469493,
+      "val": 0.09395237100359641,
+      "pitch_mae_deg": 2.931995391845703,
+      "roll_mae_deg": 2.6331541538238525,
+      "yaw_mae_deg": 1.7413336038589478
+    },
+    {
+      "epoch": 8,
+      "train": 0.09886222485949048,
+      "val": 0.09294345720033896,
+      "pitch_mae_deg": 2.970853567123413,
+      "roll_mae_deg": 2.5920379161834717,
+      "yaw_mae_deg": 1.7464535236358643
+    },
+    {
+      "epoch": 9,
+      "train": 0.09830612088659829,
+      "val": 0.09182092106263888,
+      "pitch_mae_deg": 2.8500523567199707,
+      "roll_mae_deg": 2.511044502258301,
+      "yaw_mae_deg": 1.784227728843689
+    },
+    {
+      "epoch": 10,
+      "train": 0.09737222732988414,
+      "val": 0.09004505537450314,
+      "pitch_mae_deg": 2.873867988586426,
+      "roll_mae_deg": 2.472165107727051,
+      "yaw_mae_deg": 1.7860360145568848
+    },
+    {
+      "epoch": 11,
+      "train": 0.09690498495700607,
+      "val": 0.08976648708707408,
+      "pitch_mae_deg": 2.879437208175659,
+      "roll_mae_deg": 2.561363697052002,
+      "yaw_mae_deg": 1.6688872575759888
+    },
+    {
+      "epoch": 12,
+      "train": 0.09630656271153201,
+      "val": 0.08980230604739566,
+      "pitch_mae_deg": 2.838160514831543,
+      "roll_mae_deg": 2.4794445037841797,
+      "yaw_mae_deg": 1.7088587284088135
+    },
+    {
+      "epoch": 13,
+      "train": 0.09569354373427154,
+      "val": 0.08851547088277967,
+      "pitch_mae_deg": 2.856058120727539,
+      "roll_mae_deg": 2.5598247051239014,
+      "yaw_mae_deg": 1.7351120710372925
+    },
+    {
+      "epoch": 14,
+      "train": 0.09516716494366823,
+      "val": 0.08757703378796577,
+      "pitch_mae_deg": 2.800903558731079,
+      "roll_mae_deg": 2.4999215602874756,
+      "yaw_mae_deg": 1.7200062274932861
+    },
+    {
+      "epoch": 15,
+      "train": 0.09462323989595799,
+      "val": 0.08703844112000968,
+      "pitch_mae_deg": 2.810861110687256,
+      "roll_mae_deg": 2.425896167755127,
+      "yaw_mae_deg": 1.7037086486816406
+    },
+    {
+      "epoch": 16,
+      "train": 0.09410698983524984,
+      "val": 0.08680682148980468,
+      "pitch_mae_deg": 2.833448648452759,
+      "roll_mae_deg": 2.535163402557373,
+      "yaw_mae_deg": 1.7124162912368774
+    },
+    {
+      "epoch": 17,
+      "train": 0.09365605068138805,
+      "val": 0.08569933728952157,
+      "pitch_mae_deg": 2.8009047508239746,
+      "roll_mae_deg": 2.5543301105499268,
+      "yaw_mae_deg": 1.6394764184951782
+    },
+    {
+      "epoch": 18,
+      "train": 0.09323946791999124,
+      "val": 0.0869463943925343,
+      "pitch_mae_deg": 2.83258318901062,
+      "roll_mae_deg": 2.469459056854248,
+      "yaw_mae_deg": 1.6592215299606323
+    },
+    {
+      "epoch": 19,
+      "train": 0.09272424481134804,
+      "val": 0.0835315509650268,
+      "pitch_mae_deg": 2.7636806964874268,
+      "roll_mae_deg": 2.4162306785583496,
+      "yaw_mae_deg": 1.668568730354309
+    },
+    {
+      "epoch": 20,
+      "train": 0.09233342655512934,
+      "val": 0.08383207366262611,
+      "pitch_mae_deg": 2.770354747772217,
+      "roll_mae_deg": 2.4920639991760254,
+      "yaw_mae_deg": 1.6237024068832397
+    },
+    {
+      "epoch": 21,
+      "train": 0.09185766826731216,
+      "val": 0.08318848182496272,
+      "pitch_mae_deg": 2.7563376426696777,
+      "roll_mae_deg": 2.4593088626861572,
+      "yaw_mae_deg": 1.712731122970581
+    },
+    {
+      "epoch": 22,
+      "train": 0.09151670299443013,
+      "val": 0.08192426515252967,
+      "pitch_mae_deg": 2.74179744720459,
+      "roll_mae_deg": 2.418275833129883,
+      "yaw_mae_deg": 1.71445631980896
+    },
+    {
+      "epoch": 23,
+      "train": 0.09108457101112871,
+      "val": 0.0820473910946595,
+      "pitch_mae_deg": 2.7692880630493164,
+      "roll_mae_deg": 2.402466058731079,
+      "yaw_mae_deg": 1.641359806060791
+    },
+    {
+      "epoch": 24,
+      "train": 0.0906762100339506,
+      "val": 0.08174176327884197,
+      "pitch_mae_deg": 2.733238458633423,
+      "roll_mae_deg": 2.4820797443389893,
+      "yaw_mae_deg": 1.6439942121505737
+    },
+    {
+      "epoch": 25,
+      "train": 0.09034998959712985,
+      "val": 0.08133541057376485,
+      "pitch_mae_deg": 2.7302887439727783,
+      "roll_mae_deg": 2.4201176166534424,
+      "yaw_mae_deg": 1.6617894172668457
+    },
+    {
+      "epoch": 26,
+      "train": 0.09008330495711457,
+      "val": 0.08046494296898968,
+      "pitch_mae_deg": 2.7038564682006836,
+      "roll_mae_deg": 2.3806567192077637,
+      "yaw_mae_deg": 1.6689109802246094
+    },
+    {
+      "epoch": 27,
+      "train": 0.08978113147476037,
+      "val": 0.07989089151746348,
+      "pitch_mae_deg": 2.692418336868286,
+      "roll_mae_deg": 2.4487438201904297,
+      "yaw_mae_deg": 1.6251274347305298
+    },
+    {
+      "epoch": 28,
+      "train": 0.08945507099516169,
+      "val": 0.07935293018817902,
+      "pitch_mae_deg": 2.6850106716156006,
+      "roll_mae_deg": 2.3853209018707275,
+      "yaw_mae_deg": 1.5958898067474365
+    },
+    {
+      "epoch": 29,
+      "train": 0.0891421977581348,
+      "val": 0.07996009172577608,
+      "pitch_mae_deg": 2.693944215774536,
+      "roll_mae_deg": 2.375354528427124,
+      "yaw_mae_deg": 1.6029860973358154
+    },
+    {
+      "epoch": 30,
+      "train": 0.08896089701846822,
+      "val": 0.0793423713429978,
+      "pitch_mae_deg": 2.6876747608184814,
+      "roll_mae_deg": 2.370105743408203,
+      "yaw_mae_deg": 1.6090757846832275
+    },
+    {
+      "epoch": 31,
+      "train": 0.08868927482551209,
+      "val": 0.0788348114216014,
+      "pitch_mae_deg": 2.680671453475952,
+      "roll_mae_deg": 2.3722896575927734,
+      "yaw_mae_deg": 1.5897678136825562
+    },
+    {
+      "epoch": 32,
+      "train": 0.08841876725914989,
+      "val": 0.07828495494629208,
+      "pitch_mae_deg": 2.6755311489105225,
+      "roll_mae_deg": 2.3592045307159424,
+      "yaw_mae_deg": 1.5938224792480469
+    },
+    {
+      "epoch": 33,
+      "train": 0.08829338179738142,
+      "val": 0.07879007970424075,
+      "pitch_mae_deg": 2.672231674194336,
+      "roll_mae_deg": 2.3470544815063477,
+      "yaw_mae_deg": 1.583457112312317
+    },
+    {
+      "epoch": 34,
+      "train": 0.08804849298661294,
+      "val": 0.07809831976498428,
+      "pitch_mae_deg": 2.6717796325683594,
+      "roll_mae_deg": 2.345898151397705,
+      "yaw_mae_deg": 1.5844926834106445
+    },
+    {
+      "epoch": 35,
+      "train": 0.08788964650137973,
+      "val": 0.07806965493057903,
+      "pitch_mae_deg": 2.665562629699707,
+      "roll_mae_deg": 2.343658924102783,
+      "yaw_mae_deg": 1.5944466590881348
+    },
+    {
+      "epoch": 36,
+      "train": 0.08776004950435745,
+      "val": 0.07801351490381517,
+      "pitch_mae_deg": 2.6699180603027344,
+      "roll_mae_deg": 2.347235918045044,
+      "yaw_mae_deg": 1.588186502456665
+    },
+    {
+      "epoch": 37,
+      "train": 0.08765846724136074,
+      "val": 0.07789983218045611,
+      "pitch_mae_deg": 2.6638052463531494,
+      "roll_mae_deg": 2.3336102962493896,
+      "yaw_mae_deg": 1.5877548456192017
+    },
+    {
+      "epoch": 38,
+      "train": 0.08762438727969014,
+      "val": 0.0777415705746726,
+      "pitch_mae_deg": 2.659308433532715,
+      "roll_mae_deg": 2.336526393890381,
+      "yaw_mae_deg": 1.5783231258392334
+    },
+    {
+      "epoch": 39,
+      "train": 0.08753714936622983,
+      "val": 0.0779279994925386,
+      "pitch_mae_deg": 2.662292003631592,
+      "roll_mae_deg": 2.334559440612793,
+      "yaw_mae_deg": 1.5846630334854126
+    },
+    {
+      "epoch": 40,
+      "train": 0.08748793931207821,
+      "val": 0.07779885426555809,
+      "pitch_mae_deg": 2.6598613262176514,
+      "roll_mae_deg": 2.335223436355591,
+      "yaw_mae_deg": 1.578811764717102
+    }
+  ]
+}

pose_mlp_v2.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2e237215e977b334d34b3116b77cd7e83a0bae98189fb6700886b85dcde4dfe5
+size 948800