WHENet

Overview

WHENet is a lightweight head-pose estimation model designed to run in real time on mobile devices. Given an RGB face image, the network regresses the orientation of the head as Euler angles (yaw, pitch, roll). Unlike most head-pose models, WHENet is able to regress the full 360 degrees of yaw, making it applicable to wide-range head-pose scenarios such as autonomous driving and retail. The network architecture is based on EfficientNet-B0.

This repository contains the WHENet model quantized to int8 and compiled for NXP i.MX 8M Plus, i.MX 93, i.MX 95 and i.MX 952 processors. It also contains the script used to regenerate these assets (recipe.sh) and an inference example (example.py).

Note: This model only estimates head orientation. It does not perform face detection, so the input image is expected to already contain a cropped, roughly centered face.

Key Features

  • Lightweight EfficientNet-B0 backbone for real-time head-pose estimation.
  • Full 360-degree yaw range (yaw, pitch, roll Euler angles).
  • Angle predictions decoded from binned distributions (bins of 3 degrees); see example.py and utils.py for the decoding logic.
  • Fully int8 quantized (int8 input, int8 output) for accelerated NPU inference on NXP i.MX platforms.

Model Description

This is a quantized and compiled version of the WHENet head-pose estimation model, optimized for NXP i.MX 8M Plus, i.MX 93, i.MX 95 and i.MX 952 processors.

  • Base Model: WHENet
  • Original Model Authors: Yijun Zhou and James Gregson
  • Original License: BSD-3-Clause
  • Modified by: NXP

Modifications

This model is a derivative work with the following changes from the original:

  • Format conversion: converted from Keras to TensorFlow Lite with tf.lite.TFLiteConverter.from_concrete_functions.
  • Quantization: full int8 post-training quantization (int8 input, int8 output). Because the original 300W-LP calibration dataset is released for non-commercial research use only, the representative calibration dataset used here is instead ~100 images sampled from a commercially-usable ImageNetV2 subset, resized to the 224x224 model input and normalized with the ImageNet mean/std used by the WHENet pre-processing pipeline (see create_calib_data.py).
  • Compilation (i.MX 93): compiled with Vela for the Ethos-U65 NPU, tags lf-6.18.20_2.0.0 and lf-6.18.37_2.1.0.
  • Conversion (i.MX 95 / i.MX 952): converted with the eIQ Neutron SDK for the Neutron NPU, versions 3.1.3, 3.2.1 and 3.2.3 for i.MX 95, and 3.1.3 and 3.2.1 for i.MX 952.
  • Post-processing: the binned-angle decoding (softmax over the yaw/pitch/roll bins, then weighted sum) is performed outside the model graph (see utils.py).

Original model available at: https://github.com/Ascend-Research/HeadPoseEstimation-WHENet.

Model Information

Information Value
Input shape RGB image (1, 224, 224, 3), int8
Input example (Image source, Public domain)
Output shape 3 tensors: yaw (1, 120), pitch (1, 66), roll (1, 66), int8 quantized; dequantize with each tensor's scale / zero point and decode as a binned angle distribution (3 degrees per bin). See example.py / utils.py.
Output example
FLOPS 781M
Number of parameters 4.4M
File size (int8) 5.0 MB (assets/quantized_tflite_model/whenet.tflite)
Source framework TensorFlow (Keras)

Supported Hardware and Software Configurations

All models run with the eIQ LiteRT (TensorFlow Lite) runtime and the corresponding NPU delegate (VX for i.MX 8M Plus, Ethos-U for i.MX 93, Neutron for i.MX 95 / i.MX 952). The same information is captured in release_manifest.json.

Asset Runtime Precision HW target BSP & SDK versions Link to asset
i.MX 8M Plus Quantized LiteRT INT8 i.MX 8M Plus - Download
i.MX 93 Precompiled LiteRT INT8 i.MX 93 BSP lf-6.18.20_2.0.0, VELA_TAG = lf-6.18.20_2.0.0 Download
i.MX 93 Precompiled LiteRT INT8 i.MX 93 BSP lf-6.18.37_2.1.0, VELA_TAG = lf-6.18.37_2.1.0 Download
i.MX 95 Precompiled LiteRT INT8 i.MX 95 BSP lf-6.18.20_2.0.0, Neutron SDK v3.1.3 Download
i.MX 95 Precompiled LiteRT INT8 i.MX 95 BSP lf-6.18.37_2.1.0, Neutron SDK v3.2.1 Download
i.MX 95 Precompiled LiteRT INT8 i.MX 95 BSP lf-6.18.37_2.1.0, Neutron SDK v3.2.3 Download
i.MX 952 Precompiled LiteRT INT8 i.MX 952 BSP lf-6.18.20_2.0.0, Neutron SDK v3.1.3 Download
i.MX 952 Precompiled LiteRT INT8 i.MX 952 BSP lf-6.18.37_2.1.0, Neutron SDK v3.2.1 Download

The assets/imx8mplus/whenet.tflite file is identical (same bytes) to the hardware-independent int8 model at assets/quantized_tflite_model/whenet.tflite, which is also provided for host (x86) inference and as the input to the i.MX 93 / i.MX 95 / i.MX 952 compilers.

Limitations and Biases

This model inherits all limitations from the original WHENet model. Additional limitations:

  • Task scope: head-pose (yaw/pitch/roll) estimation only. The model does not detect faces; it expects an already-cropped, roughly centered face image and will produce unreliable output otherwise.
  • Input resolution: the model only accepts a fixed 224x224 input; faces must be resized/cropped to this resolution before inference.
  • Dataset bias: the original model was trained on 300W-LP and the CMU Panoptic Dataset, and evaluated on BIWI and AFLW2000. Head-pose accuracy may vary for poses, lighting conditions or demographic groups that are under-represented in those datasets. The int8 calibration set used in this repository (ImageNetV2) is a generic image dataset, not a face dataset, and was chosen only because it is commercially usable; it was not used for training.

Training

The model was trained by the original authors on the 300W-LP dataset and the CMU Panoptic Dataset. It was evaluated by the original authors on the BIWI and AFLW2000 datasets. NXP did not retrain the model; more information about the original training and evaluation is available in the original paper.

Download and Run

Running the example

example.py runs head-pose estimation on an image on a host PC and draws the yaw/pitch/roll axes onto the output image. Install the dependencies from requirements.txt first.

# Run on PC:
python example.py -m assets/quantized_tflite_model/whenet.tflite -i example_input.jpg -o example_output.jpg

All three arguments are optional and default to the values shown above (-m/--model, -i/--input, -o/--output).

To run the model on an i.MX MPU target with the NPU delegates, see the i.MX Machine Learning User Guide.

Regenerating the assets with recipe.sh

The recipe installs the eIQ Neutron SDK from the NXP eIQ PyPI Distribution Portal. Configure pip once to use it:

python -m pip config --user set global.index https://eiq.nxp.com/repository
python -m pip config --user set global.index-url https://eiq.nxp.com/repository
python -m pip config --user set global.trusted-host eiq.nxp.com
python -m pip config --user set global.extra-index-url https://pypi.org/simple
python -m pip config --user set global.find-links https://eiq.nxp.com/repository/

To recreate the int8 TFLite model, point DATASET_PATH at a local ImageNetV2 subset (a directory of per-class image subfolders) and run:

DATASET_PATH=/path/to/imagenetv2 bash recipe.sh

The Vela tag and the eIQ Neutron SDK versions for i.MX 95 and i.MX 952 can be passed as positional arguments (defaulting to lf-6.18.20_2.0.0, 3.2.3, and 3.2.1 respectively):

DATASET_PATH=/path/to/imagenetv2 bash recipe.sh lf-6.18.37_2.1.0 3.2.3 3.2.1

recipe.sh downloads the original WHENet.h5 weights, converts and quantizes the model to int8, then compiles / converts it for each target. The generated files are:

  • assets/quantized_tflite_model/whenet.tflite - hardware-independent int8 model, also used as-is for i.MX 8M Plus (assets/imx8mplus/whenet.tflite)
  • assets/imx93/<VELA_TAG>/whenet_vela.tflite - Vela-compiled for i.MX 93
  • assets/imx95/<NEUTRON_SDK_VERSION>/whenet_converted.tflite - Neutron-converted for i.MX 95
  • assets/imx952/<NEUTRON_SDK_VERSION>/whenet_converted.tflite - Neutron-converted for i.MX 952

License

This repository, including the quantized and compiled assets and the scripts in this repository, is released under the MIT License (see LICENSE). The original WHENet model and the utils.py code derived from it are released under the BSD-3-Clause License (see MODEL_LICENSE).

Citation

If you use this model, please cite both this work and the original model:

  • Y. Zhou and J. Gregson, "WHENet: Real-time Fine-grained Estimation for Wide Range Head Pose," arXiv preprint arXiv:2005.10353, 2020.
  • M. Tan and Q. Le, "EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks," International Conference on Machine Learning (ICML), PMLR, 2019.
  • X. Zhu et al., "Face Alignment Across Large Poses: A 3D Solution," Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
  • H. Joo et al., "Panoptic Studio: A Massively Multiview System for Social Motion Capture," Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2015.
  • G. Fanelli et al., "Random Forests for Real Time 3D Face Analysis," International Journal of Computer Vision, 101 (2013): 437-458.
@article{zhou2020whenet,
  title={Whenet: Real-time fine-grained estimation for wide range head pose. arXiv 2020},
  author={Zhou, Y and Gregson, J},
  journal={arXiv preprint arXiv:2005.10353},
  year={2020}
}

@inproceedings{tan2019efficientnet,
  title={Efficientnet: Rethinking model scaling for convolutional neural networks},
  author={Tan, Mingxing and Le, Quoc},
  booktitle={International conference on machine learning},
  pages={6105--6114},
  year={2019},
  organization={PmLR}
}

@inproceedings{zhu2016face,
  title={Face alignment across large poses: A 3d solution},
  author={Zhu, Xiangyu and Lei, Zhen and Liu, Xiaoming and Shi, Hailin and Li, Stan Z},
  booktitle={Proceedings of the IEEE conference on computer vision and pattern recognition},
  pages={146--155},
  year={2016}
}

@inproceedings{joo2015panoptic,
  title={Panoptic studio: A massively multiview system for social motion capture},
  author={Joo, Hanbyul and Liu, Hao and Tan, Lei and Gui, Lin and Nabbe, Bart and Matthews, Iain and Kanade, Takeo and Nobuhara, Shohei and Sheikh, Yaser},
  booktitle={Proceedings of the IEEE international conference on computer vision},
  pages={3334--3342},
  year={2015}
}

@article{fanelli2013random,
  title={Random forests for real time 3d face analysis},
  author={Fanelli, Gabriele and Dantone, Matthias and Gall, Juergen and Fossati, Andrea and Van Gool, Luc},
  journal={International journal of computer vision},
  volume={101},
  number={3},
  pages={437--458},
  year={2013},
  publisher={Springer}
}
Downloads last month
79
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Collections including nxp/whenet-imx

Papers for nxp/whenet-imx