README.md

# PriviGaze: Privileged Distillation for Accessible Gaze Estimation

**On-device gaze estimation designed for people with disabilities.**

PriviGaze uses **privileged knowledge distillation** to train an ultra-compact student model (~80K params) that estimates gaze direction from just a grayscale face image — no eye crops, no RGB, no calibration needed.

## Why This Matters

Traditional gaze trackers fail for people with disabilities:
- 👁️ **Droopy eyes** → eye crop detectors can't find pupils
- 🔄 **Head roll/mobile instability** → calibration breaks
- 💡 **Varied lighting** → RGB-based models fail

PriviGaze's student model handles all of these by:
- Working from the **full face** (no precise eye detection needed)
- Using **grayscale only** (robust to lighting)
- Having a **large receptive field** (handles head movement)
- Being **~80K parameters** (runs on any device)

## Architecture

### Teacher (Training Only - Privileged Information)
```
┌─────────────────────────────────────────────────┐
│              PriviGazeTeacher                     │
│                                                   │
│  Left Eye RGB ──→ ConvNeXtV2-Atto ──→ 256d      │
│  Right Eye RGB ─→ ConvNeXtV2-Atto ──→ 256d      │
│                         ↓ (Fusion)                │
│  Face Blurred ──→ ConvNeXtV2-Nano ──→ 256d      │
│  (Grayscale)         ↓ (Cross-Attention)         │
│                   ┌──────────┐                    │
│                   │  Fused   │                    │
│                   │ Features │                    │
│                   │   256d   │                    │
│                   └────┬─────┘                    │
│                   ┌────┴─────┐                    │
│                   │ Pitch │ Yaw │                 │
│                   └─────────────┘                 │
└─────────────────────────────────────────────────┘
```
- 3 privileged inputs: left eye RGB, right eye RGB, blurred grayscale face
- ConvNeXtV2-Atto (3.7M) for eyes, ConvNeXtV2-Nano (15.6M) for face
- Cross-attention fusion between face and eye modalities
- L2CS-Net style binned regression

### Student (On-Device Inference)
```
┌─────────────────────────────────────────────────┐
│              PriviGazeStudent                     │
│                    ~80K params                     │
│                                                   │
│  Face Grayscale ──→ Light Correction              │
│       ↓                                           │
│  Stem (32ch, /4)                                  │
│       ↓                                           │
│  Inception Block → DSConv (/2) → 64ch            │
│       ↓                                           │
│  Inception Block → DSConv (/2) → 96ch            │
│       ↓                                           │
│  Inception Block → DSConv (/2) → 128ch           │
│       ↓                                           │
│  Inception Block → GAP → 160ch                   │
│       ↓                                           │
│  Feature Projection → 128d                        │
│       ↓                                           │
│  ┌────┴─────┐                                     │
│  │ Pitch │ Yaw │                                  │
│  └─────────────┘                                  │
└─────────────────────────────────────────────────┘
```
- 1 input: grayscale face (224×224)
- **Inception blocks** with factorized convolutions (1×3 + 3×1)
- **Depthwise separable convolutions** throughout
- **Learned light correction** (gamma + affine)
- L2CS-Net style binned regression

### Distillation Loss

The student learns from the teacher via a multi-component loss:

```
L_total = L_task + α_angular·L_angular + α_contrast·L_contrast + α_mmd·L_mmd + α_logit·L_logit
```

| Component | Weight | Description |
|-----------|--------|-------------|
| L_task | 1.0 | L2CS-Net binned regression (CE + MSE) |
| L_angular | 1.0 | Direct L1 in degrees |
| L_contrast | 0.5 | InfoNCE contrastive feature matching |
| L_mmd | 0.1 | Maximum Mean Discrepancy distribution matching |
| L_logit | 0.5 | KL divergence on soft targets |

## Training

### Quick Start

```bash
# Install dependencies
pip install -r requirements.txt

# Train teacher first, then distill to student
python train.py --mode both \
    --batch-size 32 \
    --epochs 100 \
    --teacher-epochs 50 \
    --save-dir ./checkpoints \
    --push-to-hub \
    --hub-model-id BcantCode/privi-gaze-distill
```

### Phase 1: Teacher Pre-training
```bash
python train.py --mode pretrain_teacher \
    --batch-size 32 \
    --teacher-epochs 50 \
    --save-dir ./checkpoints
```

### Phase 2: Student Distillation  
```bash
python train.py --mode distill \
    --teacher-path ./checkpoints/teacher_best.pt \
    --epochs 100 \
    --batch-size 32 \
    --save-dir ./checkpoints
```

## Model Sizes

| Model | Parameters | Input | Use |
|-------|-----------|-------|-----|
| PriviGazeTeacher | ~19M | 2×RGB eyes + blurred face | Training only |
| PriviGazeStudent | ~80K | 1×grayscale face | On-device inference |

## Research Foundation

This work builds on:

- **L2CS-Net** (Abdelrahman et al., 2022): Per-angle binned regression for gaze
- **GazeGen / DFT Gaze** (Hsieh et al., 2024): 281K distilled gaze model from 10× larger teacher
- **WCoRD** (Chen et al., 2020): Wasserstein contrastive representation distillation
- **One Eye is All You Need** (Athavale et al., 2022): Inception networks for lightweight gaze
- **ETH-XGaze** (Zhang et al., 2020): Large-scale gaze dataset with extreme head poses

## Dataset

Currently uses **SyntheticGazeDataset** for development. The synthetic generator creates realistic eye crops with pupil positions encoding gaze direction, plus face images with corresponding features.

For production use, the pipeline supports:
- **MPIIFaceGaze**: 15 subjects, face crops + eye patches + 3D gaze
- **ETH-XGaze**: 110 subjects, extreme head poses, 1.1M images (gold standard)
- **Gaze360**: 238 subjects, 360° gaze range

To use real datasets, implement the `MPIIGazeDataset` class in `models/dataset.py`.

## Requirements

- Python ≥ 3.9
- PyTorch ≥ 2.0
- Transformers ≥ 4.40
- CUDA-capable GPU (for training)

## License

Apache 2.0

## Citation

```
@software{privi_gaze_2026,
  title={PriviGaze: Privileged Distillation for Accessible Gaze Estimation},
  year={2026},
  url={https://huggingface.co/BcantCode/privi-gaze-distill}
}
```