# SignBridge — technical walkthrough

> Internal technical record of the build. Not a submission deliverable
> (Build-in-Public extra challenge was dropped on 2026-05-07).
> Kept around because it documents the AMD-specific engineering thinking
> and is useful if anyone later asks "why these design choices?".

## What we built

A real-time webcam-based ASL → English speech translator. A deaf user signs
into the webcam; the pipeline (MediaPipe Holistic → trained sign classifier
→ Llama-3.1-8B sentence composer → Coqui XTTS-v2) returns spoken English
in under 2 seconds. Designed to fit Track 3 (Vision & Multimodal AI) with
the entire model stack running concurrently on a single AMD Instinct MI300X.

## Why AMD MI300X

- 192 GB HBM3 — the trained classifier (~20 MB), Llama-3.1-8B (~16 GB FP16),
  XTTS-v2 (~2 GB), and (V2 stretch) Whisper-large-v3 (~3 GB) all fit
  concurrently with margin for KV cache.
- 5.3 TB/s memory bandwidth — bandwidth-bound streaming workload (many
  small inferences per second on the classifier + TTS chunked decode + LLM
  next-token) is exactly what bandwidth wins.

## Architecture

```
webcam frames → MediaPipe Holistic → trained classifier
                  (CPU-fast)            (TorchScript on MI300X)
                                              │
                                              ▼
                                  Llama-3.1-8B sentence composer
                                       (vLLM on MI300X)
                                              │
                                              ▼
                                          XTTS-v2 → audio
                                       (XTTS on MI300X)
```

## Models

| Component | Source | Notes |
|---|---|---|
| Pose extractor | MediaPipe Holistic (Google) | CPU-fast preprocessing — not GPU-bound |
| Sign classifier | trained from scratch on WLASL Top-100 + ASL fingerspelling alphabet | 3-layer transformer encoder over 543-dim landmark sequences; published to HF Hub at `lucas-loo/signbridge-classifier` |
| Sentence composer | `meta-llama/Llama-3.1-8B-Instruct` | Pulled from HF Hub; served on MI300X via vLLM |
| Text-to-speech | `coqui/XTTS-v2` | Multilingual; we use English V1 |

## Datasets

- [WLASL](https://github.com/dxli94/WLASL) Top-100 subset
- ASL fingerspelling alphabet (open dataset)

## ROCm / AMD Developer Cloud experience

> *Filled in across Day 1–3.*

### Day 1 — environment + sanity
TODO

### Day 2 — training the classifier
TODO

### Day 3 — serving + latency tuning
TODO

### What worked well
TODO

### What we'd flag as friction
TODO

## Why AMD MI300X — concretely

The pipeline (MediaPipe Holistic + Qwen3-VL-8B + Llama-3.1-8B + Coqui XTTS-v2)
fits comfortably on a single MI300X with KV-cache headroom. The same workload
on NVIDIA forces sharding once we add the V2 reasoner.

| Component | Weights (FP16) | MI300X 1× (192 GB) | H100 80 GB | H200 141 GB |
|---|---|---|---|---|
| Qwen3-VL-8B (vision) | ~16 GB | ✅ fits | ✅ | ✅ |
| Llama-3.1-8B (composer) | ~16 GB | ✅ fits | ✅ | ✅ |
| Whisper-large-v3 (V2 reverse direction) | ~3 GB | ✅ fits | ⚠ tight | ✅ |
| Coqui XTTS-v2 (TTS) | ~2 GB | ✅ fits | ⚠ tight | ✅ |
| (V2) Llama-3.1-70B FP8 reasoner upgrade | ~70 GB | ✅ still fits | ❌ doesn't fit at all | ⚠ FP8 only, no headroom |
| **Concurrent serving + KV cache** | ✅ comfortable | ❌ requires sharding | ⚠ tight | ✅ |

The single-GPU concurrency story is the AMD pitch. V1 fits anywhere; the
architecture has clear MI300X headroom for V2 model upgrades that NVIDIA
H100 cannot match without sharding across multiple cards.

## Deployment ethics

SignBridge is a *substrate*, not a finished product. We ship the open-source
multi-modal pipeline so Deaf-led organisations — schools-for-the-Deaf, regional
NGOs, ministries of social services — can deploy on their own AMD compute,
fine-tune for their dialect, and own the deployment.

Three principles, drawn from the Deaf-led literature on sign-language AI:

1. **ASL only V1** is a scope decision. Sign languages are not interchangeable
   — BSL, ISL, MSL, CSL, and 200+ other sign languages each deserve their own
   teams, training data, and Deaf community leadership. Bragg et al.
   ["Systemic Biases in Sign Language AI Research"](https://arxiv.org/html/2403.02563v1)
   (2024, Deaf-led position paper) is direct on this point.

2. **Deaf community engagement before deployment.** Per the ACM ASSETS 2025
   ["Exploring Collaboration to Center the Deaf Community in Sign Language AI"](https://dl.acm.org/doi/10.1145/3663547.3746390)
   the productive ML/Deaf collaboration question isn't "how do we build this?"
   but "*should* we build this, *for whom*, *with whom*?". Any deployment
   downstream of this code must answer that locally.

3. **Privacy by default.** SignBridge sessions are ephemeral — webcam frames
   and audio are processed in-memory and not persisted server-side beyond the
   request lifetime. In the spirit of [Privacy-Aware Sign Language Translation
   at Scale](https://aclanthology.org/2024.acl-long.467.pdf) (ACL 2024).

## Future work — academic foundations we'd build on next

- **SignCLIP** ([EMNLP 2024](https://aclanthology.org/2024.emnlp-main.518.pdf)) —
  learned text↔sign embeddings; replaces the prompt-only composer with a
  CLIP-style alignment head for higher-quality sign-to-English mapping.
- **SL-SLR** ([arXiv 2509.05188v1](https://arxiv.org/html/2509.05188v1)) —
  self-supervised representation learning with motion-aware data augmentation;
  the right path if we ever train a custom classifier on raw signer footage.
- **Continuous SLT trained models** (Swin-MSTP, Stack Transformer) — the
  current trained-from-scratch ceiling on WLASL is ~93.5% Top-1. The VLM
  zero-shot path we ship here is a *deployment-cost* play, not an
  accuracy-ceiling play; SignCLIP-style learned embeddings are the natural
  V2 step toward that ceiling.

## Latency

Target: ≤ 2 s from end-of-sign to start of speech.

Measured on a single MI300X (Day 3):
- MediaPipe Holistic per frame: TODO ms
- Classifier per window: TODO ms
- Llama-3.1-8B sentence composition (≤ 30 tokens): TODO ms
- XTTS-v2 first-audio-chunk: TODO ms

## MI300X vs NVIDIA H100 — the AMD pitch

| Item | MI300X (1 GPU) | H100 (1 GPU) | H100 cluster needed |
|---|---|---|---|
| Llama-3.1-8B FP16 weights | ✅ fits with margin | ✅ fits with margin | 1× |
| + XTTS-v2 + Whisper-large-v3 + classifier | ✅ all concurrent | ⚠️ tight (~28 GB total + KV) | likely 1× but no headroom |
| + 70B reasoner upgrade (V2) | ✅ 70B FP8 ~70 GB still fits | ❌ doesn't fit at all | ≥3× |

The single-GPU concurrency story is the AMD pitch. This V1 fits on H100;
the architecture has clear headroom on MI300X for higher-quality V2 models.

## License

MIT.