docs: model card with bench numbers + privacy contract

README.md

@@ -0,0 +1,305 @@
+---
+license: cc-by-nc-4.0
+language:
+  - en
+library_name: onnxruntime
+pipeline_tag: object-detection
+tags:
+  - pii
+  - privacy
+  - redaction
+  - object-detection
+  - rf-detr
+  - screen-capture
+  - accessibility
+  - computer-use
+  - agentic
+  - screenpipe
+metrics:
+  - zero-leak
+  - oversmash
+  - precision
+  - recall
+extra_gated_prompt: >-
+  This model is licensed CC BY-NC 4.0 (non-commercial). For commercial
+  use — production deployment, SaaS / API embedding, agent privacy
+  middleware, custom fine-tunes — contact louis@screenpi.pe.
+---
+
+# screenpipe-pii-image-redactor
+
+> A [screenpipe](https://screenpi.pe) project. The image-modality
+> companion to [`screenpipe/pii-redactor`](https://huggingface.co/screenpipe/pii-redactor).
+
+A fine-tuned **image PII detector** for the same three surfaces an AI
+agent sees a user's machine through:
+
+1. **Screen captures** — JPGs / PNGs of the user's screen, rendered
+   text and structured chrome (Slack, Outlook, Cursor, Terminal,
+   Confluence, GitHub, 1Password, calendars, browsers).
+2. **Computer-use traces** — the visual frames an agentic model
+   (Claude Computer Use, GPT operator, etc.) reads when it controls a
+   desktop.
+3. **Accessibility-tree visualizations** — when an agent screenshots
+   what it inferred from the AX tree to debug a tool call.
+
+These surfaces are **dense, multi-PII, semi-structured** in ways no
+prose-trained PII detector handles well. Returns pixel-space bounding
+boxes for 12 canonical PII categories.
+
+ONNX, ~108 MB. Same `.onnx` ships across macOS / Windows / Linux —
+the user's ONNX Runtime selects the Execution Provider at load time
+(CoreML, DirectML, CUDA, or CPU baseline).
+
+> **License: CC BY-NC 4.0** (non-commercial). For commercial use —
+> production redaction, SaaS / API embedding, AI-agent privacy
+> middleware, custom fine-tunes — contact **louis@screenpi.pe**. See
+> [`LICENSE`](LICENSE).
+
+## Headline numbers
+
+`rfdetr_v8` on a held-out 221-image validation split (190 PII-bearing,
+31 hard negatives) of the [screenpipe-pii-bench-image](https://github.com/screenpipe/screenpipe-pii-bench-image)
+corpus, IoU ≥ 0.30:
+
+| metric | this model | regex+OCR floor | Microsoft Presidio (published OSS) |
+|---|---:|---:|---:|
+| **zero-leak** (every gold span caught) | **95.3%** | 2.6% | 0.5% |
+| **oversmash** (false-fire on negatives) | **0.0%** | 3.2% | 48.4% |
+| micro-precision | 99% | 87% | 47% |
+| micro-recall | 97% | 26% | 42% |
+| macro-F1 | 0.871 | 0.318 | 0.190 |
+
+Per-label recall (a few highlights): `private_person` 0.99 ·
+`private_company` 1.00 · `private_repo` 1.00 · `private_url` 1.00 ·
+`secret` 0.99 · `private_email` 0.98 · `private_phone` 0.92 ·
+`private_address` 0.92.
+
+### Latency (rfdetr_v8, 320×320 input, FP32)
+
+| platform                      | EP        | p50       |
+|-------------------------------|-----------|----------:|
+| macOS Apple Silicon (M-series) | CoreML    | **66 ms** ([real-screen sample](https://github.com/screenpipe/screenpipe-pii-bench-image)) |
+| macOS Apple Silicon (M-series) | CPU       | 163 ms |
+| Windows + DX12 GPU            | DirectML  | ~30-60 ms (estimated) |
+| Linux + NVIDIA                | CUDA      | ~10-20 ms (estimated) |
+| Linux/Windows CPU-only        | CPU       | ~140 ms |
+
+Same `.onnx` everywhere — Execution Provider is selected at load time
+by the user's ONNX Runtime build. **No CUDA / Vulkan / GPU vendor SDKs
+required at the consumer.**
+
+## Why this exists (vs Presidio Image Redactor and friends)
+
+The published baselines are trained on prose / generic-document
+imagery. A typical screenpipe frame looks nothing like that:
+
+- A Slack channel sidebar with 8 names, 12 channel mentions, 3 emails,
+  and 1 pasted AWS key — all in 1440×900 px at 14 px font.
+- A 1Password vault entry with structured `[Username | Password |
+  Server | One-time password]` rows, half of which are masked dots.
+- A Cursor workspace open on `.env.production` with five secret-shaped
+  values stacked top-to-bottom.
+
+These images are **dense** (10-20 PII spans per frame), **structured**
+(rows / columns / aligned chrome), and **layout-cued** (a thing in the
+"Username" cell is a username regardless of its surface text). A
+generic NER-on-OCR pipeline misfires by over-redacting UI chrome
+(48% false-fire on negatives in our bench, vs. 0% for this model).
+
+If you're building an **agentic system that reads screen state** — a
+desktop-control agent, a memory layer for browsing, anything that
+streams screen captures into an LLM — this is the redactor designed
+for that pipe.
+
+## What it does
+
+Per-image **object detection**. Given a JPG or PNG, returns
+`[(bbox, label, score)]` where each detection is a region the model
+thinks is PII, classified into one of the 12 canonical categories
+shared with [`screenpipe/pii-redactor`](https://huggingface.co/screenpipe/pii-redactor):
+
+```
+private_person, private_email, private_phone, private_address,
+private_url, private_company, private_repo, private_handle,
+private_channel, private_id, private_date, secret
+```
+
+`secret` covers passwords, API keys, JWTs, DB connection strings,
+PRIVATE-KEY block markers, etc. — same coverage as the text model.
+
+## Inference
+
+```python
+# pip install onnxruntime pillow numpy
+import numpy as np
+import onnxruntime as ort
+from PIL import Image
+
+CLASSES = [
+    "private_person",   "private_email",   "private_phone",
+    "private_address",  "private_url",     "private_company",
+    "private_repo",     "private_handle",  "private_channel",
+    "private_id",       "private_date",    "secret",
+]
+INPUT_SIZE = 320  # rfdetr_v8 was exported at 320x320
+THRESHOLD  = 0.30
+
+sess = ort.InferenceSession(
+    "rfdetr_v8.onnx",
+    providers=["CoreMLExecutionProvider", "CPUExecutionProvider"],
+)
+
+img = Image.open("screenshot.png").convert("RGB")
+W, H = img.size
+resized = img.resize((INPUT_SIZE, INPUT_SIZE), Image.BILINEAR)
+arr = np.asarray(resized, dtype=np.float32) / 255.0
+arr = (arr - [0.485, 0.456, 0.406]) / [0.229, 0.224, 0.225]
+arr = arr.transpose(2, 0, 1)[None].astype(np.float32)  # NCHW
+
+boxes, logits = sess.run(None, {sess.get_inputs()[0].name: arr})
+boxes  = boxes[0]    # (300, 4) cx, cy, w, h normalized
+logits = logits[0]   # (300, 13) — last channel is "no-object"
+
+probs = 1.0 / (1.0 + np.exp(-logits[:, :12]))   # per-class sigmoid
+best_class = probs.argmax(axis=1)
+best_score = probs[np.arange(300), best_class]
+keep = best_score >= THRESHOLD
+
+for q in np.where(keep)[0]:
+    cx, cy, bw, bh = boxes[q]
+    x1 = (cx - bw / 2) * W
+    y1 = (cy - bh / 2) * H
+    print(f"  {CLASSES[best_class[q]]:18} score={best_score[q]:.2f} "
+          f"bbox=[{int(x1)}, {int(y1)}, {int(bw*W)}, {int(bh*H)}]")
+```
+
+Full example with image overlay → `examples/inference.py`.
+
+For Rust integration via the `ort` crate, see the
+[`rust_smoke/`](https://github.com/screenpipe/screenpipe-pii-bench-image/tree/main/rust_smoke)
+prototype and the production wiring in PR
+[`screenpipe/screenpipe#3188`](https://github.com/screenpipe/screenpipe/pull/3188).
+
+## Redacting the image (vs. just detecting)
+
+This model **detects**. To actually remove the PII, draw a solid
+rectangle over each detected bbox. Solid black, **not blur** — blur
+is reversible by super-resolution attacks; opaque rectangles aren't.
+
+```python
+from PIL import ImageDraw
+draw = ImageDraw.Draw(img)
+for det in detections:        # from the snippet above
+    x, y, w, h = det.bbox
+    draw.rectangle([x, y, x + w, y + h], fill=(0, 0, 0))
+img.save("screenshot_redacted.png")
+```
+
+That's the entire redactor wrapper. ~5 lines.
+
+## Architecture
+
+- Base: [RF-DETR-Nano](https://github.com/roboflow/rf-detr) (Roboflow,
+  ICLR 2026) — DINOv2-backbone real-time detection transformer, ~25 M
+  params, claims first real-time model to break 60 mAP on COCO.
+- Fine-tuned at 320×320 input on a 2,833-image synthetic + WebPII
+  union (synthetic via DOM-truth bbox extraction; WebPII via the
+  [arxiv 2603.17357 release](https://arxiv.org/abs/2603.17357)).
+- Output head: 300 detection queries × 13 channels (12 PII classes +
+  no-object). Per-class sigmoid (NOT softmax — RF-DETR uses
+  independent classification per query).
+- Trained on a single A100 80 GB; ~100 minutes wall-clock for the
+  best-EMA epoch.
+
+## What was the training data
+
+| source | size | labels | notes |
+|---|---:|---|---|
+| **synthetic bench** | 2,206 imgs | DOM-truth bboxes (pixel-perfect) | 9 templates rendered via headless Chromium with `data-span` attributes — labels come from the same DOM tree the browser laid out. |
+| **WebPII** | 500 imgs (balanced sample) | bbox-labeled by the original authors | March 2026 release, e-commerce screenshots. Class-imbalance capped at 2× our synthetic frequency. |
+| **cascade auto-labels** | 100 imgs | OCR + text-PII model alignment | Old screenshots from this project's own bench, weakly labeled. |
+
+**No real user data was used during fine-tuning.** Membership
+inference attacks recover no real-user content because no real-user
+content was in the training set. If you discover a failure mode on
+your real screens, the project's recipe is to add a new SYNTHETIC
+template that reproduces it — the screenshot becomes a bug report,
+never a training row.
+
+## Limitations
+
+1. **Hand-curated gold set is small** — bench `data/` has 5
+   manually-built cases. Larger-scale held-out evaluation depends on
+   the synthetic corpus, which is in-distribution by construction.
+2. **`private_handle` and `private_id` recall are 0%** in the
+   reference numbers because the val split has only 2 and 1 examples
+   respectively. Don't deploy without a domain-specific eval pass.
+3. **Synthetic-template ceiling.** 95.3% zero-leak is the bench's
+   stable ceiling at this corpus size. Gains beyond come from training
+   on more real-screen failure modes (tracked in the bench's backlog).
+4. **WebPII is e-commerce-heavy.** Adding the full WebPII split
+   actually *hurt* dev-app accuracy in our experiments (rfdetr_v4 at
+   90.5% zero-leak vs. v8's 95.3%). The 500-image balanced sample is
+   our best-of-both compromise.
+5. **CPU-only floors at ~140 ms p50.** INT8 quantization (planned)
+   gets that under 100 ms, but the FP32 release is what's on this
+   page today.
+6. **English-only.** Synthetic templates render Latin-script text;
+   the WebPII supplement is English. CJK / Arabic / Cyrillic not
+   evaluated — don't deploy without a locale-specific eval.
+7. **Adversarial robustness not tested.** A user who knows the
+   detector exists could craft layouts that confuse it (handwritten
+   PII, embedded-image PII, partial occlusion). Use this for
+   honest-user privacy, not as a security boundary.
+
+## Files
+
+```
+rfdetr_v8.onnx                108 MB · the model · sha256 below
+README.md                      this file
+LICENSE                        CC BY-NC 4.0
+NOTICE                         attribution to base model + datasets
+examples/
+  inference.py                 the snippet above, runnable
+```
+
+SHA-256 of `rfdetr_v8.onnx`:
+`431acc0f0beb22a39572b7a50af4fc446e799840fb71320dc124fbd79a121eb3`
+
+## Reproducing inference
+
+```bash
+git clone https://huggingface.co/screenpipe/pii-image-redactor
+cd pii-image-redactor
+git lfs pull
+pip install onnxruntime pillow numpy
+python examples/inference.py path/to/your_screenshot.png
+```
+
+Reproducing the eval scores requires the screenpipe-pii-bench-image
+benchmark, which is not redistributed (it's the training corpus).
+Contact **louis@screenpi.pe** for benchmark access or commercial
+licensing.
+
+## License
+
+[CC BY-NC 4.0](LICENSE) — non-commercial use only. The base model
+(RF-DETR) is Apache-2.0; obligations are preserved (see
+[`NOTICE`](NOTICE)).
+
+For commercial licensing (production deployment, redistribution
+rights, SaaS / API embedding, custom fine-tunes for your domain):
+**louis@screenpi.pe**.
+
+## Citation
+
+```bibtex
+@misc{screenpipe-pii-image-redactor-2026,
+  title  = {screenpipe-pii-image-redactor: a screen-PII detector for
+            accessibility-aware agents},
+  author = {{screenpipe}},
+  year   = {2026},
+  url    = {https://huggingface.co/screenpipe/pii-image-redactor}
+}
+```