--- license: apache-2.0 language: - en pipeline_tag: token-classification library_name: transformers tags: - pii - privacy - token-classification - bioes - moe - haremb base_model: - OpenMed/privacy-filter-nemotron - openai/privacy-filter datasets: - nvidia/Nemotron-PII --- # HarEmb · OpenMed-Nemotron PII > A **single-layer** HarEmb model on the [`OpenMed/privacy-filter-nemotron`](https://huggingface.co/OpenMed/privacy-filter-nemotron) lineage. It has 287M total parameters and predicts the full **221-class BIOES** Nemotron-PII label space. **Model**: [`fblgit/haremb-privacy-filter-opennemo`](https://huggingface.co/fblgit/haremb-privacy-filter-opennemo) ![HarEmb architecture](haremb.png) ## Lineage This model is the third leg of a three-step lineage: 1. **[`openai/privacy-filter`](https://huggingface.co/openai/privacy-filter)** — OpenAI's open release of the underlying 1.4B-parameter MoE backbone (8 transformer layers, ~50M active params/token, BIOES token classifier head). 2. **[`OpenMed/privacy-filter-nemotron`](https://huggingface.co/OpenMed/privacy-filter-nemotron)** — OpenMed's full fine-tune of that backbone on `nvidia/Nemotron-PII`, expanding the head to 221 BIOES classes (55 fine-grained PII categories). 3. **`haremb-privacy-filter-opennemo`** *(this model)* — a one-layer surgical slice of the OpenMed teacher. ## What this model does Token-level PII classification over **55 Nemotron-PII categories**. Every token receives one of `O` or `{B, I, E, S}-`, covering identity, contact, address, date/time, government ID, financial, healthcare, enterprise ID, vehicle, and digital identifier categories. In `eval()` mode the model can run constrained-BIOES Viterbi decoding internally, so `outputs.logits.argmax(-1)` is span-coherent by default. See [Output semantics](#output-semantics) for the exact fields and opt-out flags. ## Evaluation Evaluated on a 1% slice of `nvidia/Nemotron-PII:test` (1,000 documents, ctx 1024, seed 42), Viterbi-decoded. The benchmark and app both use the convention **A = `OpenMed/privacy-filter-nemotron` (teacher / baseline)**, **B = this checkpoint** (`haremb`); ratios are reported as **B ÷ A**. ### Quality (viterbi stream) | metric | **A: OpenMed teacher** | **B: haremb** (this) | B − A | |---|---:|---:|---:| | span F1 | 0.9434 | **0.9288** | −0.0146 | | span precision | 0.9531 | **0.9396** | −0.0135 | | span recall | 0.9338 | **0.9182** | −0.0156 | | token accuracy | 0.9900 | **0.9885** | −0.0015 | | non-O recall | 0.9703 | **0.9637** | −0.0066 | ### Performance (same eval set, ctx 1024, bf16, single GPU) | metric | **A: OpenMed teacher** | **B: haremb** | B vs A | |---|---:|---:|---:| | total params | 1,400M | **287M** | **4.87× smaller** | | dense params | 139M | 130M | 1.07× smaller | | MoE expert params | 1,260M | 158M | **7.97× smaller** | | **active params / token** (memory) | 178.7M | **134.5M** | 1.33× smaller | | **compute params / token** (FLOPs) | 50.7M | **6.5M** | **7.85× cheaper** | | GFLOP / token (forward) | 0.101 | **0.013** | **7.85× cheaper** | | weights on disk | (HF repo) | **548 MiB** | — | | weights in RAM | 2,669 MiB | 548 MiB | **4.87× smaller** | | peak GPU memory (eval) | 3.30 GiB | **1.22 GiB** | **2.70× less** | | throughput | 3,275 tok/s | **6,361 tok/s** | **1.94× faster** | `active params / token` estimates memory bandwidth pressure, while `compute params / token` estimates matmul FLOPs and excludes the embedding table row-gather. GFLOP/token is `2 × compute_params_per_token`. `infer.log` and `compare.log` contain the full breakdown, including peak GPU memory from `torch.cuda.max_memory_allocated`. ![Performance profile — absolute footprint and B/A ratio, A teacher vs B candidate](eval_performance.png) ### Quality breakdown ![Eval summary — headline metrics, raw-vs-viterbi span F1, and selected per-category deltas](eval_summary.png) ### Per-category highlights (viterbi span F1) **At or near 1.000 (B)** — `biometric_identifier`, `blood_type`, `coordinate`, `health_plan_beneficiary_number`, `ipv4`, `ipv6`, `license_plate`, `mac_address`, `national_id`, `postcode` (≥ 0.99 with ≥ 100 gold spans). **Categories where B beats A** — `gender` (0.987 vs 0.841), `political_view` (0.872 vs 0.839), `religious_belief` (0.935 vs 0.926), `state` (0.908 vs 0.829), `language` (0.897 vs 0.804), `race_ethnicity` (0.864 vs 0.861), `country` (0.952 vs 0.936). Several "fuzzy" world-knowledge categories where the 1-layer student carries the right inductive bias. **Categories where A leads** — `occupation` (0.727 vs 0.605), `company_name` (0.929 vs 0.776), `last_name` (0.976 vs 0.931), `first_name` (0.970 vs 0.930), `user_name` (0.961 vs 0.942). Identity-noun categories where the teacher's deeper-layer mixing helps. ### Token-outcome breakdown — A: OpenMed teacher vs B: haremb (viterbi) ![Pairwise token outcome and net category wins on gold non-O tokens](eval_confusion.png) ## Quick start ### Recommended — via OpenMed The OpenMed wrapper is the same UX the teacher card recommends and works on this checkpoint as a drop-in: ```bash pip install -U "openmed[hf]" ``` ```python from openmed import extract_pii, deidentify text = ( "Patient Sarah Johnson (DOB 03/15/1985), MRN 4872910, " "phone 415-555-0123, email sarah.johnson@example.com." ) result = extract_pii(text, model_name="fblgit/haremb-privacy-filter-opennemo") for ent in result.entities: print(f"{ent.label:30s} {ent.text!r} conf={ent.confidence:.2f}") masked = deidentify(text, method="mask", model_name="fblgit/haremb-privacy-filter-opennemo") fake = deidentify(text, method="replace", model_name="fblgit/haremb-privacy-filter-opennemo", consistent=True, seed=42) ``` ### HuggingFace `transformers` pipeline ```python from transformers import pipeline pipe = pipeline( "token-classification", model="fblgit/haremb-privacy-filter-opennemo", tokenizer="fblgit/haremb-privacy-filter-opennemo", trust_remote_code=True, aggregation_strategy="simple", ) pipe("Send the invoice to billing@acmecorp.io, account 1234-5678.") # → [{'entity_group': 'email', 'word': 'billing@acmecorp.io', ...}, # {'entity_group': 'account_number', 'word': '1234-5678', ...}] ``` ### Raw `transformers` API ```python import torch from transformers import AutoModelForTokenClassification, AutoTokenizer repo = "fblgit/haremb-privacy-filter-opennemo" model = AutoModelForTokenClassification.from_pretrained( repo, trust_remote_code=True, dtype=torch.bfloat16, ).to("cuda").eval() tok = AutoTokenizer.from_pretrained(repo) enc = tok("My email is foo@bar.com.", return_tensors="pt").to("cuda") with torch.no_grad(): out = model(**enc) # By default, `outputs.logits.argmax(-1)` follows the Viterbi-decoded path. labels = out.logits.argmax(-1)[0] ``` ## Output semantics The forward pass — in `eval()` mode — runs constrained-BIOES Viterbi over the per-token logits and attaches three things to the output: - `outputs.logits` — a tensor whose `argmax(-1)` equals the Viterbi prediction (so HF `pipeline()` and naive `argmax` consumers get span-coherent predictions automatically). - `outputs.predicted_labels` — a `[B, T]` LongTensor of Viterbi-decoded label ids (`-1` at padded positions). - `outputs.raw_logits` — the original per-token logits, preserved for callers that want raw confidences. To opt out: ```python model.config.viterbi_replace_logits = False # raw logits in outputs.logits model.config.use_viterbi_decode = False # also skip Viterbi entirely ``` The model supports the upstream context length (max position embeddings 131,072 tokens). Practical batch sizes depend on hardware; bf16 + batch 1 + full-length is comfortable on 24 GB. ## Limitations & intended use - **English-only training data.** Nemotron-PII is predominantly English. Performance on non-English text is not guaranteed. - **Synthetic training data.** Real clinical notes, legal documents, and live web text may show different surface forms. For high-stakes deployments, collect a domain-specific eval set and re-calibrate. - **Fuzzier categories** — `occupation`, `company_name`, and identity nouns (`first_name`, `last_name`, `user_name`) carry more uncertainty than formatted identifiers; downstream pipelines that only need strict PII can ignore low-confidence predictions on these. - **Not a substitute for legal compliance review.** Use alongside a governance layer (human review, deterministic regex pre-filters, etc.). ## Reproducibility Every metric, log, and plot in this card is regenerated by the single-file [`benchmark.py`](benchmark.py) shipped alongside the weights: ```bash python benchmark.py # full benchmark vs OpenMed teacher python benchmark.py --no-base # skip teacher download (logs only) python benchmark.py --no-plots # skip matplotlib (logs + JSON only) python benchmark.py --eval-pct 0.1 # smaller slice for a quick check ``` Outputs into the model folder: - `infer.log` - `compare.log` - `eval_summary.png` - `eval_confusion.png` - `eval_performance.png` Raw per-doc eval data is held in memory only. Pass `--out` to write artifacts somewhere else. The Gradio demo in [`app.py`](app.py) supports **side-by-side A-vs-B comparison** between any two token-classification checkpoints with the same label space. Defaults match the report convention: **A = OpenMed/privacy-filter-nemotron** (teacher / baseline), **B = this checkpoint**. Disable either model to run single-model inference; both expose a runtime "active experts per token" slider so you can sweep MoE routing density. From inside the model folder: ```bash python app.py # A=OpenMed teacher, B=. (this) python app.py --model-a /path/to/another/repo # swap baseline A python app.py --model-b /path/to/another/repo # swap candidate B python app.py --port 7860 --share # public share link ``` ## License Apache-2.0, same as the lineage. Subject to the license terms of [`openai/privacy-filter`](https://huggingface.co/openai/privacy-filter) and the dataset terms of [`nvidia/Nemotron-PII`](https://huggingface.co/datasets/nvidia/Nemotron-PII). ## Citation ```bibtex @misc{haremb-privacy-filter-opennemo, title = {HarEmb · OpenMed-Nemotron PII: a single-layer privacy-filter slice with span-coherent inference}, author = {fblgit}, year = {2026}, publisher = {Hugging Face}, url = {https://huggingface.co/fblgit/haremb-privacy-filter-opennemo}, howpublished = {\url{https://huggingface.co/fblgit/haremb-privacy-filter-opennemo}}, note = {Single-transformer-layer model on the openai/privacy-filter → OpenMed/privacy-filter-nemotron lineage; 287M total params, 221 BIOES classes (55 fine-grained PII categories), with inlined constrained-BIOES Viterbi decoding so outputs.logits.argmax(-1) is span-coherent.} } @misc{openmed-privacy-filter-nemotron, title = {OpenMed/privacy-filter-nemotron: fine-grained PII extraction with 55 categories}, author = {OpenMed}, year = {2026}, publisher = {Hugging Face}, url = {https://huggingface.co/OpenMed/privacy-filter-nemotron} } @misc{openai-privacy-filter, title = {Privacy Filter}, author = {OpenAI}, year = {2026}, publisher = {Hugging Face}, url = {https://huggingface.co/openai/privacy-filter} } @misc{nvidia-nemotron-pii, title = {Nemotron-PII}, author = {NVIDIA}, year = {2025}, publisher = {Hugging Face}, url = {https://huggingface.co/datasets/nvidia/Nemotron-PII} } ```