| # GrandgemMa — Gemma 4 Scam Detection Eval & Fine-Tune Kit |
|
|
| > **Goal:** Zero-shot test `google/gemma-4-E2B-it` (2B text) on real scam-call transcripts. |
| > If accuracy < 90 % or F1(SCAM) < 85 % → fine-tune with Unsloth 4-bit LoRA. |
|
|
| ## Quick Links |
|
|
| | Artifact | Path | |
| |---|---| |
| | Zero-shot eval script | `eval_zero_shot.py` | |
| | Unsloth SFT trainer | `train_sft_unsloth.py` | |
| | Dataset formatter | `format_dataset.py` | |
| | Decision rubric | below | |
|
|
| ## Datasets Used |
|
|
| - **Primary:** [`BothBosu/scam-dialogue`](https://huggingface.co/datasets/BothBosu/scam-dialogue) — 800+ synthetic scam/legit call transcripts, labeled `dialogue` + `label` (1=SCAM, 0=LEGIT). |
| - **Secondary (optional):** [`BothBosu/Scammer-Conversation`](https://huggingface.co/datasets/BothBosu/Scammer-Conversation) — extra mixed conversations. |
|
|
| ## 1. Zero-Shot Evaluation |
|
|
| ```bash |
| # Quick smoke test (100 rows) |
| python eval_zero_shot.py --limit 100 |
| |
| # Full test split (~400 rows) |
| python eval_zero_shot.py --limit -1 |
| |
| # CPU-only fallback |
| python eval_zero_shot.py --device cpu --dtype fp32 --limit 20 |
| ``` |
|
|
| **Output:** `results_zero_shot.json` + console report with accuracy / precision / recall / F1 / confusion matrix. |
|
|
| ## 2. Decision Rubric |
|
|
| | Condition | Verdict | Action | |
| |---|---|---| |
| | Accuracy ≥ 90 % **and** F1(SCAM) ≥ 85 % | ✅ PASS | Base model is strong enough. Fine-tuning optional. | |
| | Accuracy 75–89 % **or** F1(SCAM) 70–84 % | ⚠️ MARGINAL | **Fine-tune recommended.** Expected uplift +5–15 pp. | |
| | Accuracy < 75 % **or** F1(SCAM) < 70 % | ❌ FAIL | **Fine-tune REQUIRED.** Run `train_sft_unsloth.py`. | |
|
|
| > **Why these thresholds?** For elder-scam defense, missing a scam (false negative) is catastrophic. High recall on SCAM class is mandatory. |
|
|
| ## 3. Fine-Tuning (Unsloth SFT) |
|
|
| ```bash |
| # Install deps |
| pip install unsloth transformers datasets trl peft accelerate |
| |
| # Train & push to HF Hub |
| python train_sft_unsloth.py \ |
| --output grandgemma-scam-sft \ |
| --push_to_hub s23deepak/grandgemma-scam-sft |
| ``` |
|
|
| **Hardware:** Kaggle T4×2 (free) or any single GPU ≥ 16 GB VRAM. |
| **Config:** 4-bit + LoRA r=16, 3 epochs, lr=2e-4, batch=2, grad_accum=4. |
| **Expected time:** ~3–5 min / epoch on T4×2. |
| |
| ## 4. Re-Eval After Fine-Tuning |
| |
| After training, run the same `eval_zero_shot.py` but point `--model` at your fine-tuned checkpoint: |
| |
| ```bash |
| python eval_zero_shot.py \ |
| --model s23deepak/grandgemma-scam-sft \ |
| --limit -1 |
| ``` |
| |
| Compare the delta in `accuracy`, `recall_scam`, and `f1_scam`. |
| |
| ## 5. Team Notes |
| |
| - This repo is **evaluation-only** — no app code. App code lives in your monorepo (`/android`, `/ios`, `/extensions`, `/portal`). |
| - Fine-tuned weights produced here should be quantized to `.litertlm` for on-device Android inference (Stream A) and converted for iOS/browser WebGPU (Stream B). |
| - Track all runs in a spreadsheet: run_id | model | dataset | accuracy | f1_scam | notes. |
| |
