Step 10: audit and fallback spec — documenting already-implemented modules
Browse filesAdds specs/10_audit_and_fallback.md. Both modules were implemented in Step 6
to unblock the criteria extractor. Spec records the SQLite schema, action
vocabulary, filter API, and fallback safety net behavior.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
specs/10_audit_and_fallback.md
ADDED
|
@@ -0,0 +1,83 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Spec 10 — Audit and Fallback
|
| 2 |
+
|
| 3 |
+
**Step:** 10 of 15
|
| 4 |
+
**Time budget:** ~20 min
|
| 5 |
+
|
| 6 |
+
---
|
| 7 |
+
|
| 8 |
+
## Goal
|
| 9 |
+
|
| 10 |
+
Document and finalize `core/audit.py` and `core/fallback.py`. Both were implemented early (Step 6) to unblock the criteria extractor. This spec records their contracts.
|
| 11 |
+
|
| 12 |
+
---
|
| 13 |
+
|
| 14 |
+
## `core/audit.py`
|
| 15 |
+
|
| 16 |
+
### SQLite schema
|
| 17 |
+
|
| 18 |
+
```sql
|
| 19 |
+
CREATE TABLE IF NOT EXISTS audit_log (
|
| 20 |
+
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
| 21 |
+
ts TEXT NOT NULL,
|
| 22 |
+
action TEXT NOT NULL,
|
| 23 |
+
actor TEXT NOT NULL,
|
| 24 |
+
model_version TEXT,
|
| 25 |
+
bidder_id TEXT,
|
| 26 |
+
criterion_id TEXT,
|
| 27 |
+
payload_json TEXT
|
| 28 |
+
);
|
| 29 |
+
```
|
| 30 |
+
|
| 31 |
+
Single file: `AUDIT_DB = str(BASE_DIR / "audit.db")`.
|
| 32 |
+
|
| 33 |
+
### `log(action: str, actor: str = "system", **fields) -> int`
|
| 34 |
+
|
| 35 |
+
- Writes one row. Returns the inserted `rowid`.
|
| 36 |
+
- `ts`: UTC ISO timestamp.
|
| 37 |
+
- `model_version`: from `fields` if present, else `config.MODEL_VERSION`.
|
| 38 |
+
- `bidder_id`, `criterion_id`: extracted from `fields` if present.
|
| 39 |
+
- Remaining `fields` → `payload_json = json.dumps(fields)`.
|
| 40 |
+
|
| 41 |
+
### `query(filters: dict | None = None) -> list[dict]`
|
| 42 |
+
|
| 43 |
+
- Returns rows from `audit_log` ordered by `id DESC`.
|
| 44 |
+
- Supports filters: `bidder_id`, `action`, `date_from` (ts >=), `date_to` (ts <=).
|
| 45 |
+
|
| 46 |
+
### Action vocabulary
|
| 47 |
+
|
| 48 |
+
| Action | When logged |
|
| 49 |
+
|---|---|
|
| 50 |
+
| `criteria_extracted` | After successful LLM criteria extraction |
|
| 51 |
+
| `bidder_processed` | After each document is indexed |
|
| 52 |
+
| `criterion_evaluated` | After each (bidder, criterion) verdict |
|
| 53 |
+
| `human_review_action` | When evaluator approves/edits/rejects a verdict |
|
| 54 |
+
| `precomputed_fallback_used` | When LLM is unavailable and fallback fires |
|
| 55 |
+
| `vision_ocr_invoked` | When Tier 3 vision LLM is called |
|
| 56 |
+
|
| 57 |
+
---
|
| 58 |
+
|
| 59 |
+
## `core/fallback.py`
|
| 60 |
+
|
| 61 |
+
### `load_criteria() -> list[Criterion]`
|
| 62 |
+
|
| 63 |
+
- Reads `data/precomputed/criteria.json` if it exists, parses `{"criteria": [...]}`.
|
| 64 |
+
- Falls back to `_HARDCODED_CRITERIA` (5 hardcoded criteria matching the mock tender exactly) if file is missing.
|
| 65 |
+
|
| 66 |
+
### `load_evaluation(bidder_id: str, criterion_id: str) -> Verdict`
|
| 67 |
+
|
| 68 |
+
- Reads `data/precomputed/eval_{bidder_id}.json` if it exists.
|
| 69 |
+
- Finds the dict where `criterion_id` matches.
|
| 70 |
+
- Falls back to a `needs_review` Verdict with reason "Pre-computed evaluation not available."
|
| 71 |
+
|
| 72 |
+
### `_HARDCODED_CRITERIA`
|
| 73 |
+
|
| 74 |
+
Five criteria matching the mock tender (C1–C5), with correct rules and query_hints. These are the ultimate safety net if `precompute_results.py` has not been run.
|
| 75 |
+
|
| 76 |
+
---
|
| 77 |
+
|
| 78 |
+
## Acceptance Criteria
|
| 79 |
+
|
| 80 |
+
1. `audit.log("test")` inserts a row; `audit.query()` returns it.
|
| 81 |
+
2. `audit.query({"action": "criteria_extracted"})` filters correctly.
|
| 82 |
+
3. `fallback.load_criteria()` returns 5 criteria even with no precomputed file.
|
| 83 |
+
4. `fallback.load_evaluation("bidder_a", "C1")` returns a `Verdict` with `verdict_id` set.
|