JaydeepR Claude Sonnet 4.6 commited on
Commit
8f34e2c
·
1 Parent(s): 27760c8

Step 10: audit and fallback spec — documenting already-implemented modules

Browse files

Adds specs/10_audit_and_fallback.md. Both modules were implemented in Step 6
to unblock the criteria extractor. Spec records the SQLite schema, action
vocabulary, filter API, and fallback safety net behavior.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Files changed (1) hide show
  1. specs/10_audit_and_fallback.md +83 -0
specs/10_audit_and_fallback.md ADDED
@@ -0,0 +1,83 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Spec 10 — Audit and Fallback
2
+
3
+ **Step:** 10 of 15
4
+ **Time budget:** ~20 min
5
+
6
+ ---
7
+
8
+ ## Goal
9
+
10
+ Document and finalize `core/audit.py` and `core/fallback.py`. Both were implemented early (Step 6) to unblock the criteria extractor. This spec records their contracts.
11
+
12
+ ---
13
+
14
+ ## `core/audit.py`
15
+
16
+ ### SQLite schema
17
+
18
+ ```sql
19
+ CREATE TABLE IF NOT EXISTS audit_log (
20
+ id INTEGER PRIMARY KEY AUTOINCREMENT,
21
+ ts TEXT NOT NULL,
22
+ action TEXT NOT NULL,
23
+ actor TEXT NOT NULL,
24
+ model_version TEXT,
25
+ bidder_id TEXT,
26
+ criterion_id TEXT,
27
+ payload_json TEXT
28
+ );
29
+ ```
30
+
31
+ Single file: `AUDIT_DB = str(BASE_DIR / "audit.db")`.
32
+
33
+ ### `log(action: str, actor: str = "system", **fields) -> int`
34
+
35
+ - Writes one row. Returns the inserted `rowid`.
36
+ - `ts`: UTC ISO timestamp.
37
+ - `model_version`: from `fields` if present, else `config.MODEL_VERSION`.
38
+ - `bidder_id`, `criterion_id`: extracted from `fields` if present.
39
+ - Remaining `fields` → `payload_json = json.dumps(fields)`.
40
+
41
+ ### `query(filters: dict | None = None) -> list[dict]`
42
+
43
+ - Returns rows from `audit_log` ordered by `id DESC`.
44
+ - Supports filters: `bidder_id`, `action`, `date_from` (ts >=), `date_to` (ts <=).
45
+
46
+ ### Action vocabulary
47
+
48
+ | Action | When logged |
49
+ |---|---|
50
+ | `criteria_extracted` | After successful LLM criteria extraction |
51
+ | `bidder_processed` | After each document is indexed |
52
+ | `criterion_evaluated` | After each (bidder, criterion) verdict |
53
+ | `human_review_action` | When evaluator approves/edits/rejects a verdict |
54
+ | `precomputed_fallback_used` | When LLM is unavailable and fallback fires |
55
+ | `vision_ocr_invoked` | When Tier 3 vision LLM is called |
56
+
57
+ ---
58
+
59
+ ## `core/fallback.py`
60
+
61
+ ### `load_criteria() -> list[Criterion]`
62
+
63
+ - Reads `data/precomputed/criteria.json` if it exists, parses `{"criteria": [...]}`.
64
+ - Falls back to `_HARDCODED_CRITERIA` (5 hardcoded criteria matching the mock tender exactly) if file is missing.
65
+
66
+ ### `load_evaluation(bidder_id: str, criterion_id: str) -> Verdict`
67
+
68
+ - Reads `data/precomputed/eval_{bidder_id}.json` if it exists.
69
+ - Finds the dict where `criterion_id` matches.
70
+ - Falls back to a `needs_review` Verdict with reason "Pre-computed evaluation not available."
71
+
72
+ ### `_HARDCODED_CRITERIA`
73
+
74
+ Five criteria matching the mock tender (C1–C5), with correct rules and query_hints. These are the ultimate safety net if `precompute_results.py` has not been run.
75
+
76
+ ---
77
+
78
+ ## Acceptance Criteria
79
+
80
+ 1. `audit.log("test")` inserts a row; `audit.query()` returns it.
81
+ 2. `audit.query({"action": "criteria_extracted"})` filters correctly.
82
+ 3. `fallback.load_criteria()` returns 5 criteria even with no precomputed file.
83
+ 4. `fallback.load_evaluation("bidder_a", "C1")` returns a `Verdict` with `verdict_id` set.