| # Monday handoff (May 4, 2026) |
|
|
| State of the repo at end of Sunday May 3 / overnight into May 4. |
| Demo is **Sunday May 10**. |
|
|
| ## Overnight pass (Sunday evening → Monday) |
|
|
| Eight priorities closed against `audit/2026-05-03-evening-audit.md`: |
|
|
| 1. `pitch/cold_open.md` restored (was accidentally deleted in 1cb5ee6). |
| 2. Granite Guardian / refusal-classification leftovers removed. |
| Mellea is the sole grounding mechanism, period. |
| 3. **Trace UI is now clickable.** Click any specialist row to reveal |
| its raw structured output (formatted JSON, copy button, |
| max-height + scroll). This is the auditability contract: every |
| claim in the briefing is traceable to the specialist that produced |
| it directly inside the UI, not just the citation appendix. |
| 4. Buffered-footprint overlap for the three Point-geometry register |
| specialists. NYU Langone / Stuyvesant HS / P.S. 89 now correctly |
| register `inside_sandy_2012=true`. Each output records its |
| `footprint_buffer_m`. |
| 5. Map renders register-asset pins (subway / school / hospital / |
| NYCHA-centroid) coloured by Sandy exposure with click popups |
| showing name + `[doc_id]`. NYCHA polygon-fill is queued for when |
| `geometry_geojson` lands in the dataclass. |
| 6. **`floodnet_forecast` specialist**. TTM r2 forecast on the |
| nearest FloodNet sensor's flood-event recurrence. Reuses the |
| (512, 96) singleton already loaded for `ttm_311_forecast`. |
| *no new model class loaded into memory*. The strongest single |
| TTM win for the NYU CUSP audience. |
| 7. Trace UI groups TTM specialists under one parent node |
| `forecasting.granite-timeseries-ttm-r2 [N instances]` so the |
| "one foundation model, multiple data streams" architectural story |
| is legible without reading per-row metadata. |
| 8. `experiments/` cleanup: dropped two empty dirs (`05_sam2_promptable`, |
| `06_chronos_bolt_forecast`), renamed `05_terramind_finetune` → |
| `05a_terramind_finetune_micro` to dedupe with the active NYC |
| fine-tune dir, removed `Riprap.zip` from repo root. |
| |
| Commit chain: `a2143fc` … through `ed6ae9d`. Morning handoff doc |
| at `audit/2026-05-04-morning-handoff.md` summarises what to verify |
| and what's queued next. |
| |
| ## Where Sunday ended |
| |
| All four keep-list items resolved + 4 register specialists shipped + AMD |
| fine-tune prep green. |
| |
| | Item | Status | Path | |
| |---|---|---| |
| | Pitch cold-open locked | ✓ | `pitch/cold_open.md` | |
| | TerraMind-NYC fine-tune eval spec | ✓ | `experiments/05_terramind_nyc_finetune/eval/eval_spec.md` | |
| | 200-query adversarial set + refusal eval | ✓ (planner pivot) | `experiments/06_granite_guardian/` | |
| | Subway-entrance specialist (Sheepshead Bay) | ✓ | `experiments/07_mta_entrances/` | |
| | NYCHA-developments specialist (Red Hook) | ✓ | `experiments/08_nycha_developments/` | |
| | DOE-schools specialist (Coney Island) | ✓ | `experiments/09_doe_schools/` | |
| | DOH-hospitals specialist (Coney Island) | ✓ | `experiments/10_doh_hospitals/` | |
| | FSM integration of all 4 register specialists | ✓ | `app/registers/`, `app/fsm.py`, `app/reconcile.py`, `web/static/agent.js` | |
| | AMD droplet TerraMind smoke + STAC manifest | ✓ | `129.212.182.52:/root/terramind_nyc/` | |
| |
| End-to-end smoke on "Coney Island Brooklyn" produced citations |
| `[mta_entrance_56]`, `[nycha_dev_239]`, `[nycha_dev_166]` alongside |
| `[rag_mta]` and `[nyc311]`. Family-prefix chip routing works. |
| |
| Last commit: `86861be` (FSM integration of 4 register specialists). |
| |
| ## Decisions locked |
| |
| - **Refusal classification dropped entirely.** Planner-level |
| classifier hit FN=0% but FP=7% (gate was <5%). Granite Guardian |
| itself was already abandoned (laptop-infeasible). After the audit |
| surfaced that the planner shim was documented-but-never-wired, |
| the decision is now Option C: drop refusal handling. Cold-start |
| framing scopes the audience; Mellea rejection sampling enforces |
| grounding integrity; the four-tier glyph margin carries the |
| epistemic-honesty signal. The `GuardianRefusal.svelte` component |
| is deleted (was only ever rendered on a documentation page). |
| Demo's integrity beat is the **Mellea grounding-failure reroll on |
| the curated Hollis 0.19% → 19% case**. `experiments/06_granite_guardian/` |
| is preserved as a "considered and rejected" artifact for the |
| methodology paper. |
| - **AMD path: `129.212.182.52` is production**, not `165.245.134.44`. |
| CLAUDE.md says the latter; **fix CLAUDE.md to match reality**. |
| Production vLLM is on `.52`. The TerraMind container shares the |
| GPU with vLLM; both fit on one MI300X. |
| - **TerraMind manifest is 1028 paired chips**, 2021-05 → 2026-04, |
| NYC 5-borough hull +5 km, S2-cloud <30%, ≤3-day pair window. One |
| year (2022-05 → 2023-04) returned 0 due to PC API intermittency. |
| acceptable for the micro-fine-tune. |
|
|
| ## First thing Monday morning |
|
|
| 1. **Refresh Microsoft Planetary Computer signed URLs.** They have |
| ~1 hr TTL; the manifest from Sunday evening is stale by morning. |
| On the droplet: |
| ```bash |
| ssh root@129.212.182.52 |
| docker exec -it terramind bash |
| cd /root/terramind_nyc |
| python build_manifest.py --refresh-only manifest_train.jsonl |
| python build_manifest.py --refresh-only manifest_holdout.jsonl |
| ``` |
| (Recipe is in `/root/terramind_nyc/NOTES.md` on the droplet.) |
|
|
| 2. **Kick off TerraMind-NYC fine-tune.** Spec at |
| `experiments/05_terramind_nyc_finetune/eval/eval_spec.md`. Budget |
| is 30 GPU-hours; alarm at 25 (set on the droplet). Predicted |
| actual: ~0.16 GPU-hours at bs=8 / 3 epochs. Don't run anything |
| experimental until eval-spec gates pass on the held-out set. |
|
|
| 3. **Decide bucket** (A ship-in-demo / B publish-only / C revert): |
| - A: ship the fine-tuned checkpoint as a Riprap specialist. |
| - B: publish to HF as `msradam/TerraMind-1.0-NYC` with model card, |
| don't ship in demo. **Bucket B is fully acceptable** per the |
| spec. Civic-tech publication discipline is the durable goal. |
| - C: discard checkpoint, no public artefact. |
| |
| ## Working on Monday |
|
|
| - TerraMind-NYC fine-tune (above). |
| - **Mellea grounding-failure demo prep.** The pitch demo is the |
| Hollis 0.19% → 19% case where Granite emits a number with the |
| wrong order of magnitude and Mellea catches it. Demo script |
| needs to: |
| - Show the failed first attempt (banner: "Mellea reroll: numerics |
| grounding failed"). |
| - Show the second attempt with the corrected number. |
| - Show the audit panel with the pass/fail per-requirement. |
| - Show wall-clock for the reroll (target: under 30 s end-to-end). |
| - Currently reproducible via `scripts/probe_mellea.py --query |
| "Hollis" --runs 5`. The demo script is the *visual* version. |
| - **MTA Sandy-recovery citation layer.** Parse the MTA "Hurricane |
| Sandy: Three Years Later" report into per-station-id facts so |
| the subway-entrance specialist can emit |
| `[mta_recovery_<station_id>]` doc messages alongside the |
| exposure ones. |
| - **NYCHA polygon-fill on the map.** Overnight session shipped |
| NYCHA developments as centroid pins on the map (graded by |
| `pct_inside_sandy ≥ 50%`). The next tightening is to add a |
| `geometry_geojson` field to `app/registers/nycha.py`'s |
| `DevelopmentFinding` dataclass and route through SSE so |
| `register-polygons` actually renders graded fills (the layer + |
| source are already present in `RipMap.svelte`). |
| - **PLUTO/Building-Footprints join** for Stuyvesant Town etc. |
| Overnight pass shipped buffered-point overlap (NYU Langone, |
| Stuyvesant HS, P.S. 89 now correctly flip to |
| `inside_sandy_2012=true`). The 100m hospital buffer / 50m school |
| buffer is honest but coarse; PLUTO + actual building footprints |
| is the next step for the very-large-campus assets. |
| |
| ## Outstanding through Friday |
|
|
| In rough priority order: |
|
|
| 1. **More specialists**: |
| - FEMA OpenFEMA NFIP claims tract-aggregated (pending). |
| - NWS NWPS reach-level forecast + USGS NWIS Bronx / Saw Mill / |
| Hutchinson rivers. |
| - NYC DEP CSO outfalls + Bluebelt + Green Infrastructure |
| specialist (CSS-vs-MS4 distinction for ASCE). |
| - Three more TTM r2 specialists (USGS streamgage stage, NWS |
| rainfall accumulation, NYC 311 sewer-backup citywide rate). |
| **FloodNet forecast already shipped in the overnight pass.** |
| 2. **Visual identity refresh**: Carto Positron, IBM Plex, four-tier |
| epistemic palette, WeasyPrint PDF export, trace UI as `<details>` |
| tree. |
| 3. **WCAG 2.2 AA pass.** |
| 4. **Methodology paper draft** (6-8 page PDF). Goal: Saturday May 9. |
| 5. **Historical-event mode**. Vintage-cutoff queries. Saturday. |
| 6. **Five Build-in-Public posts** through the week. |
| 7. **5-minute hackathon pitch + 3 demo queries.** Friday rehearsal. |
| 8. **ASCE talk materials**. May 13 (post-hackathon). |
| |
| ## Sharp edges to remember |
|
|
| - **Static assets cache hard.** When iterating on Svelte or |
| agent.js, hard-reload (⌘⇧R). No cache-busting in place. |
| - **HF Space sleeps after idle.** Free tier; first request after |
| sleep is a 30-90 s cold start. Ping the space before any demo. |
| - **vLLM cold compile.** First few requests against a fresh |
| `vllm serve` log surprisingly low throughput while ROCm kernels |
| JIT. Run benchmarks 3+ times before believing them. |
| - **Sandy GeoJSON has self-intersection issues** that blow up |
| `unary_union`. Use `buffer(0)` (caught and fixed for NYCHA; |
| may surface again for any new polygon-overlap specialist). |
| - **DEP column is `Flooding_Category` (int16)**, not `depth_class`. |
| Documented in NYCHA RESULTS.md. |
| - **Centroid-edge join false-negatives** on NYU Langone / Stuyvesant |
| / P.S. 89 because their centroid points lie just outside the OEM |
| Sandy polygon despite real 2012 basement flooding. PLUTO |
| footprint join is the queued fix. |
| - **Don't restart uvicorn while a model is mid-generation.** Ollama |
| keeps the request alive but the FastAPI handler dies, leaving |
| the user staring at a dead stream. |
|
|
| ## Files to read in order on Monday morning |
|
|
| 1. This file. |
| 2. `experiments/05_terramind_nyc_finetune/eval/eval_spec.md`. The |
| contract for what training output triggers ship/publish/revert. |
| 3. `experiments/06_granite_guardian/RESULTS.md`. The Guardian → |
| planner pivot decision record (so you know why Guardian is in |
| the repo but not on the demo path). |
| 4. `experiments/07_mta_entrances/RESULTS.md`. The canonical |
| register-specialist pattern (the other three follow it). |
| 5. `CLAUDE.md`. Fix the AMD droplet IP (165.245.134.44 → |
| 129.212.182.52) at the same time as the first edit of the day. |
|
|
| ## Status as of 2026-05-03 ~12:50 ET |
|
|
| - Both git remotes (origin + huggingface) up-to-date through |
| `86861be`. |
| - HF Space rebuild was *not* triggered on the FSM-integration |
| commit; do `git push huggingface main` when you want to deploy. |
| (You may want to wait until Monday afternoon so a broken HF |
| rebuild doesn't eat morning time.) |
| - Local Ollama has both `granite4.1:3b` and `granite4.1:8b` warm. |
| - AMD droplet `129.212.182.52` has the `terramind` container |
| running with TerraTorch 1.2.7 + pystac-client + planetary- |
| computer installed in system Python; HF cache populated. |
| - 200-query adversarial set + planner-pivot eval results |
| reproducible from `experiments/06_granite_guardian/` in ~3 min. |
| - Mellea probe still works: `scripts/probe_mellea.py --query |
| "Hollis" --runs 5`. |
| |
