chore: clean up dead code, stale comments, and misleading names

- Rename gpt_data -> metadata on STrack (tracker.py)
- Remove dead inject_metadata method and phantom METADATA_SYNC_KEYS
- Fix _sync_data to sync depth_rel (the key actually written)
- Remove dead _build_display_label and gpt_distance_m checks
- Remove no-op add_no_cache_header middleware
- Remove unused get_segmenter_detector import
- Cache index.html at module load instead of reading per request
- Extract _parse_queries helper to deduplicate query parsing
- Remove empty set_track_data calls in segmentation writer loop
- Remove sentence-transformers semantic matching from coco_classes
- Clean stale GPT/mission references from docstrings
- Update CLAUDE.md to reflect stripped-down architecture

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

CLAUDE.md

@@ -4,251 +4,105 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co
 
 ## Project Overview
 
-Simple video object detection system with three modes:
-- **Object Detection**: Detect custom objects using text queries (fully functional)
-- **Segmentation**: Mask overlays using SAM3
-- **Drone Detection**: (Coming Soon) Specialized UAV detection
-
-## Core Architecture
-
-### Simple Detection Flow
-
-```
-User → demo.html → POST /detect → inference.py → detector → processed video
-```
-
-1. User selects mode and uploads video via web interface
-2. Frontend sends video + mode + queries to `/detect` endpoint
-3. Backend runs detection inference with selected model
-4. Returns processed video with bounding boxes
-
-### Available Detectors
-
-The system includes 4 pre-trained object detection models:
-
-| Detector | Key | Type | Best For |
-|----------|-----|------|----------|
-| **OWLv2** | `owlv2_base` | Open-vocabulary | Custom text queries (default) |
-| **YOLOv8** | `hf_yolov8` | COCO classes | Fast real-time detection |
-| **DETR** | `detr_resnet50` | COCO classes | Transformer-based detection |
-| **Grounding DINO** | `grounding_dino` | Open-vocabulary | Text-grounded detection |
-
-All detectors implement the `ObjectDetector` interface in `models/detectors/base.py` with a single `predict()` method.
+Reusable video analysis base combining object detection, segmentation, depth estimation, and multi-object tracking. Deployed as a Hugging Face Space (Docker SDK). Designed for multi-GPU inference with async job processing and live MJPEG streaming.
 
 ## Development Commands
 
-### Setup
 ```bash
-python -m venv .venv
-source .venv/bin/activate  # or `.venv/bin/activate` on macOS/Linux
+# Setup
+python -m venv .venv && source .venv/bin/activate
 pip install -r requirements.txt
-```
 
-### Running the Server
-```bash
-# Development
+# Run dev server
 uvicorn app:app --host 0.0.0.0 --port 7860 --reload
 
-# Production (Docker)
-docker build -t object_detectors .
-docker run -p 7860:7860 object_detectors
-```
+# Docker (production / HF Spaces)
+docker build -t detection_base . && docker run -p 7860:7860 detection_base
 
-### Testing the API
-```bash
-# Test object detection
-curl -X POST http://localhost:7860/detect \
+# Test async detection
+curl -X POST http://localhost:7860/detect/async \
   -F "video=@sample.mp4" \
   -F "mode=object_detection" \
-  -F "queries=person,car,dog" \
-  -F "detector=owlv2_base" \
-  --output processed.mp4
-
-# Test placeholder modes (returns JSON)
-curl -X POST http://localhost:7860/detect \
-  -F "video=@sample.mp4" \
-  -F "mode=segmentation"
-```
-
-## Key Implementation Details
-
-### API Endpoint: `/detect`
-
-**Parameters:**
-- `video` (file): Video file to process
-- `mode` (string): Detection mode - `object_detection`, `segmentation`, or `drone_detection`
-- `queries` (string): Comma-separated object classes (for object_detection mode)
-- `detector` (string): Model key (default: `owlv2_base`)
-
-**Returns:**
-- For `object_detection`: MP4 video with bounding boxes
-- For `segmentation`: MP4 video with mask overlays
-- For `drone_detection`: JSON with `{"status": "coming_soon", "message": "..."}` 
-
-### Inference Pipeline
-
-The `run_inference()` function in `inference.py` follows these steps:
-
-1. **Extract Frames**: Decode video using OpenCV
-2. **Parse Queries**: Split comma-separated text into list (defaults to common objects if empty)
-3. **Select Detector**: Load detector by key (cached via `@lru_cache`)
-4. **Process Frames**: Run detection on each frame
-   - Call `detector.predict(frame, queries)`
-   - Draw green bounding boxes on detections
-5. **Write Video**: Encode processed frames back to MP4
-
-Default queries (if none provided): `["person", "car", "truck", "motorcycle", "bicycle", "bus", "train", "airplane"]`
-
-### Detector Loading
-
-Detectors are registered in `models/model_loader.py`:
-
-```python
-_REGISTRY: Dict[str, Callable[[], ObjectDetector]] = {
-    "owlv2_base": Owlv2Detector,
-    "hf_yolov8": HuggingFaceYoloV8Detector,
-    "detr_resnet50": DetrDetector,
-    "grounding_dino": GroundingDinoDetector,
-}
+  -F "queries=person,car" \
+  -F "detector=yolo11"
 ```
 
-Loaded via `load_detector(name)` which caches instances for performance.
+No test suite exists. Verify changes by running the server and testing through the UI at `http://localhost:7860`.
 
-### Detection Result Format
+## Architecture
 
-All detectors return a `DetectionResult` namedtuple:
-```python
-DetectionResult(
-    boxes: np.ndarray,        # Nx4 array [x1, y1, x2, y2]
-    scores: Sequence[float],  # Confidence scores
-    labels: Sequence[int],    # Class indices
-    label_names: Optional[Sequence[str]]  # Human-readable names
-)
-```
-
-## File Structure
+### Request Flow
 
 ```
-.
-├── app.py                    # FastAPI server with /detect endpoint
-├── inference.py              # Video processing and detection pipeline
-├── demo.html                 # Web interface with mode selector
-├── requirements.txt          # Python dependencies
-├── models/
-│   ├── model_loader.py      # Detector registry and loading
-│   └── detectors/
-│       ├── base.py          # ObjectDetector interface
-│       ├── owlv2.py         # OWLv2 implementation
-│       ├── yolov8.py        # YOLOv8 implementation
-│       ├── detr.py          # DETR implementation
-│       └── grounding_dino.py # Grounding DINO implementation
-├── utils/
-│   └── video.py             # Video encoding/decoding utilities
-└── coco_classes.py          # COCO dataset class definitions
+index.html → POST /detect/async → app.py
+  ├─ process_first_frame()           # Fast preview (~1-2s)
+  ├─ Return job_id + URLs immediately
+  └─ Background: process_video_async()
+      ├─ run_inference()                 # Detection mode
+      └─ run_grounded_sam2_tracking()    # Segmentation mode
 ```
 
-## Adding New Detectors
-
-To add a new detector:
-
-1. **Create detector class** in `models/detectors/`:
-   ```python
-   from .base import ObjectDetector, DetectionResult
+The async pipeline returns instantly with a `job_id`. The frontend polls `/detect/status/{job_id}` and streams live frames via `/detect/stream/{job_id}` (MJPEG).
 
-   class MyDetector(ObjectDetector):
-       name = "my_detector"
+### API Endpoints (app.py)
 
-       def predict(self, frame, queries):
-           # Your detection logic
-           return DetectionResult(boxes, scores, labels, label_names)
-   ```
+**Core:** `POST /detect` (sync), `POST /detect/async` (async with streaming)
+**Job management:** `GET /detect/status/{job_id}`, `DELETE /detect/job/{job_id}`, `GET /detect/video/{job_id}`, `GET /detect/stream/{job_id}`
+**Per-frame data:** `GET /detect/tracks/{job_id}/{frame_idx}`, `GET /detect/first-frame/{job_id}`, `GET /detect/first-frame-depth/{job_id}`, `GET /detect/depth-video/{job_id}`
+**Benchmarking:** `POST /benchmark`, `POST /benchmark/profile`, `POST /benchmark/analysis`, `GET /gpu-monitor`, `GET /benchmark/hardware`
 
-2. **Register in model_loader.py**:
-   ```python
-   _REGISTRY = {
-       ...
-       "my_detector": MyDetector,
-   }
-   ```
+### Model Registries
 
-3. **Update frontend** `demo.html` detector dropdown:
-   ```html
-   <option value="my_detector">My Detector</option>
-   ```
+All models use a registry + factory pattern with `@lru_cache` for singleton loading. Use `load_*_on_device(name, device)` for multi-GPU (no cache).
 
-## Adding New Detection Modes
+**Detectors** (`models/model_loader.py`):
+| Key | Model | Vocabulary |
+|-----|-------|-----------|
+| `yolo11` (default) | YOLO11m | COCO classes only |
+| `detr_resnet50` | DETR | COCO classes only |
+| `grounding_dino` | Grounding DINO | Open-vocabulary (arbitrary text) |
+| `drone_yolo` | Drone YOLO | Specialized UAV detection |
 
-To implement additional modes such as drone detection:
+All implement `ObjectDetector.predict(frame, queries)` → `DetectionResult(boxes, scores, labels, label_names)` from `models/detectors/base.py`.
 
-1. **Create specialized detector** (if needed):
-   - For segmentation: Extend `SegmentationResult` to include masks
-   - For drone detection: Create `DroneDetector` with specialized filtering
+**Segmenters** (`models/segmenters/model_loader.py`):
+- `GSAM2-S/B/L` — Grounded SAM2 (small/base/large) backed by grounding_dino
+- `YSAM2-S/B/L` — YOLO-SAM2 (small/base/large) backed by yolo11
 
-2. **Update `/detect` endpoint** in `app.py`:
-   ```python
-   if mode == "segmentation":
-       # Run segmentation inference
-       # Return video with masks rendered
-   ```
+**Depth** (`models/depth_estimators/model_loader.py`):
+- `depth` — DepthAnythingV2
 
-3. **Update frontend** to remove "disabled" class from mode card
+### Inference Pipeline (inference.py)
 
-4. **Update inference.py** if needed to handle new output types
+Three public entry points:
+- **`process_first_frame()`** — Extract + detect on frame 0 only. Returns processed frame + detections.
+- **`run_inference()`** — Full detection pipeline. Multi-GPU data parallelism with worker threads per GPU, reorder buffer for out-of-order completion, ByteTracker for object tracking, optional depth.
+- **`run_grounded_sam2_tracking()`** — SAM2 segmentation with temporal coherence. Uses `SharedFrameStore` (in-memory decoded frames, 12 GiB budget) or falls back to JPEG extraction. `step` parameter controls keyframe interval.
 
-## Common Patterns
-
-### Query Processing
-Queries are parsed from comma-separated strings:
-```python
-queries = [q.strip() for q in "person, car, dog".split(",") if q.strip()]
-# Result: ["person", "car", "dog"]
-```
-
-### Frame Processing Loop
-Standard pattern for processing video frames:
-```python
-processed_frames = []
-for idx, frame in enumerate(frames):
-    processed_frame, detections = infer_frame(frame, queries, detector_name)
-    processed_frames.append(processed_frame)
-```
-
-### Temporary File Management
-FastAPI's `BackgroundTasks` cleans up temp files after response:
-```python
-_schedule_cleanup(background_tasks, input_path)
-_schedule_cleanup(background_tasks, output_path)
-```
+### Async Job System (jobs/)
 
-## Performance Notes
+- **`jobs/models.py`** — `JobInfo` dataclass, `JobStatus` enum (PROCESSING/COMPLETED/FAILED/CANCELLED)
+- **`jobs/storage.py`** — Thread-safe in-memory storage at `/tmp/detection_jobs/{job_id}/`. Auto-cleanup every 10 minutes.
+- **`jobs/background.py`** — `process_video_async()` dispatches to the correct inference function, updates job status.
+- **`jobs/streaming.py`** — Event-driven MJPEG frame publishing. Non-blocking (drops if consumer is slow). Frames pre-resized to 640px width.
 
-- **Detector Caching**: Models are loaded once and cached via `@lru_cache`
-- **Default Resolution**: Videos processed at original resolution
-- **Frame Limit**: Use `max_frames` parameter in `run_inference()` for testing
-- **Memory Usage**: Entire video is loaded into memory (frames list)
+### Concurrency Model
 
-## Troubleshooting
+- Per-model `RLock` for GPU serialization (`inference.py:_get_model_lock`)
+- Multi-GPU workers use separate model instances per device
+- `AsyncVideoReader` prefetches frames in a background thread to prevent GPU starvation
 
-### "No module named 'fastapi'"
-Install dependencies: `pip install -r requirements.txt`
+### Frontend (index.html)
 
-### "Video decoding failed"
-Check video codec compatibility. System expects MP4/H.264.
+Single HTML page with vanilla JS. Upload video, pick mode/model, view first frame, live MJPEG stream, download processed video, inspect detection JSON.
 
-### "Detector not found"
-Verify detector key exists in `model_loader._REGISTRY`
+## Adding a New Detector
 
-### Slow processing
-- Try faster detector: YOLOv8 (`hf_yolov8`)
-- Reduce video resolution before uploading
-- Use `max_frames` parameter for testing
+1. Create class in `models/detectors/` implementing `ObjectDetector` from `base.py`
+2. Register in `models/model_loader.py` `_REGISTRY`
+3. Add option to detector dropdown in `index.html`
 
-## Dependencies
+## Dual Remotes
 
-Core packages:
-- `fastapi` + `uvicorn`: Web server
-- `torch` + `transformers`: Deep learning models
-- `opencv-python-headless`: Video processing
-- `ultralytics`: YOLOv8 implementation
-- `huggingface-hub`: Model downloading
-- `pillow`, `scipy`, `accelerate`, `timm`: Supporting libraries
+- `hf` → Hugging Face Space (deployment)
+- `github` → GitHub (version control)

coco_classes.py

@@ -6,7 +6,6 @@ import logging
 import re
 from typing import Dict, Optional, Tuple
 
-import numpy as np
 
 logger = logging.getLogger(__name__)
 
@@ -157,69 +156,6 @@ _COCO_SYNONYMS: Dict[str, str] = {
 _ALIAS_LOOKUP: Dict[str, str] = {_normalize(alias): canonical for alias, canonical in _COCO_SYNONYMS.items()}
 
 
-# ---------------------------------------------------------------------------
-# Semantic similarity fallback (lazy-loaded)
-# ---------------------------------------------------------------------------
-
-_SEMANTIC_MODEL = None
-_COCO_EMBEDDINGS: Optional[np.ndarray] = None
-_SEMANTIC_THRESHOLD = 0.65  # Minimum cosine similarity to accept a match
-
-
-def _get_semantic_model():
-    """Lazy-load a lightweight sentence-transformer for semantic matching."""
-    global _SEMANTIC_MODEL, _COCO_EMBEDDINGS
-    if _SEMANTIC_MODEL is not None:
-        return _SEMANTIC_MODEL, _COCO_EMBEDDINGS
-
-    try:
-        from sentence_transformers import SentenceTransformer
-        _SEMANTIC_MODEL = SentenceTransformer("all-MiniLM-L6-v2")
-        # Prefix with "a photo of a" to anchor embeddings in visual/object space
-        coco_phrases = [f"a photo of a {cls}" for cls in COCO_CLASSES]
-        _COCO_EMBEDDINGS = _SEMANTIC_MODEL.encode(
-            coco_phrases, normalize_embeddings=True
-        )
-        logger.info("Loaded semantic similarity model for COCO class mapping")
-    except Exception:
-        logger.warning("sentence-transformers unavailable; semantic COCO mapping disabled", exc_info=True)
-        _SEMANTIC_MODEL = False  # Sentinel: tried and failed
-        _COCO_EMBEDDINGS = None
-
-    return _SEMANTIC_MODEL, _COCO_EMBEDDINGS
-
-
-def _semantic_coco_match(value: str) -> Optional[str]:
-    """Find the closest COCO class by embedding cosine similarity.
-
-    Returns the COCO class name if similarity >= threshold, else None.
-    """
-    model, coco_embs = _get_semantic_model()
-    if model is False or coco_embs is None:
-        return None
-
-    query_emb = model.encode(
-        [f"a photo of a {value}"], normalize_embeddings=True
-    )
-    similarities = query_emb @ coco_embs.T  # (1, 80)
-    best_idx = int(np.argmax(similarities))
-    best_score = float(similarities[0, best_idx])
-
-    if best_score >= _SEMANTIC_THRESHOLD:
-        matched = COCO_CLASSES[best_idx]
-        logger.info(
-            "Semantic COCO match: '%s' -> '%s' (score=%.3f)",
-            value, matched, best_score,
-        )
-        return matched
-
-    logger.debug(
-        "Semantic COCO match failed: '%s' best='%s' (score=%.3f < %.2f)",
-        value, COCO_CLASSES[best_idx], best_score, _SEMANTIC_THRESHOLD,
-    )
-    return None
-
-
 @functools.lru_cache(maxsize=512)
 def canonicalize_coco_name(value: str | None) -> str | None:
     """Map an arbitrary string to the closest COCO class name if possible.
@@ -230,7 +166,6 @@ def canonicalize_coco_name(value: str | None) -> str | None:
     3. Substring match (alias then canonical)
     4. Token-level match
     5. Fuzzy string match (difflib)
-    6. Semantic embedding similarity (sentence-transformers)
     """
 
     if not value:
@@ -261,5 +196,4 @@ def canonicalize_coco_name(value: str | None) -> str | None:
     if close:
         return _CANONICAL_LOOKUP[close[0]]
 
-    # Last resort: semantic embedding similarity
-    return _semantic_coco_match(value)
+    return None

models/detectors/detr.py

@@ -9,7 +9,7 @@ from models.detectors.base import DetectionResult, ObjectDetector
 
 
 class DetrDetector(ObjectDetector):
-    """Wrapper around facebook/detr-resnet-50 for mission-aligned detection."""
+    """Wrapper around facebook/detr-resnet-50 for object detection."""
 
     MODEL_NAME = "facebook/detr-resnet-50"
 

models/detectors/grounding_dino.py

@@ -9,7 +9,7 @@ from models.detectors.base import DetectionResult, ObjectDetector
 
 
 class GroundingDinoDetector(ObjectDetector):
-    """IDEA-Research Grounding DINO-B detector for open-vocabulary missions."""
+    """IDEA-Research Grounding DINO-B detector for open-vocabulary detection."""
 
     MODEL_NAME = "IDEA-Research/grounding-dino-base"
 

models/segmenters/model_loader.py

@@ -40,7 +40,7 @@ _REGISTRY: Dict[str, Callable[..., Segmenter]] = {
 
 
 def get_segmenter_detector(segmenter_name: str) -> str:
-    """Return the detector key associated with a segmenter (for mission parsing)."""
+    """Return the detector key associated with a segmenter."""
     spec = _SEGMENTER_SPECS.get(segmenter_name)
     if spec is None:
         available = ", ".join(sorted(_REGISTRY))

utils/profiler.py

@@ -390,7 +390,6 @@ def run_profiled_segmentation(
             queries,
             segmenter_name=segmenter_name,
             step=step,
-            enable_gpt=False,
             max_frames=max_frames,
             _perf_metrics=metrics,
             _perf_lock=lock,

utils/tracker.py

@@ -3,7 +3,6 @@ import numpy as np
 from scipy.optimize import linear_sum_assignment
 import scipy.linalg
 
-from utils.schemas import AssessmentStatus
 
 
 class KalmanFilter:
@@ -198,24 +197,6 @@ class KalmanFilter:
         return ret
 
 
-# Default staleness threshold: GPT metadata older than this many frames is flagged STALE
-MAX_STALE_FRAMES = 300
-
-GPT_SYNC_KEYS = frozenset({
-    # Legacy / polyfilled fields (consumed by frontend cards)
-    "gpt_distance_m", "gpt_direction", "gpt_description", "gpt_raw",
-    "threat_level_score", "distance_m", "direction", "description",
-    # Universal schema fields
-    "object_type", "size", "visible_weapons", "weapon_readiness",
-    "motion_status", "range_estimate", "bearing",
-    "threat_level", "threat_classification", "tactical_intent",
-    "dynamic_features",
-    # Provenance and temporal validity
-    "assessment_frame_index", "assessment_status",
-    # Mission relevance
-    "mission_relevant", "relevance_reason",
-})
-
 
 class STrack:
     """
@@ -247,8 +228,8 @@ class STrack:
         self.mean = None
         self.covariance = None
         
-        # GPT attributes (persistent)
-        self.gpt_data = {}
+        # Per-track metadata (persistent across frames)
+        self.metadata = {}
 
     def _tlwh_from_xyxy(self, xyxy):
         """Convert xyxy to tlwh."""
@@ -566,29 +547,18 @@ class ByteTracker:
             d_out['bbox'] = [float(x) for x in tracked_bbox]
             d_out['track_id'] = f"T{str(track.track_id).zfill(2)}"
 
-            # Restore GPT data if track has it and current detection didn't
-            for k, v in track.gpt_data.items():
+            # Restore metadata if track has it and current detection didn't
+            for k, v in track.metadata.items():
                 if k not in d_out:
                     d_out[k] = v
 
-            # --- Temporal validity check (INV-5, INV-11) ---
-            assessment_frame = d_out.get('assessment_frame_index')
-            if assessment_frame is not None:
-                frames_since = self.frame_id - assessment_frame
-                if frames_since > MAX_STALE_FRAMES:
-                    d_out['assessment_status'] = AssessmentStatus.STALE
-                    d_out['assessment_age_frames'] = frames_since
-            elif d_out.get('assessment_status') != AssessmentStatus.ASSESSED:
-                # INV-6: Unassessed objects get explicit UNASSESSED status
-                d_out['assessment_status'] = AssessmentStatus.UNASSESSED
-
             # Update history
-            if 'history' not in track.gpt_data:
-                track.gpt_data['history'] = []
-            track.gpt_data['history'].append(d_out['bbox'])
-            if len(track.gpt_data['history']) > 30:
-                track.gpt_data['history'].pop(0)
-            d_out['history'] = track.gpt_data['history']
+            if 'history' not in track.metadata:
+                track.metadata['history'] = []
+            track.metadata['history'].append(d_out['bbox'])
+            if len(track.metadata['history']) > 30:
+                track.metadata['history'].pop(0)
+            d_out['history'] = track.metadata['history']
 
             results.append(d_out)
 
@@ -606,42 +576,11 @@ class ByteTracker:
         return results
 
     def _sync_data(self, track, det_source):
-        """Propagate attributes like GPT data between track and detection."""
-        # 1. From Source to Track (Update)
+        """Propagate metadata (e.g. depth) between detection and track."""
         source_data = det_source.original_data if hasattr(det_source, 'original_data') else {}
-        for k in GPT_SYNC_KEYS:
+        for k in ("depth_rel",):
             if k in source_data:
-                track.gpt_data[k] = source_data[k]
-
-        # 2. From Track to Source (Forward fill logic handled in output construction)
-
-    def inject_metadata(self, tracked_dets):
-        """Push metadata from post-processed detection dicts back into internal STrack objects.
-
-        Needed because GPT results are added to detection dicts *after* tracker.update()
-        returns, so the tracker's internal state doesn't have GPT data unless we
-        explicitly push it back in.
-
-        Records assessment_frame_index for temporal validity tracking (INV-5).
-        """
-        meta_by_tid = {}
-        for d in tracked_dets:
-            tid = d.get('track_id')
-            if not tid:
-                continue
-            meta = {k: d[k] for k in GPT_SYNC_KEYS if k in d}
-            if meta:
-                # Ensure assessment_frame_index is recorded
-                if "assessment_frame_index" not in meta and any(
-                    k in meta for k in ("threat_level_score", "gpt_raw", "object_type")
-                ):
-                    meta["assessment_frame_index"] = self.frame_id
-                    meta["assessment_status"] = AssessmentStatus.ASSESSED
-                meta_by_tid[tid] = meta
-        for track in self.tracked_stracks:
-            tid_str = f"T{str(track.track_id).zfill(2)}"
-            if tid_str in meta_by_tid:
-                track.gpt_data.update(meta_by_tid[tid_str])
+                track.metadata[k] = source_data[k]
 
 
 # --- Helper Functions ---