ARCHITECTURE.md

# AirTrackLM: LLM4STP Adapted for ADS-B Air Track Prediction

## Complete Architecture & Implementation Plan

---

## 1. Executive Summary

We adapt the LLM4STP multi-feature fusion architecture (originally for maritime AIS ship trajectory prediction) to work with **ADS-B air track data**. The model uses a **decoder-only transformer** with four specialized embedding types — Prompt, Uncertainty, Geohash, and Temporal — fused together for **next-state prediction** pretraining. Once pretrained, the model is adaptable to downstream tasks like activity classification.

This design is grounded in published results from:
- **FTP-LLM** (arXiv:2501.17459) — LLaMA-3.1-8B for flight trajectory prediction
- **H3-CLM** (arXiv:2405.09596) — H3 geohash + causal LM for maritime trajectories
- **GeoFormer** (arXiv:2311.05092) — GPT-style geospatial tokenization
- **TrAISFormer** (arXiv:2109.03958) — Discrete tokenization of AIS features

---

## 2. System Architecture Overview

```
┌─────────────────────────────────────────────────────────────────────┐
│                        RAW ADS-B INPUT                              │
│              (timestamp, latitude, longitude, altitude)             │
└─────────────────────────┬───────────────────────────────────────────┘
                          │
                          ▼
┌─────────────────────────────────────────────────────────────────────┐
│                   FEATURE DERIVATION PIPELINE                       │
│                                                                     │
│   Raw:     lat, lon, alt                                           │
│   Derived: COG, SOG, ROT, altitude_rate                            │
│   Meta:    timestamp → (hour, day_of_week, month)                  │
│                                                                     │
│   Output per timestep:                                              │
│   state_t = [lat, lon, alt, COG, SOG, ROT, alt_rate]              │
└─────────────────────────┬───────────────────────────────────────────┘
                          │
                          ▼
┌─────────────────────────────────────────────────────────────────────┐
│                    TOKENIZATION / ENCODING                          │
│                                                                     │
│   ┌──────────────┐  ┌──────────────┐  ┌──────────────┐            │
│   │   Geohash     │  │  Continuous   │  │   Temporal   │            │
│   │  Tokenizer    │  │  Discretizer  │  │   Encoder    │            │
│   │              │  │              │  │              │            │
│   │ lat,lon,alt  │  │ COG,SOG,ROT  │  │ hour,dow,    │            │
│   │ → H3 cell + │  │ alt_rate     │  │ month        │            │
│   │   alt_band   │  │ → bin IDs    │  │ → time IDs   │            │
│   └──────┬───────┘  └──────┬───────┘  └──────┬───────┘            │
│          │                 │                 │                      │
│          ▼                 ▼                 ▼                      │
│   ┌──────────────┐  ┌──────────────┐  ┌──────────────┐            │
│   │  Geohash     │  │  Feature     │  │  Temporal    │            │
│   │  Embedding   │  │  Embeddings  │  │  Embedding   │            │
│   │  Table       │  │  Tables      │  │  Table       │            │
│   │  (d_model)   │  │  (d_model)   │  │  (d_model)   │            │
│   └──────┬───────┘  └──────┬───────┘  └──────┬───────┘            │
│          │                 │                 │                      │
└──────────┼─────────────────┼─────────────────┼──────────────────────┘
           │                 │                 │
           ▼                 ▼                 ▼
┌─────────────────────────────────────────────────────────────────────┐
│                    EMBEDDING FUSION LAYER                            │
│                                                                     │
│   ┌────────────┐ ┌────────────┐ ┌────────────┐ ┌──────────────┐   │
│   │  Geohash   │ │  Feature   │ │  Temporal  │ │ Uncertainty  │   │
│   │  Embed     │ │  Embed     │ │  Embed     │ │   Embed      │   │
│   │  (d_model) │ │  (d_model) │ │  (d_model) │ │  (d_model)   │   │
│   └─────┬──────┘ └─────┬──────┘ └─────┬──────┘ └──────┬───────┘   │
│         │              │              │               │            │
│         └──────────┬───┴──────┬───────┘               │            │
│                    │          │                        │            │
│                    ▼          ▼                        ▼            │
│              E_state = E_geo + E_feat + E_temp + E_uncert           │
│                              │                                      │
│                              ▼                                      │
│   ┌───────────────────────────────────────────┐                    │
│   │  Prompt Embedding (prepended prefix)      │                    │
│   │  [PROMPT_1, PROMPT_2, ..., PROMPT_k]      │                    │
│   └───────────────────┬───────────────────────┘                    │
│                       │                                             │
│                       ▼                                             │
│   Input: [PROMPT_TOKENS | STATE_1 | STATE_2 | ... | STATE_T]      │
│                       │                                             │
│                       ▼                                             │
│              Linear Projection → d_model                           │
│                       │                                             │
│                       ▼                                             │
│              + Positional Encoding (sinusoidal)                    │
│                                                                     │
└───────────────────────┬─────────────────────────────────────────────┘
                        │
                        ▼
┌─────────────────────────────────────────────────────────────────────┐
│              DECODER-ONLY TRANSFORMER BACKBONE                      │
│                                                                     │
│   ┌─────────────────────────────────────────────────────┐          │
│   │  Transformer Block ×N_layers                        │          │
│   │                                                     │          │
│   │  ┌─────────────────────────────────────────┐       │          │
│   │  │  Causal Multi-Head Self-Attention        │       │          │
│   │  │  (masked: each position attends only     │       │          │
│   │  │   to itself and earlier positions)        │       │          │
│   │  └──────────────────┬──────────────────────┘       │          │
│   │                     │                               │          │
│   │                     ▼                               │          │
│   │  ┌─────────────────────────────────────────┐       │          │
│   │  │  LayerNorm + Residual Connection         │       │          │
│   │  └──────────────────┬──────────────────────┘       │          │
│   │                     │                               │          │
│   │                     ▼                               │          │
│   │  ┌─────────────────────────────────────────┐       │          │
│   │  │  Feed-Forward Network                    │       │          │
│   │  │  (Linear → GELU → Linear)               │       │          │
│   │  │  d_model → 4*d_model → d_model          │       │          │
│   │  └──────────────────┬──────────────────────┘       │          │
│   │                     │                               │          │
│   │                     ▼                               │          │
│   │  ┌─────────────────────────────────────────┐       │          │
│   │  │  LayerNorm + Residual Connection         │       │          │
│   │  └─────────────────────────────────────────┘       │          │
│   │                                                     │          │
│   └─────────────────────────────────────────────┘      │          │
│                                                                     │
└───────────────────────┬─────────────────────────────────────────────┘
                        │
                        ▼
┌─────────────────────────────────────────────────────────────────────┐
│                      OUTPUT HEADS                                   │
│                                                                     │
│   ┌─────────────────────────────────────────────────────────┐      │
│   │  PRETRAINING: Next-State Prediction Head                │      │
│   │                                                         │      │
│   │  For each position t, predict state at t+1:             │      │
│   │                                                         │      │
│   │  h_t → Linear → softmax → P(geohash_token_{t+1})       │      │
│   │  h_t → Linear → softmax → P(COG_bin_{t+1})             │      │
│   │  h_t → Linear → softmax → P(SOG_bin_{t+1})             │      │
│   │  h_t → Linear → softmax → P(ROT_bin_{t+1})             │      │
│   │  h_t → Linear → softmax → P(alt_rate_bin_{t+1})        │      │
│   │  h_t → Linear → softmax → P(alt_band_{t+1})            │      │
│   │                                                         │      │
│   │  Loss = Σ CrossEntropy(predicted_feature, true_feature) │      │
│   └─────────────────────────────────────────────────────────┘      │
│                                                                     │
│   ┌─────────────────────────────────────────────────────────┐      │
│   │  DOWNSTREAM: Activity Classification Head               │      │
│   │  (attached after pretraining, frozen or fine-tuned)     │      │
│   │                                                         │      │
│   │  h_[BOS] or mean(h_1:T) → MLP → softmax → class label  │      │
│   └─────────────────────────────────────────────────────────┘      │
│                                                                     │
└─────────────────────────────────────────────────────────────────────┘
```

---

## 3. The Four Embedding Types (Detailed)

### 3.1 Geohash Embeddings — Spatial Position Encoding

**Purpose**: Encode the aircraft's 3D geographic position as a discrete token.

**Method**: We use **H3 hexagonal hierarchical spatial index** (Uber's H3) at resolution 5 (hex area ≈ 252 km², edge ≈ 9.85 km) for en-route flight, with an option to use resolution 7 (≈ 5.16 km², edge ≈ 1.22 km) for terminal areas. This follows the H3-CLM paper's approach but adapted for aviation's larger spatial scale.

**3D Extension**: Since aircraft operate in 3D, we combine the H3 cell with an **altitude band**:
```
Geohash Token = H3_cell_index × N_alt_bands + alt_band_index

Altitude bands (1000 ft increments):
  Band 0:     0 - 1,000 ft    (ground / taxi)
  Band 1:  1,000 - 2,000 ft   (initial climb / approach)
  ...
  Band 45: 44,000 - 45,000 ft (high cruise)

  N_alt_bands = 46
```

**Vocabulary size**: At H3 resolution 5, the number of unique cells covering typical airspace is ~100K-200K. With altitude bands: `~200K × 46 ≈ 9.2M` — too large for direct embedding.

**Solution — Factored Embedding**:
```
E_geohash = E_h3[h3_cell_id] + E_alt[alt_band_id]

E_h3:  learned embedding table, vocab = N_h3_cells (~200K or hashing trick to 50K)
E_alt: learned embedding table, vocab = 46

Both project to d_model dimensions.
```

The **hashing trick**: Map H3 cell indices through a hash function to a fixed vocabulary of ~50,000 buckets. This bounds memory while maintaining spatial discrimination.

**Why H3 over traditional geohash**: H3 hexagons have uniform area (no polar distortion), hierarchical nesting, and consistent neighbor relationships — critical for trajectory continuity.

### 3.2 Temporal Embeddings — When Is the Aircraft Flying?

**Purpose**: Encode temporal context — time of day affects traffic density, routes, and behavior.

**Method**: Additive composition of multiple temporal scales:
```
E_temporal = E_hour[hour_of_day] + E_dow[day_of_week] + E_month[month]

E_hour:  24 entries  (captures rush hour vs. night patterns)
E_dow:    7 entries  (weekday vs. weekend traffic)
E_month: 12 entries  (seasonal routes, weather patterns)

All project to d_model dimensions.
```

**Optional — Sinusoidal Sub-minute Encoding**: For sub-minute resolution:
```
E_minute = sin(2π × minute / 60), cos(2π × minute / 60)  → linear → d_model
```

### 3.3 Uncertainty Embeddings — How Confident Are We?

**Purpose**: Encode the model's uncertainty about the current trajectory state. Aircraft in straight-and-level cruise have low uncertainty; aircraft maneuvering near airports have high uncertainty.

**Method**: Compute a **trajectory smoothness score** from recent states, then discretize:

```
Uncertainty sources (sliding window of k=5 recent states):

1. Position variance:  σ²_pos = var(Δlat) + var(Δlon)
2. Heading variance:   σ²_COG = circular_var(COG_{t-k:t})
3. Speed variance:     σ²_SOG = var(SOG_{t-k:t})
4. Altitude variance:  σ²_alt = var(alt_rate_{t-k:t})

Combined uncertainty score:
  U_t = w1·σ²_pos + w2·σ²_COG + w3·σ²_SOG + w4·σ²_alt

Discretize into N_uncert = 16 bins (quantile binning on training data)

E_uncertainty = E_uncert_table[bin(U_t)]   →  d_model
```

**Weights w1-w4**: Hyperparameters tuned on validation data, or learned as part of the model.

**During inference**: For multi-step prediction, uncertainty can be updated using MC-Dropout or ensemble disagreement.

### 3.4 Prompt Embeddings — Task and Context Metadata

**Purpose**: Provide metadata context about the flight, analogous to system prompts in LLMs. Enables task conditioning and multi-task learning.

**Method**: Learnable prompt tokens prepended to the trajectory:

```
Prompt token vocabulary:
  - Aircraft category:  [HEAVY, LARGE, SMALL, ROTORCRAFT, GLIDER, UAV, UNKNOWN]  (7)
  - Flight phase:       [CLIMB, CRUISE, DESCENT, APPROACH, GROUND, UNKNOWN]       (6)
  - Region:             [CONUS, EUROPE, ASIA, OTHER]                               (4)
  - Task:               [PREDICT, CLASSIFY, DETECT_ANOMALY]                        (3)
  - Special:            [BOS, EOS, PAD, MASK]                                      (4)

Total prompt vocab: ~24 tokens

Prompt sequence (prepended):
  [BOS, TASK_TOKEN, AIRCRAFT_TOKEN, PHASE_TOKEN, REGION_TOKEN]

Each has a learned embedding of dimension d_model.
```

**For downstream classification**: Change TASK_TOKEN to CLASSIFY; output at BOS position is used for classification.

---

## 4. Feature Derivation Pipeline

### 4.1 Raw Input
```
timestamp (Unix epoch seconds)
latitude  (degrees, WGS84)
longitude (degrees, WGS84)
altitude  (feet, barometric or geometric)
```

### 4.2 Derived Features

```python
import numpy as np

def derive_features(timestamps, lats, lons, alts):
    """
    Derive COG, SOG, ROT, and altitude rate from raw position data.
    All inputs: numpy arrays of shape (N,) for a single trajectory.
    Returns arrays of shape (N,) — first element is NaN.
    """
    dt = np.diff(timestamps)  # seconds
    dt = np.maximum(dt, 1e-6)  # avoid division by zero
    
    # --- Course Over Ground (COG) ---
    lat1, lat2 = np.radians(lats[:-1]), np.radians(lats[1:])
    dlon = np.radians(np.diff(lons))
    
    x = np.sin(dlon) * np.cos(lat2)
    y = np.cos(lat1) * np.sin(lat2) - np.sin(lat1) * np.cos(lat2) * np.cos(dlon)
    COG = np.degrees(np.arctan2(x, y)) % 360  # [0, 360)
    
    # --- Speed Over Ground (SOG) ---
    dlat = np.radians(np.diff(lats))
    a = np.sin(dlat/2)**2 + np.cos(lat1) * np.cos(lat2) * np.sin(dlon/2)**2
    c = 2 * np.arctan2(np.sqrt(a), np.sqrt(1-a))
    distance_nm = 3440.065 * c  # Earth radius in nautical miles
    SOG = distance_nm / (dt / 3600)  # knots
    
    # --- Rate of Turn (ROT) ---
    dCOG = np.diff(COG)
    dCOG = (dCOG + 180) % 360 - 180  # normalize to [-180, 180]
    ROT = np.full(len(lats), np.nan)
    ROT[2:] = dCOG / dt[1:]  # degrees per second
    
    # --- Rate of Altitude Change ---
    dalt = np.diff(alts)  # feet
    alt_rate = dalt / (dt / 60)  # feet per minute
    
    # Pad first elements
    COG_full = np.concatenate([[np.nan], COG])
    SOG_full = np.concatenate([[np.nan], SOG])
    alt_rate_full = np.concatenate([[np.nan], alt_rate])
    
    return COG_full, SOG_full, ROT, alt_rate_full
```

### 4.3 Feature Discretization

| Feature       | Range             | Bin Width    | N_bins | Notes              |
|---------------|-------------------|--------------|--------|--------------------|
| COG           | [0, 360)          | 5°           | 72     | Circular           |
| SOG           | [0, 600] kts      | 5 knots      | 121    | Capped at ~Mach 1  |
| ROT           | [-6, 6] °/s       | 0.25 °/s     | 49     | Capped ±6°/s       |
| Altitude Rate | [-6000, 6000] fpm | 200 ft/min   | 61     | Capped ±6000 fpm   |

Outliers beyond caps clipped to boundary bin.

### 4.4 Trajectory Preprocessing Pipeline

```
1. Segment raw ADS-B by ICAO24 + temporal gaps > 15 min → individual flights
2. Resample to fixed Δt = 60 seconds (linear interp for position, circular for heading)
3. Derive features (COG, SOG, ROT, alt_rate)
4. Drop first 2 points per trajectory (NaN from derivation)
5. Filter: remove trajectories with < 20 points (< 20 minutes)
6. Compute H3 cell (res 5) + altitude band for each point
7. Discretize all continuous features into bins
8. Compute uncertainty scores (sliding window k=5)
9. Extract temporal features (hour, dow, month)
10. Construct prompt tokens from metadata (if available)
```

---

## 5. Model Hyperparameters

### 5.1 Model Dimensions

| Parameter        | Value  | Rationale                                          |
|------------------|--------|----------------------------------------------------|
| d_model          | 256    | H3-CLM found 256-1024 effective                    |
| n_heads          | 8      | head_dim = 32                                       |
| n_layers         | 8      | Moderate depth for ~10M param model                 |
| d_ff             | 1024   | 4× d_model (standard)                              |
| max_seq_len      | 128    | 128 states × 60s ≈ 2 hours of flight               |
| n_prompt_tokens  | 5      | [BOS, TASK, AIRCRAFT, PHASE, REGION]                |
| dropout          | 0.1    |                                                     |

**Total parameters**: ~8-12M (trainable on single GPU in hours)

### 5.2 Vocabulary Sizes

| Embedding        | Vocab  | Dim |
|------------------|--------|-----|
| H3 cells         | 50,000 | 256 |
| Altitude bands   | 46     | 256 |
| COG bins         | 72     | 256 |
| SOG bins         | 121    | 256 |
| ROT bins         | 49     | 256 |
| Alt rate bins    | 61     | 256 |
| Hour of day      | 24     | 256 |
| Day of week      | 7      | 256 |
| Month            | 12     | 256 |
| Uncertainty bins | 16     | 256 |
| Prompt tokens    | 24     | 256 |

### 5.3 State Token Composition

Each timestep → single state token via additive fusion:

```
E_state_t = E_h3[h3_id_t] + E_alt_band[alt_band_t]            # Geohash (3D position)
          + E_COG[cog_bin_t] + E_SOG[sog_bin_t]                # Kinematics
          + E_ROT[rot_bin_t] + E_alt_rate[alt_rate_bin_t]       # Dynamics
          + E_hour[hour_t] + E_dow[dow_t] + E_month[month_t]   # Temporal
          + E_uncert[uncert_bin_t]                               # Uncertainty

E_state_t ∈ R^{d_model}
```

This additive fusion follows BERT (token + segment + position) and TrAISFormer.

---

## 6. Training Recipe

### 6.1 Pretraining: Next-State Prediction (Causal LM)

**Objective**: Given states 1..T, predict state at T+1 (applied autoregressively at every position).

**Loss**:
```
L = Σ_{t=1}^{T-1} [ λ_geo · CE(ŷ_geo_t, y_geo_{t+1})
                    + λ_COG · CE(ŷ_COG_t, y_COG_{t+1})
                    + λ_SOG · CE(ŷ_SOG_t, y_SOG_{t+1})
                    + λ_ROT · CE(ŷ_ROT_t, y_ROT_{t+1})
                    + λ_alt · CE(ŷ_alt_rate_t, y_alt_rate_{t+1})
                    + λ_altb · CE(ŷ_alt_band_t, y_alt_band_{t+1}) ]

λ values default to 1.0 (equal weighting).
```

**Training hyperparameters** (based on FTP-LLM + H3-CLM):

| Parameter            | Value               |
|----------------------|---------------------|
| Optimizer            | AdamW               |
| Learning rate        | 5e-4                |
| LR Schedule          | Cosine + 5% warmup  |
| Batch size (per GPU) | 64                  |
| Gradient accumulation| 4 (effective = 256) |
| Max epochs           | 30 (early stop p=5) |
| Weight decay         | 0.01                |
| Gradient clipping    | 1.0                 |
| Mixed precision      | bf16                |

**Data windowing**: Sliding window size=128, stride=64 (50% overlap).

### 6.2 Downstream: Activity Classification

After pretraining, attach classification head:
```
h_BOS → Linear(256, 128) → GELU → Dropout(0.1) → Linear(128, N_classes)
```

**Fine-tuning options**:
- **A**: Freeze backbone, train head only (fast, small data)
- **B**: Full fine-tune, backbone lr=1e-5, head lr=1e-3

---

## 7. Dataset Strategy

### 7.1 Prototyping — `traffic` Python Library

```python
from traffic.data.samples import landing_zurich_2019
# ~2,000 flights near Zurich
# Columns: timestamp, icao24, callsign, latitude, longitude, altitude,
#          groundspeed, track, vertical_rate, ...
```

Instant access, clean, well-documented. Single airport, limited diversity.

### 7.2 Training — OpenSky Network

```python
from pyopensky.trino import Trino
trino = Trino()
df = trino.rawquery("""
    SELECT time, icao24, lat, lon, baroaltitude, velocity, heading, vertrate
    FROM state_vectors_data4
    WHERE hour >= '2024-01-15 00:00:00'
      AND hour <  '2024-01-15 12:00:00'
      AND lat BETWEEN 40 AND 55
      AND lon BETWEEN -10 AND 20
    ORDER BY icao24, time
""")
```

**Target**:
- **Region A** (train): Europe, 1 month → ~500K-1M flights
- **Region B** (OOD test): US CONUS, 1 week → ~200K flights
- **Region C** (far test): East Asia, 1 week → ~100K flights

### 7.3 Alternative: SCAT Dataset

~170K en-route flights over Sweden, Zenodo. Pre-segmented, clean.

### 7.4 Data Split

```
Training:    70% of Region A flights
Validation:  15% of Region A flights  
Test (IID):  15% of Region A flights
Test (OOD):  100% of Region B flights
Test (Far):  100% of Region C flights
```

Split by **flight** (not time window) to avoid data leakage.

---

## 8. Ablation Study: Geohash Geographic Dependency

### 8.1 Hypothesis

> Geohash embeddings encode **absolute geographic position**, causing the model to memorize region-specific patterns (airways, approach paths, airspace structure). This improves in-distribution performance but degrades transfer to unseen regions.

### 8.2 Experimental Variants

| Variant | Geohash Type | Description |
|---------|-------------|-------------|
| **V1: Full Model** | H3 absolute | Complete architecture as described |
| **V2: No Geohash** | None | Remove geohash entirely; model sees only kinematics + temporal + uncertainty |
| **V3: Relative Geohash** | H3 relative | H3 cell of (Δlat, Δlon) from trajectory start — position-invariant |
| **V4: Multi-Resolution** | H3 res 3+5+7 | 3 resolutions summed (coarse→fine) |
| **V5: Continuous Position** | Linear projection | `Linear([lat, lon, alt] → d_model)` — no discretization |

### 8.3 Evaluation Metrics

For each variant × each test set (IID, OOD, Far):

| Metric | Description |
|--------|-------------|
| Geo Accuracy | % correct H3 cell prediction |
| Position MAE | Mean absolute error in km |
| COG MAE | Heading error in degrees |
| SOG MAE | Speed error in knots |
| Multi-step ADE | Average displacement error over 5 predicted steps |
| Multi-step FDE | Final displacement error at step 5 |

### 8.4 Key Comparisons

| Comparison | Tests |
|-----------|-------|
| V1 vs V2 (IID) | How much geohash helps when test = train region |
| V1 vs V2 (OOD) | If V2 > V1 on OOD → geohash causes geographic overfitting |
| V1 vs V3 (OOD) | If V3 good on both IID and OOD → relative geohash is the sweet spot |
| V4 (all) | Multi-resolution: coarse cells transfer, fine cells specialize? |
| V5 (all) | Does continuous encoding avoid discretization issues? |

### 8.5 Expected Outcomes

- **V1**: Best IID, worst OOD (hypothesis)
- **V3**: Best compromise — predicted winner
- **V5**: May struggle (loses discrete token structure transformers excel at)
- **V2**: Strong OOD baseline, sacrifices IID

### 8.6 Additional Analysis

- **Attention visualization**: V1 vs V3 attention patterns
- **Embedding clustering**: t-SNE of geohash embeddings colored by region
- **Learning curves**: IID vs OOD performance vs training data size

---

## 9. Implementation Phases

### Phase 1: Data Pipeline (Week 1)
- Set up `traffic` library, extract sample trajectories
- Implement feature derivation (COG, SOG, ROT, alt_rate)
- Implement H3 geohash encoding + altitude banding
- Implement feature discretization (binning)
- Implement uncertainty score computation
- Build PyTorch Dataset class with sliding window
- Unit tests for all derivation functions

### Phase 2: Model Architecture (Week 1-2)
- Implement all embedding tables
- Implement additive fusion layer
- Implement prompt token prepending
- Implement decoder-only transformer backbone
- Implement multi-head output (6 prediction heads)
- Implement classification head (for downstream)
- Forward pass test with dummy data

### Phase 3: Pretraining (Week 2-3)
- Implement training loop with multi-task loss
- Prototyping run on `traffic` data (small, fast iteration)
- Scale to OpenSky data
- Monitor loss curves, validate convergence
- Save best checkpoint

### Phase 4: Downstream Adaptation (Week 3-4)
- Implement classification fine-tuning pipeline
- Test on activity classification task
- Compare frozen vs. fine-tuned backbone

### Phase 5: Ablation Study (Week 4-5)
- Implement all 5 geohash variants
- Train each variant with identical hyperparameters
- Evaluate on IID, OOD, and Far test sets
- Generate comparison tables and visualizations
- Write analysis of geographic dependency findings

---

## 10. Key Design Decisions & Rationale

| Decision | Choice | Why |
|----------|--------|-----|
| Custom model vs. pretrained LLM | Custom ~10M param transformer | FTP-LLM showed text-tokenized LLMs work, but custom allows proper multi-feature fusion. 10M params trains in hours. |
| H3 vs. traditional geohash | H3 | Uniform hexagonal cells, no polar distortion, hierarchical. Proven by H3-CLM. |
| Additive vs. concatenative fusion | Additive | BERT/TrAISFormer paradigm. Keeps d_model constant. Concatenation → d_model × N_features = massive. |
| 60s time resolution | 60 seconds | FTP-LLM validated 1-min aggregation. 128 steps ≈ 2+ hours. |
| Factored geohash (H3 + alt) | Separate tables, summed | Avoids combinatorial explosion (9.2M → 50K + 46). |
| Multi-head output | Separate softmax per feature | More interpretable, allows per-feature analysis. |
| Uncertainty from smoothness | Variance-based | Computable at data time, no inference overhead. |

---

## 11. Risk Analysis

| Risk | Likelihood | Impact | Mitigation |
|------|-----------|--------|------------|
| Geohash overfits to region | High | High | Ablation study; V3 (relative) is fallback |
| OpenSky access issues | Medium | High | Fallback: `traffic` samples + SCAT |
| 60s too coarse for terminal | Medium | Low | Separate terminal model at 10s |
| Model too small | Low | Medium | Scale: d_model→512, n_layers→16 (~40M) |
| Alt discretization too coarse | Low | Low | Refine to 500ft bands (92) |

---

## 12. Monitoring & Evaluation

**During training** (Trackio):
- Total loss + per-feature loss curves
- Validation loss each epoch
- LR schedule, GPU utilization

**After training**:
- Next-state accuracy (top-1, top-5 per feature)
- Position error in km
- Multi-step prediction (1, 5, 10, 20 steps ahead)
- Downstream classification F1/precision/recall

---

*Grounded in: FTP-LLM, H3-CLM, GeoFormer, TrAISFormer, and LLM4STP (reconstructed). Ready for implementation upon approval.*