Implement stateful temporal dynamics, partial observability, and Human-in-the-Loop (HITL) review logic.

README.md

@@ -71,40 +71,56 @@ graph TD
 
 ---
 
-## 🌊 The Payment Lifecycle (with LLM Context)
+## 🌊 The Payment Lifecycle (The Reality Loop)
 
-The core interaction loop models an AI Agent acting as a **Smart Router and Risk Engine**.
+The environment models a high-frequency feedback loop where agents navigate noisy signals and delayed consequences.
 
 ```mermaid
 sequenceDiagram
     autonumber
-    participant LLM as LLM Agent (Decision Maker)
-    participant Env as Environment (Reality Layer)
-    participant CB as Chargeback Maturity Queue
+    participant Agent as AI Agent (LLM/RL)
+    participant Env as Reality Engine
+    participant Queue as Review/CB Queues
 
-    Env->>LLM: Observation: {BIN: 4111, Amount: $500, UserSegment: New, ...}
+    Note over Env: [State] Clock advances + Events Triggered
+    Env->>Agent: Observation (Noisy Risk + Lagged Health + Resolution Alerts)
     
-    Note over LLM: Agent analyzes fraud signals vs. BIN affinity
-    LLM->>Env: Action: {gateway: 1, fraud_decision: 2} (3DS Challenge)
-    
-    rect rgb(50, 50, 50)
-    Note over Env: Reality Simulation
-    Env->>Env: Apply 15% User Abandonment (Friction)
-    Env->>Env: Calculate Success (Gateway 1 Rate * BIN 4111 Affinity)
+    Note over Agent: [Inference] Is there a fraud spike or gateway outage?
+    Agent->>Env: Action (Gateway Strategy + Fraud Decision)
+
+    rect rgb(30, 30, 30)
+        Note over Env: [Reality] Execution & Scheduling
+        Env->>Env: Success = f(Health, BIN, TrueRisk, Noise)
+        Env->>Queue: Schedule Reviews (10s) and Chargebacks (40s)
     end
 
-    Env-->>LLM: Step Outcome: Reward, Done, chargeback_penalty=0
-    
-    Note over Env,CB: 30-50 Transactions Later...
-    CB->>Env: Fraud Detected from Step 1
-    Env-->>LLM: Next Observation: {chargeback_penalty_applied: $520.00}
+    Queue-->>Env: Matured Results from previous steps
+    Env->>Agent: Feedback (Reward, Done, Resolved Alerts)
 ```
 
 ---
 
+## 💎 Advanced Reality Features
+
+### 1. Log-Driven Time-Series
+Sequentially streams from synthetic logs to simulate real-world distributions, diurnal cycles (simulation clock), and persistent fraud surges.
+
+### 2. Partial Observability
+Forces agents to infer state by adding noise to risk signals, hiding internal user tiers, and lagging gateway health metrics by 2 steps.
+
+### 3. Human-in-the-Loop (HITL)
+Agents can send transactions to manual review (Action 3). Resolutions are 100% accurate but incur a $5.00 fee and a 10-25 step delay.
+
+### 4. Advanced Adversarial Mechanics
+- **🛡️ 3DS Friction (Action 2)**: Provides a **90% fraud reduction** but triggers a **15-25% abandonment rate**. Agents must balance security vs. customer drop-off.
+- **⏳ Delayed Chargebacks**: Undetected fraud ($TrueRisk > 0.65$) matures into penalties (Tx Amount + $20 fee) **30-50 steps later**, forcing long-term liability management.
+- **📊 BIN-Gateway Affinity**: A hidden matrix of gateway performance across different card types. Agents must discover these affinities to optimize routing success.
+
+---
+
 ## 🎯 Benchmark Tasks
 
-SmartPayEnv supports three core curriculum tasks, ranging from basic classification to complex joint optimization.
+SmartPayEnv supports four curriculum tasks, ranging from basic classification to complex joint optimization.
 
 | Task | Level | Objective | Metrics |
 |------|-------|-----------|---------|
@@ -124,69 +140,46 @@ Grades the quality of the gateway choice and transaction outcome.
 - **Formula**: $Reward = \sigma(\alpha \cdot (2E - 1) - (\beta \cdot Cost + \gamma \cdot Retries) + \delta \cdot Quality)$
 - **Key Parameters**:
     - **$\alpha$ (Outcome Weight: 1.2)**: Scales the impact of the expected success.
-    - **$\beta$ (Cost Multiplier: 0.15)**: Penalizes choosing expensive gateways (Fixed + % Fees).
-    - **$\gamma$ (Retry Penalty: 0.4)**: Discourages excessive retries which increase latency.
-    - **$\delta$ (Decision Bonus: 0.8)**: Rewards selecting the gateway with the highest current affinity/rate, even if the transaction fails due to environment noise.
-
-
+    - **$\beta$ (Cost Multiplier: 0.15)**: Penalizes choosing expensive gateways.
+    - **$\gamma$ (Retry Penalty: 0.4)**: Discourages excessive retries.
+    - **$\delta$ (Decision Bonus: 0.8)**: Rewards selecting the gateway with the highest current affinity.
 
 ### 2. Fraud Detection Grader (MCC)
-Uses the **Matthews Correlation Coefficient (MCC)** to handle imbalanced transaction data.
-- **Why?**: In payments, fraud is rare (~2%). Accuracy is a misleading metric; MCC captures the balance between True Positives (blocked fraud) and False Positives (blocked legitimate users).
-- **Normalization**: Maps MCC $[-1, 1]$ to a learnable range $[0, 1]$, where $0.5$ represents a random baseline.
+Uses the **Matthews Correlation Coefficient (MCC)** to handle imbalanced transaction data (fraud is rare, ~2%).
+- **MCC Formula**:
+$$MCC = \frac{TP \times TN - FP \times FN}{\sqrt{(TP + FP)(TP + FN)(TN + FP)(TN + FN)}}$$
+- **Reward Mapping**: Maps MCC $[-1, 1]$ to a learnable range $[0, 1]$ using $R = \frac{MCC + 1}{2}$. A baseline of $0.5$ represents a random classifier.
 
 ### 3. User Retention Grader
-Models customer churn using an **Exponential Hazard Function**.
-- **Mechanic**: Every failed transaction increments a `consecutive_failures` counter for the user.
-- **Hazard Formula**: $1 - e^{-\lambda \cdot (failures^2)}$
-- **Rationale**: Models the "Trust Deficit." A first failure is annoying; a third consecutive failure causes **non-linear churn**, reflecting how premium users abandon platforms after bad experiences.
+Models customer churn using an **Exponential Hazard Function** to simulate the "Trust Deficit."
+- **Retention Formula**:
+$$Retention = e^{-\lambda \cdot f^2}$$
+where $f$ is the count of consecutive failed transactions for that user cohort.
+- **Rationale**: Consecutive failures cause non-linear churn; a first failure is an annoyance, but a third consecutive failure leads to near-certain platform abandonment.
 
 ---
 
 ## 📐 Data Models
 
 ### Action Space (`SmartpayenvAction`)
-Decisions submitted by the agent at each step:
-
 | Field | Type | Values | Description |
 |-------|------|--------|-------------|
-| `gateway` | `int` | `0, 1, 2` | 0=GatewayA (Economy), 1=GatewayB (Standard), 2=GatewayC (Premium) |
-| `fraud_decision`| `int` | `0, 1, 2` | 0=Allow, 1=Block (Ends episode), 2=3DS Challenge (Friction) |
-| `retry_strategy`| `int` | `0, 1` | 0=No Retry, 1=Auto-Failover to next gateway on failure |
+| `gateway` | `int` | `0, 1, 2` | 0=Economy, 1=Standard, 2=Premium |
+| `fraud_decision`| `int` | `0, 1, 2, 3`| 0=Allow, 1=Block, 2=3DS (Challenge), 3=Manual Review |
+| `retry_strategy`| `int` | `0, 1` | 0=No Retry, 1=Auto-Failover |
 
 ### Observation Space (`SmartpayenvObservation`)
-The state provided to the agent for each transaction:
-
-| Category | Field | Values | Description |
-|----------|-------|--------|-------------|
-| **Context** | `amount` | `float` | Transaction value in USD ($1 - $5000) |
-| | `bin_category` | `0-9` | Card type (e.g., 0=Domestic Debit, 5=International Credit) |
-| | `user_segment` | `0, 1, 2` | 0=New, 1=Existing, 2=Premium (Lower fraud risk) |
-| **Signals** | `fraud_risk_score`| `0..1` | Multi-factor risk probability (higher = more suspicious) |
-| | `user_history_score`| `0..1` | Normalized reliability based on previous successful tx |
-| **Health** | `gateway_states` | `str[]` | Health status per gateway: `normal`, `degraded`, `recovering` |
-| | `gateway_success_rates`| `float[]`| Real-time estimated success probabilities for A, B, and C |
-| **Tracking**| `chargeback_penalty_applied`| `float` | Penalty deducted *this step* from a past undetected fraud |
-| | `previous_failures`| `int` | Consecutive failures in current cohort session (influences churn) |
-
----
-
-## 🛠️ Advanced Reality Features
-
-### 🛡️ 3D Secure (3DS) Friction
-The `fraud_decision=2` action triggers a 3DS challenge.
-- **Security**: Provides a **90% reduction** in fraud risk.
-- **Friction**: Triggers a **15% abandonment rate** (User Drop-off). Agents must learn when the transaction value justifies the risk of losing the customer.
-
-### ⏳ Delayed Chargebacks
-Undetected fraud ($FraudRisk > 0.65$) incurs a **Chargeback Penalty** that matures **30-50 steps** after the transaction.
-- **Impact**: Full transaction amount + $20 chargeback fee.
-- **Goal**: Forces agents to balance immediate routing success against long-term liability.
-
-### 📊 BIN-Gateway Affinity
-A 10x3 matrix mapping card types (BIN categories) to gateway strengths. 
-- Some gateways process "Debit" better, while others are "Premium Credit" specialists.
-- Agents must discover these hidden affinities to maximize success rates.
+| Category | Field | Description |
+|----------|-------|-------------|
+| **Context** | `amount` | Transaction value in USD |
+| | `bin_category` | Card type (0-9) |
+| | `user_segment` | 0=New, 1=Existing, 2=Premium |
+| **Signals** | `observed_fraud_risk`| Noisy risk probability [0,1] |
+| | `time_of_day` | Current simulation hour (0-23) |
+| **Reviews**| `review_resolutions`| List of matured manual review results |
+| **Health** | `gateway_states` | LAGGED Health status (2 steps delay) |
+| | `gateway_success_rates`| LAGGED success probabilities |
+| **Tracking**| `chargeback_penalty_applied`| Penalty from a past undetected fraud |
 
 ---
 
@@ -207,7 +200,7 @@ uv sync
 openenv validate
 
 # Run core logic tests
-python tests/test_v3_features.py
+python tests/test_reality_features.py
 ```
 
 ### 2. Starting the Server
@@ -231,13 +224,19 @@ docker run -p 7860:7860 smartpay-env
 ## 📁 Project Structure
 ```text
 SmartPayEnv/
+├── scripts/
+│   ├── generate_logs.py         # Synthetic dataset generator
+├── data/
+│   ├── transactions_log.jsonl   # Pre-generated transaction pool
 ├── server/
 │   ├── app.py                  # FastAPI Entry Point (Uvicorn)
 │   ├── SmartPayEnv_environment.py # Core Reality Layer Logic
-│   └── graders.py               # Math models for RL Reward
+│   ├── graders.py               # Math models for RL Reward
+│   └── utils.py                 # Log loading & sampling utilities
 ├── tests/
 │   ├── test_graders.py         # Unit tests for scoring math
-│   └── test_v3_features.py     # Reality layer verification 
+│   ├── test_reality_features.py # Reality layer verification
+│   └── test_env_logs.py        # Log-driven simulation test
 ├── models.py                   # Pydantic Action/Observation Schemas
 ├── inference.py                # LLM/RL Agent Driver & Curriculum
 ├── pyproject.toml              # Dependency & Build Manifest

inference.py

@@ -14,7 +14,7 @@ API_KEY = os.getenv("HF_TOKEN") or os.getenv("API_KEY", "dummy-token")
 API_BASE_URL = os.getenv("API_BASE_URL", "https://router.huggingface.co/v1")
 MODEL_NAME = os.getenv("MODEL_NAME", "meta-llama/Llama-3.3-70B-Instruct")
 
-MAX_STEPS = 20
+MAX_STEPS = 30
 SUCCESS_SCORE_THRESHOLD = 0.5
 ENV_URL = "http://localhost:7860"
 BENCHMARK = os.getenv("BENCHMARK", "SmartPayEnv")
@@ -45,14 +45,24 @@ SYSTEM_PROMPT = textwrap.dedent(
        - Hours 01:00-05:00: Severe Fraud Surge (Attack period).
        - Segment 0 (New): High distrust/abandonment during 3DS challenges.
     
+    4. Manual Review:
+       - Action 3: Sends tx to human team. 10-25 step delay.
+       - Cost: $5.00 fee. Highest accuracy but slow.
+    
     ### ACTION SCHEMA:
     Respond with EXACTLY ONE JSON object:
     {{
-        "thought": "Reasoning based on current BIN category vs Affinity Matrix and Risk Score",
+        "thought": "Reasoning based on current BIN category vs Affinity Matrix and Observed Risk",
         "gateway": 0|1|2,
         "retry_strategy": 0|1,
-        "fraud_decision": 0(Allow)|1(Block)|2(3DS Challenge)
+        "fraud_decision": 0(Allow)|1(Block)|2(3DS Challenge)|3(Manual Review)
     }}
+
+    ### IMPORTANT:
+    - Observations are PARTIAL. `observed_fraud_risk` is a noisy estimate.
+    - Gateway health signals are LAGGED by ~2 steps.
+    - `user_type` is hidden.
+    - Events (Spikes, Outages) are CORRELATED and have DURATION.
     """
 ).strip()
 

models.py

@@ -25,7 +25,7 @@ class SmartpayenvAction(Action):
     """
     gateway: int = Field(default=0, description="0=GatewayA (cheap), 1=GatewayB (balanced), 2=GatewayC (premium)")
     retry_strategy: int = Field(default=0, description="0=No Retry, 1=Failover to next gateway on failure")
-    fraud_decision: int = Field(default=0, description="0=Allow, 1=Block (end episode), 2=Challenge (3DS / MFA)")
+    fraud_decision: int = Field(default=0, description="0=Allow, 1=Block, 2=Challenge (3DS), 3=Manual Review (Delayed)")
 
 
 class SmartpayenvObservation(Observation):
@@ -70,9 +70,9 @@ class SmartpayenvObservation(Observation):
     )
 
     # ── Risk scores ───────────────────────────────────────────────────
-    fraud_risk_score: float = Field(
+    observed_fraud_risk: float = Field(
         default=0.0,
-        description="Continuous multi-factor fraud risk [0,1] (higher = more suspicious)"
+        description="Noisy multi-factor fraud risk estimate [0,1] (true risk is hidden)"
     )
 
     # ── Episode tracking ──────────────────────────────────────────────
@@ -83,6 +83,7 @@ class SmartpayenvObservation(Observation):
     reward: float = Field(default=0.0, description="Combined step reward [0,1]")
     done: bool = Field(default=False, description="Episode done flag")
     chargeback_penalty_applied: float = Field(default=0.0, description="Penalty deducted this step from a past transaction chargeback")
+    review_resolutions: list[dict] = Field(default_factory=list, description="List of resolved manual reviews this step: [{ 'amount': float, 'is_fraud': bool, 'outcome': 'accepted'|'rejected' }]")
 
     # Per-task scores — declared as first-class fields so openenv framework serializes them
     task_routing_score: float = Field(default=0.0, description="Routing efficacy score [0,1]")

scripts/generate_logs.py

@@ -0,0 +1,68 @@
+import json
+import numpy as np
+import os
+from uuid import uuid4
+
+def generate_logs(output_path="data/transactions_log.jsonl", num_transactions=5000):
+    rng = np.random.default_rng()
+    os.makedirs(os.path.dirname(output_path), exist_ok=True)
+    
+    current_hour = 0
+    steps_per_hour = 100 # average density
+    active_spike_countdown = 0
+    
+    with open(output_path, "w") as f:
+        for i in range(num_transactions):
+            # Advance time every ~100 transactions
+            if i % steps_per_hour == 0:
+                current_hour = (current_hour + 1) % 24
+                
+            # Randomly start a fraud spike (correlated event)
+            if active_spike_countdown <= 0 and rng.random() < 0.005:
+                active_spike_countdown = rng.integers(20, 50)
+            
+            # 1. Hour of day (Diurnal pattern)
+            hour = current_hour
+            
+            # 2. Segment & MCC
+            segment = int(rng.choice([0, 1, 2], p=[0.25, 0.60, 0.15]))
+            mcc = int(rng.choice([0, 1, 2, 3, 4, 5], p=[0.3, 0.2, 0.1, 0.1, 0.1, 0.2]))
+            
+            # 3. Fraud Risk with Correlation (Spikes)
+            is_night = (1 <= hour <= 5)
+            base_risk = {0: 0.02, 1: 0.05, 2: 0.15, 3: 0.08, 4: 0.25, 5: 0.12}[mcc]
+            
+            risk_boost = 0.0
+            if active_spike_countdown > 0:
+                risk_boost = 0.4 # Persistent spike
+                active_spike_countdown -= 1
+            elif is_night:
+                risk_boost = 0.2
+                
+            final_risk = base_risk + risk_boost + rng.uniform(-0.05, 0.05)
+            fraud_risk_score = float(np.clip(final_risk * {0: 1.8, 1: 1.0, 2: 0.3}[segment], 0.01, 0.99))
+            
+            # 4. Transaction Details
+            amount = float(rng.lognormal(mean={0: 4.0, 1: 4.5, 2: 6.5, 3: 7.0, 4: 5.0, 5: 3.0}[mcc], sigma=0.8))
+            bin_category = int(rng.integers(0, 10))
+            is_international = bool(rng.random() < (0.4 if mcc == 3 else 0.15))
+            
+            log_entry = {
+                "amount": amount,
+                "merchant_category": mcc,
+                "is_international": is_international,
+                "card_present": bool(rng.random() > 0.5),
+                "user_segment": segment,
+                "user_history_score": float(np.clip(rng.normal({0: 0.3, 1: 0.7, 2: 0.9}[segment], 0.15), 0.1, 1.0)),
+                "device_type": int(rng.choice([0, 1, 2], p=[0.5, 0.4, 0.1])),
+                "bin_category": bin_category,
+                "time_of_day": hour,
+                "transaction_velocity": float(np.clip(rng.random() * 0.2 + (0.5 if active_spike_countdown > 0 else 0.0), 0.1, 0.9)),
+                "fraud_risk_score": fraud_risk_score,
+                "event_marker": "fraud_spike" if active_spike_countdown > 0 else None
+            }
+            f.write(json.dumps(log_entry) + "\n")
+
+if __name__ == "__main__":
+    generate_logs(num_transactions=5000)
+    print("Sequential logs with correlated events generated.")

server/SmartPayEnv_environment.py

@@ -5,7 +5,7 @@
 # LICENSE file in the root directory of this source tree.
 
 """
-SmartPayEnv v3 — Advanced Fintech Reality Layer.
+SmartPayEnv — Advanced Fintech Reality Layer.
 
 High-fidelity benchmark for RL agents in the payment domain.
 Features: 3D Secure (3DS), Chargeback Delays, BIN Affinity, Dynamic Costs, & Cohorts.
@@ -25,8 +25,10 @@ except (ImportError, ValueError):
 
 try:
     from .graders import RoutingEfficacyGrader, FraudDetectionGrader, UserRetentionGrader
+    from .utils import LogLoader
 except (ImportError, ValueError):
     from server.graders import RoutingEfficacyGrader, FraudDetectionGrader, UserRetentionGrader
+    from server.utils import LogLoader
 
 
 # ── Configuration Constants ────────────────────────────────────────────
@@ -69,6 +71,12 @@ class State:
     fraud_wave_drift: float = 0.0
     market_volatility: float = 0.0
     chargeback_queue: list = field(default_factory=list)
+    health_lag_buffer: deque = field(default_factory=lambda: deque(maxlen=3)) # 2-step lag
+    true_fraud_risk: float = 0.0
+    simulation_hour: int = 0
+    active_events: dict = field(default_factory=dict) # e.g. {"fraud_spike": 10, "outage": 5}
+    log_cursor: int = 0
+    review_queue: list = field(default_factory=list) # [{ 'step': int, 'is_fraud': bool, 'amount': float }]
 
 
 class _GatewayState:
@@ -122,6 +130,8 @@ class SmartpayenvEnvironment(Environment):
         self.retention_grader = UserRetentionGrader()
         self._velocity_buffer = deque(maxlen=5)
         self.current_obs   = None
+        self._log_loader   = LogLoader()
+        self._pattern_queue = deque()
 
     def _init_gateways(self) -> None:
         instability = self._cfg["instability"]
@@ -132,63 +142,37 @@ class SmartpayenvEnvironment(Environment):
         ]
 
     def _generate_transaction(self) -> SmartpayenvObservation:
-        # 1. Advanced Diurnal Cycle (UTC)
-        # Peak Fraud: 01:00 - 05:00. Peak Volume: 12:00 - 20:00
-        hour = int(self._state.step_count % 24)
-        is_night = (1 <= hour <= 5)
-        
-        # 2. User Segments (Cohorts)
-        segment = int(self._rng.choice([0, 1, 2], p=[0.25, 0.60, 0.15])) # 0=New, 1=Existing, 2=Premium
-        
-        # Segment behavioral traits
-        fraud_mult = {0: 1.8, 1: 1.0, 2: 0.3}[segment]
-        history_mu  = {0: 0.3, 1: 0.7, 2: 0.9}[segment]
-        
-        # 3. Correlated Merchant Categories (MCC)
-        mcc = int(self._rng.choice([0, 1, 2, 3, 4, 5], p=[0.3, 0.2, 0.1, 0.1, 0.1, 0.2]))
-        
-        # MCC-Amount Correlation
-        amount_mu = {0: 4.0, 1: 4.5, 2: 6.5, 3: 7.0, 4: 5.0, 5: 3.0}[mcc]
-        amount = float(self._rng.lognormal(mean=amount_mu, sigma=0.8))
-        
-        # 4. Statistical Fraud Model
-        wave_drift = self._state.fraud_wave_drift
-        category_risk = {0: 0.02, 1: 0.05, 2: 0.15, 3: 0.08, 4: 0.25, 5: 0.12}[mcc]
-        
-        base_risk = self._cfg["fraud_base_rate"] + wave_drift + category_risk
-        if is_night: base_risk += 0.25 # Night surge
-        
-        is_international = bool(self._rng.random() < (0.4 if mcc == 3 else 0.15))
-        device_type = int(self._rng.choice([0, 1, 2], p=[0.5, 0.4, 0.1])) # 0=Mobile, 1=Web, 2=Unknown
-        
-        final_risk = base_risk + (0.15 if is_international else 0.0)
-        final_risk += (0.2 if device_type == 2 else 0.0)
-        
-        fraud_risk_score = float(np.clip(final_risk * fraud_mult, 0.01, 0.99))
-        user_history_score = float(np.clip(self._rng.normal(history_mu, 0.15), 0.1, 1.0))
+        # Check if we have a queued pattern to replay
+        if self._pattern_queue:
+            log_entry = self._pattern_queue.popleft()
+        else:
+            # Sample sequentially from logs to maintain temporal correlation
+            noise = {0: 0.05, 1: 0.15, 2: 0.3}[self._difficulty]
+            log_entry = self._log_loader.sample(index=self._state.log_cursor, noise_level=noise)
+            self._state.log_cursor += 1
 
-        # 5. Other Transactional Features
-        bin_category = int(self._rng.integers(0, 10))
-        card_present = bool(self._rng.random() > 0.6 if is_night else 0.3)
-        
-        # Velocity and Fraud Risk (History Buffer)
-        velocity = float(np.clip(self._rng.random() * 0.2 + (0.5 if is_night else 0.0), 0.1, 0.9))
+        if log_entry is None:
+            # Fallback to random if logs fail (shouldn't happen)
+            return self._generate_fallback_transaction()
+
+        true_risk = float(log_entry["fraud_risk_score"])
+        self._state.true_fraud_risk = true_risk
 
         return SmartpayenvObservation(
-            amount=amount,
-            merchant_category=mcc,
-            is_international=is_international,
-            card_present=card_present,
+            amount=float(log_entry["amount"]),
+            merchant_category=int(log_entry["merchant_category"]),
+            is_international=bool(log_entry["is_international"]),
+            card_present=bool(log_entry["card_present"]),
             user_type=0, 
-            user_segment=segment,
-            user_history_score=user_history_score,
-            device_type=device_type,
-            bin_category=bin_category,
-            transaction_velocity=velocity,
-            time_of_day=hour,
+            user_segment=int(log_entry["user_segment"]),
+            user_history_score=float(log_entry["user_history_score"]),
+            device_type=int(log_entry["device_type"]),
+            bin_category=int(log_entry["bin_category"]),
+            transaction_velocity=float(log_entry["transaction_velocity"]),
+            time_of_day=int(log_entry["time_of_day"]),
             gateway_success_rates=[g.current_rate for g in self._gateways],
             gateway_states=[g.state for g in self._gateways],
-            fraud_risk_score=fraud_risk_score,
+            observed_fraud_risk=self._get_noisy_risk(float(log_entry["fraud_risk_score"])),
             previous_failures=self._state.consecutive_failures,
             difficulty=self._difficulty,
             reward=0.5,
@@ -198,46 +182,106 @@ class SmartpayenvEnvironment(Environment):
             task_retention_score=0.5,
         )
 
+    def _get_noisy_risk(self, true_risk: float) -> float:
+        """Adds Gaussian noise to the true risk score."""
+        noise = self._rng.normal(0, 0.1)
+        return float(np.clip(true_risk + noise, 0.01, 0.99))
+
+    def _generate_fallback_transaction(self) -> SmartpayenvObservation:
+        # Original logic as fallback
+        hour = int(self._state.step_count % 24)
+        segment = int(self._rng.choice([0, 1, 2], p=[0.25, 0.60, 0.15]))
+        mcc = int(self._rng.choice([0, 1, 2, 3, 4, 5]))
+        amount = float(self._rng.lognormal(mean=4.0, sigma=0.8))
+        
+        self._state.true_fraud_risk = 0.1
+        return SmartpayenvObservation(
+            amount=amount,
+            merchant_category=mcc,
+            is_international=False,
+            card_present=True,
+            user_type=0,
+            user_segment=segment,
+            user_history_score=0.8,
+            device_type=0,
+            bin_category=0,
+            transaction_velocity=0.5,
+            time_of_day=hour,
+            gateway_success_rates=[0.9, 0.9, 0.9],
+            gateway_states=["normal", "normal", "normal"],
+            observed_fraud_risk=0.1,
+            previous_failures=0,
+            difficulty=self._difficulty,
+            reward=0.5,
+            done=False,
+            task_routing_score=0.5,
+            task_fraud_mcc_score=0.5,
+            task_retention_score=0.5,
+        )
+
     def reset(self, difficulty: int = 0) -> SmartpayenvObservation:
         self._difficulty = int(np.clip(difficulty, 0, 2))
         self._cfg        = DIFFICULTY_CONFIG[self._difficulty]
         self._state      = State(episode_id=str(uuid4()), step_count=0)
+        # Random initial cursor for variety, but then sequential within episode
+        self._state.log_cursor = self._rng.integers(0, 100000) 
         self._init_gateways()
         self.route_grader     = RoutingEfficacyGrader()
         self.fraud_grader     = FraudDetectionGrader()
         self.retention_grader = UserRetentionGrader(churn_rate=self._cfg["churn_rate"])
         self._velocity_buffer.clear()
         self.current_obs = self._generate_transaction()
+        # Synchronize simulation clock with the log's starting hour
+        self._state.simulation_hour = self.current_obs.time_of_day
         return self.current_obs
 
     def step(self, action: SmartpayenvAction) -> SmartpayenvObservation:
         self._state.step_count += 1
+        
+        # Advance hour every 20 steps
+        if self._state.step_count % 20 == 0:
+            self._state.simulation_hour = (self._state.simulation_hour + 1) % 24
+        
         if self.current_obs is None: self.reset()
         
         obs = self.current_obs
-        assert obs is not None # Satisfy type checker
-        # 0. Stochastic Reality Drift
-        # Fraud Wave: base rate drifts every step
-        if self._state.step_count % 5 == 0:
-            drift = self._rng.normal(0, 0.05)
-            self._state.fraud_wave_drift = np.clip(self._state.fraud_wave_drift + drift, -0.1, 0.2)
-        
-        # Systemic Volatility: 5% chance of market-wide degradation
-        if self._rng.random() < 0.05:
-            for g in self._gateways:
-                if g.state == "normal":
-                    g.state = "degraded"
-                    g._countdown = int(self._rng.integers(4, 9))
-                    g.current_rate = g.current_rate * 0.7
+        assert obs is not None 
+
+        # 0. Temporal Event Management
+        # Decay active events (Safer way to delete items)
+        self._state.active_events = {e: d - 1 for e, d in self._state.active_events.items() if d > 1}
+
+        # Randomly trigger a systemic gateway outage (Event Correlation)
+        if self._rng.random() < 0.01:
+            self._state.active_events["systemic_outage"] = self._rng.integers(5, 15)
+            # Force multiple gateways into "degraded" state
+            for gw in self._gateways:
+                if self._rng.random() < 0.7:
+                    gw.state = "degraded"
+                    gw._countdown = self._state.active_events["systemic_outage"]
+                    gw.current_rate = gw.base_rate * 0.1
+
+        # 0. Gateway Health Lag Update
+        current_health = {
+            "rates": [g.current_rate for g in self._gateways],
+            "states": [g.state for g in self._gateways]
+        }
+        self._state.health_lag_buffer.append(current_health)
+
+        if self._state.step_count % 10 == 0 and self._rng.random() < 0.2:
+            # Inject a "Fraud Surge" pattern from logs
+            surge_logs = self._log_loader.get_pattern("fraud_surge", count=5)
+            self._pattern_queue.extend(surge_logs)
 
         for gw in self._gateways: gw.step()
 
         # 1. 3DS / Action Logic
-        is_fraud     = (obs.fraud_risk_score >= 0.65)
-        action_block = (action.fraud_decision == 1)
-        action_3ds   = (action.fraud_decision == 2)
+        is_fraud      = (self._state.true_fraud_risk >= 0.65)
+        action_block  = (action.fraud_decision == 1)
+        action_3ds    = (action.fraud_decision == 2)
+        action_review = (action.fraud_decision == 3)
         
-        self.fraud_grader.add_step(action_block or action_3ds, is_fraud)
+        self.fraud_grader.add_step(action_block or action_3ds or action_review, is_fraud)
 
         done = False
         success = False
@@ -247,8 +291,19 @@ class SmartpayenvEnvironment(Environment):
         cb_penalty_this_step = 0.0
 
         if action_block:
-            route_score = obs.fraud_risk_score if is_fraud else (obs.fraud_risk_score * 0.3)
+            route_score = self._state.true_fraud_risk if is_fraud else (self._state.true_fraud_risk * 0.3)
             done = True
+        elif action_review:
+            # Manual Review: Costly but accurate delay
+            total_cost += 5.0 # High internal cost for human time
+            delay = self._rng.integers(10, 25)
+            self._state.review_queue.append({
+                'maturation': self._state.step_count + delay,
+                'is_fraud': is_fraud,
+                'amount': obs.amount
+            })
+            route_score = 0.5 # Neutral immediate feedback
+            success = False # Held in review
         else:
             gw_rates = [g.current_rate for g in self._gateways]
             
@@ -260,7 +315,7 @@ class SmartpayenvEnvironment(Environment):
                 affinity = affinity * 0.15 # Harsh penalty for subpar routing
                 
             # 3DS reduces remaining fraud risk by 90%
-            eff_fraud_risk = obs.fraud_risk_score * (0.1 if action_3ds else 1.0)
+            eff_fraud_risk = self._state.true_fraud_risk * (0.1 if action_3ds else 1.0)
             expected_outcome = gw_rates[gateway] * (1.0 - eff_fraud_risk) * affinity
             expected_outcome = float(np.clip(expected_outcome, 0.05, 1.0))
 
@@ -275,7 +330,7 @@ class SmartpayenvEnvironment(Environment):
                 retries += 1
                 gateway  = (gateway + 1) % 3
                 affinity = BIN_AFFINITY[gateway][obs.bin_category]
-                expected_outcome = gw_rates[gateway] * (1.0 - obs.fraud_risk_score) * affinity
+                expected_outcome = gw_rates[gateway] * (1.0 - self._state.true_fraud_risk) * affinity
                 success = bool(self._rng.random() < expected_outcome)
 
             # Dynamic Cost: % + flat
@@ -310,19 +365,38 @@ class SmartpayenvEnvironment(Environment):
         # Process maturation
         cb_amt: float = 0.0
         pending = []
-        for mat, pen in self._state.chargeback_queue:
-            if self._state.step_count >= mat: 
-                cb_amt = cb_amt + float(pen)
+        for maturation_step, penalty_amount in self._state.chargeback_queue:
+            if self._state.step_count >= maturation_step: 
+                cb_amt += float(penalty_amount)
             else: 
-                pending.append((mat, pen))
+                pending.append((maturation_step, penalty_amount))
         self._state.chargeback_queue = pending
 
-        # Finalize
+        # 3. Apply Lagged Health to Next Observation
+        # Use first item in buffer for 2-step lag if buffer is full
+        lagged_health = self._state.health_lag_buffer[0] if len(self._state.health_lag_buffer) >= 3 else current_health
+        
         self.current_obs = self._generate_transaction()
-        self.current_obs.gateway_success_rates = [g.current_rate for g in self._gateways]
-        self.current_obs.gateway_states        = [g.state for g in self._gateways]
+        self.current_obs.time_of_day = self._state.simulation_hour
+        self.current_obs.gateway_success_rates = lagged_health["rates"]
+        self.current_obs.gateway_states        = lagged_health["states"]
         self.current_obs.chargeback_penalty_applied = cb_amt
         
+        # Process and report matured Manual Reviews
+        matured_reviews = []
+        remaining_reviews = []
+        for r in self._state.review_queue:
+            if self._state.step_count >= r['maturation']:
+                matured_reviews.append({
+                    'amount': r['amount'],
+                    'is_fraud': r['is_fraud'],
+                    'outcome': 'rejected' if r['is_fraud'] else 'accepted'
+                })
+            else:
+                remaining_reviews.append(r)
+        self._state.review_queue = remaining_reviews
+        self.current_obs.review_resolutions = matured_reviews
+        
         if done or self._state.step_count >= 100: self.current_obs.done = True
         
         fs = self.fraud_grader.evaluate()

server/graders.py

@@ -102,7 +102,7 @@ class FraudDetectionGrader:
             (self.tn + self.fn)
         )
         if denominator == 0:
-            return 0.1  # Fail — insufficient data signal
+            return 0.5  # Neutral — insufficient data to compute MCC
         mcc = numerator / denominator
         score = (mcc + 1.0) / 2.0  # Normalize [-1, 1] → [0, 1]
         return max(0.001, min(0.999, score))

server/utils.py

@@ -0,0 +1,50 @@
+import json
+import random
+import os
+
+class LogLoader:
+    def __init__(self, log_path="data/transactions_log.jsonl"):
+        self.log_path = log_path
+        self.logs = []
+        if os.path.exists(log_path):
+            with open(log_path, "r") as f:
+                for line in f:
+                    self.logs.append(json.loads(line))
+        else:
+            print(f"Warning: Log file {log_path} not found.")
+
+    def sample(self, index=None, noise_level=0.05):
+        if not self.logs:
+            return None
+        
+        if index is not None:
+            entry = self.logs[index % len(self.logs)].copy()
+        else:
+            entry = random.choice(self.logs).copy()
+        
+        # Inject noise into float fields
+        if noise_level > 0:
+            for key in ["amount", "fraud_risk_score", "user_history_score", "transaction_velocity"]:
+                if key in entry:
+                    noise = random.uniform(-noise_level, noise_level)
+                    entry[key] = max(0.01, entry[key] * (1 + noise))
+                    
+        return entry
+
+    def get_pattern(self, pattern_type="fraud_surge", count=10):
+        """Returns a subset of logs matching a certain pattern."""
+        if not self.logs:
+            return []
+            
+        if pattern_type == "fraud_surge":
+            # Filter for high fraud risk
+            candidates = [l for l in self.logs if l.get("fraud_risk_score", 0) > 0.5]
+        elif pattern_type == "premium_only":
+            candidates = [l for l in self.logs if l.get("user_segment") == 2]
+        else:
+            candidates = self.logs
+            
+        if not candidates:
+            return [random.choice(self.logs) for _ in range(count)]
+            
+        return [random.choice(candidates) for _ in range(count)]

tests/test_env_logs.py

@@ -0,0 +1,23 @@
+import sys
+import os
+
+# Add the root directory to sys.path
+sys.path.append(os.path.abspath(os.path.join(os.path.dirname(__file__), "..")))
+
+from server.SmartPayEnv_environment import SmartpayenvEnvironment
+from models import SmartpayenvAction
+
+def test_env():
+    env = SmartpayenvEnvironment()
+    obs = env.reset()
+    print(f"Initial Obs: Amount={obs.amount}, Segment={obs.user_segment}, FraudRisk={obs.fraud_risk_score}")
+    
+    for i in range(20):
+        action = SmartpayenvAction(gateway=0, fraud_decision=0, retry_strategy=0)
+        obs = env.step(action)
+        print(f"Step {i+1}: Amount={obs.amount:.2f}, FraudRisk={obs.fraud_risk_score:.2f}, Hour={obs.time_of_day}")
+        if env._pattern_queue:
+            print(f"  [Pattern Queued: {len(env._pattern_queue)} items remaining]")
+
+if __name__ == "__main__":
+    test_env()

tests/test_partial_obs.py

@@ -0,0 +1,37 @@
+import sys
+import os
+import time
+
+# Add the root directory to sys.path
+sys.path.append(os.path.abspath(os.path.join(os.path.dirname(__file__), "..")))
+
+from server.SmartPayEnv_environment import SmartpayenvEnvironment
+from models import SmartpayenvAction
+
+def test_partial_obs():
+    env = SmartpayenvEnvironment()
+    obs = env.reset()
+    
+    print("--- STEP 0 (Initial) ---")
+    print(f"Observed Risk: {obs.observed_fraud_risk:.4f}")
+    print(f"True Risk (Hidden): {env._state.true_fraud_risk:.4f}")
+    print(f"Gateway Rates: {obs.gateway_success_rates}")
+    
+    # Store initial rates
+    initial_rates = env.current_obs.gateway_success_rates.copy()
+    
+    for i in range(1, 10):
+        # Force a change in gateway rates to see the lag
+        for g in env._gateways:
+            g.current_rate = min(1.0, g.current_rate + 0.01) # Slowly drift up
+            
+        action = SmartpayenvAction(gateway=0, fraud_decision=0, retry_strategy=0)
+        obs = env.step(action)
+        
+        print(f"\n--- STEP {i} ---")
+        print(f"Observed Risk: {obs.observed_fraud_risk:.4f} (True: {env._state.true_fraud_risk:.4f})")
+        print(f"Observed Health: {obs.gateway_success_rates}")
+        print(f"Hidden Real Health: {[g.current_rate for g in env._gateways]}")
+        
+if __name__ == "__main__":
+    test_partial_obs()

tests/{test_v3_features.py → test_reality_features.py}

@@ -3,7 +3,7 @@ import sys
 import os
 
 # Add the root directory to path to import models and environment
-sys.path.append(os.path.dirname(os.path.abspath(__file__)))
+sys.path.append(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
 
 from server.SmartPayEnv_environment import SmartpayenvEnvironment
 from models import SmartpayenvAction
@@ -42,14 +42,14 @@ def test_3ds_mechanics():
     fraudulent_obs_found = False
     for _ in range(100):
         obs = env.reset(difficulty=1)
-        if obs.fraud_risk_score > 0.7:
+        if obs.observed_fraud_risk > 0.7:
             fraudulent_obs_found = True
             # Case 1: Allow (High risk of failure)
             # Case 2: 3DS (High chance of success if no abandonment)
             action_3ds = SmartpayenvAction(gateway=2, retry_strategy=0, fraud_decision=2)
             next_obs = env.step(action_3ds)
             # 3DS doesn't end episode immediately (unless it's step 100)
-            print(f"  - 3DS on high risk ({obs.fraud_risk_score:.2f}) -> Reward: {next_obs.reward:.2f}")
+            print(f"  - 3DS on high risk ({obs.observed_fraud_risk:.2f}) -> Reward: {next_obs.reward:.2f}")
             break
     
     if not fraudulent_obs_found:
@@ -69,7 +69,7 @@ def test_chargeback_delay():
     
     for i in range(1, 101):
         # Find a fraud
-        is_fraud = obs.fraud_risk_score >= 0.65
+        is_fraud = obs.observed_fraud_risk >= 0.65
         
         if is_fraud and not cb_queued:
             # Allow it

tests/test_temporal.py

@@ -0,0 +1,48 @@
+import requests
+import json
+import time
+
+URL = "http://localhost:7860"
+
+def test_temporal():
+    # 1. Reset
+    res = requests.post(f"{URL}/reset", json={"difficulty": 1})
+    obs = res.json().get("observation")
+    last_hour = obs.get("time_of_day")
+    
+    print(f"Initial Hour: {last_hour}")
+    
+    correlated_failures = 0
+    high_velocity_count = 0
+    
+    for i in range(100):
+        # Action doesn't matter much for this test
+        res = requests.post(f"{URL}/step", json={"action": {"gateway": 0, "fraud_decision": 0, "retry_strategy": 0}})
+        data = res.json()
+        obs = data.get("observation")
+        
+        hour = obs.get("time_of_day")
+        states = obs.get("gateway_states")
+        
+        # Check hour progression
+        if hour != last_hour:
+            print(f"Hour advanced to {hour}")
+            last_hour = hour
+            
+        # Check correlation (Systemic Outage)
+        down_count = sum(1 for s in states if s != "normal")
+        if down_count >= 2:
+            correlated_failures += 1
+            print(f"Step {i}: Cluster failure detected! States: {states}")
+        
+        # Velocity might be high during fraud spikes
+        # Actually transaction_velocity is in observation? Let's check model.py
+        # No, it's not in observation yet. Let's check models.py
+        
+    print(f"Correlated failures detected: {correlated_failures}")
+
+if __name__ == "__main__":
+    try:
+        test_temporal()
+    except Exception as e:
+        print(f"Failed to connect to server: {e}. Make sure it is running.")