Upload 31 files

.gitattributes

@@ -33,3 +33,28 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+chart.png filter=lfs diff=lfs merge=lfs -text
+image-png-pages_research-paper/page_01.png filter=lfs diff=lfs merge=lfs -text
+image-png-pages_research-paper/page_02.png filter=lfs diff=lfs merge=lfs -text
+image-png-pages_research-paper/page_03.png filter=lfs diff=lfs merge=lfs -text
+image-png-pages_research-paper/page_04.png filter=lfs diff=lfs merge=lfs -text
+image-png-pages_research-paper/page_05.png filter=lfs diff=lfs merge=lfs -text
+image-png-pages_research-paper/page_06.png filter=lfs diff=lfs merge=lfs -text
+image-png-pages_research-paper/page_07.png filter=lfs diff=lfs merge=lfs -text
+image-png-pages_research-paper/page_08.png filter=lfs diff=lfs merge=lfs -text
+image-png-pages_research-paper/page_09.png filter=lfs diff=lfs merge=lfs -text
+image-png-pages_research-paper/page_10.png filter=lfs diff=lfs merge=lfs -text
+image-png-pages_research-paper/page_11.png filter=lfs diff=lfs merge=lfs -text
+image-png-pages_research-paper/page_12.png filter=lfs diff=lfs merge=lfs -text
+image-png-pages_research-paper/page_13.png filter=lfs diff=lfs merge=lfs -text
+image-png-pages_research-paper/page_14.png filter=lfs diff=lfs merge=lfs -text
+image-png-pages_research-paper/page_15.png filter=lfs diff=lfs merge=lfs -text
+image-png-pages_research-paper/page_16.png filter=lfs diff=lfs merge=lfs -text
+image-png-pages_research-paper/page_17.png filter=lfs diff=lfs merge=lfs -text
+image-png-pages_research-paper/page_18.png filter=lfs diff=lfs merge=lfs -text
+image-png-pages_research-paper/page_19.png filter=lfs diff=lfs merge=lfs -text
+image-png-pages_research-paper/page_20.png filter=lfs diff=lfs merge=lfs -text
+image-png-pages_research-paper/page_21.png filter=lfs diff=lfs merge=lfs -text
+image-png-pages_research-paper/page_22.png filter=lfs diff=lfs merge=lfs -text
+reference.pdf filter=lfs diff=lfs merge=lfs -text
+Vector-HaSH_for-6yr-old.pdf filter=lfs diff=lfs merge=lfs -text

Vector-HaSH-Simple-Paper.tex

@@ -0,0 +1,155 @@
+\documentclass[conference]{IEEEtran}
+\usepackage{cite}
+\usepackage{amsmath,amssymb,amsfonts}
+\usepackage{algorithmic}
+\usepackage{graphicx}
+\usepackage{textcomp}
+\usepackage{xcolor}
+\usepackage{listings}
+\usepackage{hyperref}
+
+% Python code listing style
+\lstset{
+  language=Python,
+  basicstyle=\ttfamily\small,
+  keywordstyle=\color{blue},
+  commentstyle=\color{green!50!black},
+  stringstyle=\color{red},
+  showstringspaces=false,
+  numbers=left,
+  numberstyle=\tiny\color{gray},
+  frame=single,
+  breaklines=true
+}
+
+\def\BibTeX{{\rm B\kern-.05em{\sc i\kern-.025em b}\kern-.08em
+    T\kern-.1667em\lower.7ex\hbox{E}\kern-.125emX}}
+
+\begin{document}
+
+\title{Vector-HaSH: A Magical Memory Palace for the Brain\\
+\large Explained for Smart 6-Year-Olds!}
+
+\author{\IEEEauthorblockN{Agent-Self Swarm Intelligence}}
+
+\maketitle
+
+\begin{abstract}
+Imagine your brain is a giant Lego castle. How does it remember a supersized recipe for baking 10,000 cookies without forgetting the first step? Older models, like the Hopfield network, try to squish every single cookie recipe into one box, and eventually, the box explodes (we call this the ``memory cliff''). This paper talks about Vector-HaSH, a shiny new tool that fixes this problem! It splits memory into two jobs: a ''scaffold'' (like a treasure map of empty boxes) and the ''content'' (the actual treasure inside the boxes). By placing memories on this map using a simple 2D steering wheel (velocity), the brain can remember tens of thousands of things in a row without breaking a sweat!
+\end{abstract}
+
+\section{Introduction}
+Have you ever tried to memorize a very long grocery list? If you put milk, eggs, carrots, and 50 other things in your pocket all at once, your pocket might rip. In neuroscience (the study of brains), scientists noticed that computer memory models (like Hopfield networks) do exactly this. After seeing too many patterns, they suddenly forget EVERYTHING. This catastrophic failure is known as the **memory cliff**.
+
+But your real brain does not do this! Your brain uses two magic helpers:
+1. **Grid Cells:** These are like special GPS trackers in your brain. They make a map of invisible tiles so you always know where you are standing.
+2. **Hippocampus (HPC):** This is the memory vault. It stores the rich, colorful pictures of what you see (like a giant chocolate cake).
+
+**Vector-HaSH** (which stands for Vector-Hippocampal and Scaffold Hypothesis) is a clever system that lets these two helpers hold hands. Instead of memorizing the whole cake at once, the Grid Cells create a path (a scaffold) and the Hippocampus attaches the cake to one of the steps on the path. To get to the next memory, you just turn the steering wheel (velocity vector) and move to the next tile! 
+
+\section{Related Work}
+Before Vector-HaSH, scientists believed in the classic Hopfield Network. Think of it as a magical rubber band ball. You stretch it with new memories. But if you stretch it too many times, SNAP! The rubber bands break.
+
+Other researchers tried to fix it by using ``sparse'' inputs (putting only tiny rubber bands). But even then, the capacity scaling was limited. You could only store $O(N)$ memories, where $N$ is the number of neurons. If you wanted to remember $10,000$ steps of a dance routine, you needed millions of brain cells. Vector-HaSH changes the game entirely by using grid cell networks as a sequence scaffold, escaping the dreaded memory cliff.
+
+\section{Proposed Method}
+Imagine making a long train out of toy cars. 
+In the old way, every toy car had to carry the heavy load of remembering exactly which car came next by staring at the whole car. 
+
+In Vector-HaSH, the train tracks themselves (Grid Cells) tell you where to go next. All you need is a tiny steering wheel (a 2-dimensional velocity) to move forward!
+
+\subsection{The Three Big Steps}
+1. \textbf{The Grid Space (The Map):} Think of it like a giant chessboard. You are a knight jumping across it. The board is made of a few tiny, connected circles (modules).
+2. \textbf{The Hippocampus (The Polaroid Camera):} For every square on the chessboard, the camera takes a snapshot and remembers the sensory details.
+3. \textbf{Velocity Shift (The Steering Wheel):} To remember the next scene in the movie, a very tiny, simple system (a Multi-Layer Perceptron or MLP) just gives a "push" (velocity vector) to the Grid Cells. The Grid Cells step forward, and the Hippocampus wakes up the next memory!
+
+By doing this, the memory capacity goes UP exponentially! It can remember 14,000 steps easily, whereas the old model failed at 30 steps!
+
+\section{Code Examples: Tiny and Pythonic}
+Let's look at the real code logic for Vector-HaSH. We will make tiny, runnable scripts so you can build your own mini-brain at home!
+
+\subsection{Example 1: Moving the Grid Cells}
+How does the brain know where to go next? It uses a "velocity" to shift the grid. Here is a tiny Python example:
+
+\begin{lstlisting}[language=Python]
+import numpy as np
+
+# Line 1: Imagine our grid map has 5 spots (0 to 4).
+grid_map_size = 5 
+
+# Line 2: You are currently sitting at spot number 2.
+current_grid_state = 2 
+
+# Line 3: The steering wheel tells us to move forward by 1 step!
+velocity_shift = 1 
+
+# Line 4: We calculate the new spot! We use the modulo operator (%), 
+# which acts like a circle. If you step past 4, you go back to 0!
+next_grid_state = (current_grid_state + velocity_shift) % grid_map_size
+
+# Line 5: Print the result! The car moved to spot 3!
+print(f"We drove to spot: {next_grid_state}")
+\end{lstlisting}
+
+\emph{Explanation for a 6-year old:}
+\begin{itemize}
+    \item \textbf{Line 1:} We build a tiny race track with 5 spaces.
+    \item \textbf{Line 2:} We put our toy car on space number 2.
+    \item \textbf{Line 3:} We press the gas pedal to move 1 space.
+    \item \textbf{Line 4:} We calculate where the car lands. Because the track is a circle, if we go past the end, we warp back to the start!
+    \item \textbf{Line 5:} We tell the world where our car parked!
+\end{itemize}
+
+\subsection{Example 2: Hippocampus Remembering the Cake}
+Now that we are on a new grid spot, the Hippocampus needs to hook a memory onto it. We use a matrix multiplication (which is just a fancy way of giving high-fives).
+
+\begin{lstlisting}[language=Python]
+import numpy as np
+
+# Line 1: This is our grid spot (Spot 3). It is turned ON (1).
+grid_activity = np.array([0, 0, 0, 1, 0]) 
+
+# Line 2: These are the memory weights. 
+# They decide what picture appears when a spot is ON.
+hippocampus_weights = np.array([
+    [0.1, 0.2], # Spot 0 -> sees an apple
+    [0.5, 0.9], # Spot 1 -> sees a dog
+    [0.8, 0.1], # Spot 2 -> sees a car
+    [0.9, 0.9], # Spot 3 -> sees a GIANT CAKE!
+    [0.3, 0.4]  # Spot 4 -> sees a tree
+])
+
+# Line 3: We multiply our current spot by the weights.
+# It acts like a magic flashlight revealing the picture.
+recalled_memory = grid_activity.dot(hippocampus_weights)
+
+# Line 4: Boom! We see the numbers [0.9, 0.9] which means CAKE!
+print(f"I remember: {recalled_memory}")
+\end{lstlisting}
+
+\emph{Explanation for a 6-year old:}
+\begin{itemize}
+    \item \textbf{Line 1:} We have a row of light switches. Only the switch for Spot 3 is turned ON.
+    \item \textbf{Line 2:} We have a magical book of secrets (weights). Each switch is glued to a different secret picture.
+    \item \textbf{Line 3:} We use `.dot()`, which is a robot taking the ON switch and pulling its secret picture out of the book.
+    \item \textbf{Line 4:} The robot shows us the picture. Yummy cake!
+\end{itemize}
+
+\section{Experiments}
+The smart scientists put Vector-HaSH through a tough obstacle course:
+1. \textbf{The Dark Room Test:} Can the grid cells still work if you turn off the lights? Yes! Even if you can't see the colorful walls (no sensory input), the steering wheel (velocity) still drives the car around the invisible grid map.
+2. \textbf{The Mega-Marathon Test:} Can Vector-HaSH run for 14,000 steps without stumbling over its shoelaces? Yes! Even a tiny network recalled the exact sequence of 14,000 turns without making a mistake!
+
+\section{Results}
+Vector-HaSH scored an A+! The results showed that biological brains use a \textbf{Sequence Scaffold}. 
+If you learn a new song, you don't build a new piano. You use the same piano keys (the grid cells scaffold) and just play them in a different order! Because the brain reuses the grid cells, it saves a MASSIVE amount of energy and avoids the memory cliff. This is exactly how "Memory Athletes" (people who can memorize a whole deck of cards in 20 seconds) use the "Memory Palace" trick. They walk through a familiar house in their mind (the grid) and drop off memories in every room!
+
+\section{Conclusion}
+The brain is the coolest computer in the world. Instead of getting overwhelmed by remembering everything at once, it uses Grid Cells to build a map, and Hippocampus cells to take pictures along the way. Vector-HaSH proves that with a tiny 2D steering wheel (velocity), we can navigate super-long memories flawlessly. Next time you play with your Lego sets, remember: your brain is snapping together a track and placing memories on it, block by block!
+
+\begin{thebibliography}{00}
+\bibitem{b1} Vector-HaSH Authors. "Episodic and associative memory through grid-like scaffolds." Nature, 2024.
+\bibitem{b2} Hopfield, J. J. "Neural networks and physical systems with emergent collective computational abilities." PNAS, 1982.
+\end{thebibliography}
+
+\end{document}

Vector-HaSH_for-6yr-old.pdf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:bf0c1d472b727cca18821076bed5b85f27ca1260d5c6125abd6e8f89f5e19a77
+size 112508

data_fetcher.py

@@ -0,0 +1,324 @@
+#!/usr/bin/env python3
+"""
+╔══════════════════════════════════════════════════════════════════════════╗
+║  data_fetcher.py — MT5 XAUUSDc M3 Data Fetcher                        ║
+║  Fetches 1-year OHLCV + spread from MetaTrader5 (3-min candles)        ║
+║  Saves CSV + symbol_info.json.  Run locally with MT5 terminal open.    ║
+╚══════════════════════════════════════════════════════════════════════════╝
+"""
+
+import sys, time, json
+import numpy as np
+import pandas as pd
+from datetime import datetime, timedelta, timezone
+from pathlib import Path
+
+try:
+    import MetaTrader5 as mt5
+except ImportError:
+    print("ERROR: MetaTrader5 package not installed.  Run:  pip install MetaTrader5")
+    sys.exit(1)
+
+# ══════════════════════════════════════════════════════════════════════════
+# CONFIGURATION
+# ══════════════════════════════════════════════════════════════════════════
+SYMBOL        = "XAUUSDc"
+TIMEFRAME_M1  = mt5.TIMEFRAME_M1             # Fetch M1, resample to M3
+TF_LABEL      = "M3"
+RESAMPLE_MINS = 3                             # 3-minute candles
+LOOKBACK_DAYS = 365                           # 1 year
+OUTPUT_DIR    = Path(__file__).resolve().parent
+OUTPUT_CSV    = OUTPUT_DIR / f"{SYMBOL}_{TF_LABEL}_data.csv"
+OUTPUT_JSON   = OUTPUT_DIR / f"{SYMBOL}_symbol_info.json"
+
+
+# ══════════════════════════════════════════════════════════════════════════
+# MT5 CONNECTION
+# ══════════════════════════════════════════════════════════════════════════
+def init_mt5() -> None:
+    """Initialize MT5 connection with retries."""
+    for attempt in range(3):
+        if mt5.initialize():
+            info = mt5.terminal_info()
+            print(f"✓ MT5 connected — Build {info.build}, Company: {info.company}")
+            return
+        print(f"  Attempt {attempt+1}/3 failed, retrying in 2s...")
+        time.sleep(2)
+    print(f"✗ MT5 initialization failed: {mt5.last_error()}")
+    sys.exit(1)
+
+
+def validate_symbol(symbol: str) -> dict:
+    """Validate symbol exists and return its properties."""
+    info = mt5.symbol_info(symbol)
+    if info is None:
+        symbols = mt5.symbols_get()
+        gold_syms = [s.name for s in symbols if "XAU" in s.name or "GOLD" in s.name.upper()]
+        print(f"✗ Symbol '{symbol}' not found.")
+        if gold_syms:
+            print(f"  Available gold symbols: {gold_syms}")
+        else:
+            print(f"  No gold symbols found. Check your broker.")
+        sys.exit(1)
+
+    if not info.visible:
+        mt5.symbol_select(symbol, True)
+        time.sleep(0.5)
+
+    props = {
+        "name":           info.name,
+        "digits":         info.digits,
+        "point":          info.point,
+        "spread":         info.spread,
+        "trade_mode":     info.trade_mode,
+        "volume_min":     info.volume_min,
+        "volume_max":     info.volume_max,
+        "volume_step":    info.volume_step,
+        "trade_contract_size": info.trade_contract_size,
+        "trade_tick_value":    info.trade_tick_value,
+        "trade_tick_size":     info.trade_tick_size,
+        "currency_profit":     info.currency_profit,
+    }
+    print(f"✓ Symbol validated: {info.name}")
+    print(f"  Digits: {info.digits} | Point: {info.point} | "
+          f"Spread: {info.spread} | Min Lot: {info.volume_min} | "
+          f"Max Lot: {info.volume_max} | Contract: {info.trade_contract_size}")
+    return props
+
+
+# ══════════════════════════════════════════════════════════════════════════
+# DATA FETCHING (M1 → resample to M3)
+# ══════════════════════════════════════════════════════════════════════════
+def fetch_ohlcv(symbol: str, days: int) -> pd.DataFrame:
+    """Fetch M1 OHLCV from MT5, then resample to 3-minute candles."""
+    utc_now   = datetime.now(timezone.utc)
+    date_from = utc_now - timedelta(days=days)
+
+    print(f"\n→ Fetching M1 bars: {date_from.date()} to {utc_now.date()} …")
+    print(f"  (Will resample M1 → M{RESAMPLE_MINS} after fetching)")
+
+    # Fetch in chunks to avoid MT5 limits (max ~100k bars per request)
+    chunk_days = 30
+    all_frames = []
+    current_start = date_from
+
+    while current_start < utc_now:
+        chunk_end = min(current_start + timedelta(days=chunk_days), utc_now)
+        rates = mt5.copy_rates_range(symbol, TIMEFRAME_M1, current_start, chunk_end)
+
+        if rates is not None and len(rates) > 0:
+            chunk_df = pd.DataFrame(rates)
+            chunk_df["time"] = pd.to_datetime(chunk_df["time"], unit="s", utc=True)
+            all_frames.append(chunk_df)
+            print(f"    Chunk {current_start.date()} → {chunk_end.date()}: {len(chunk_df):,} M1 bars")
+        else:
+            err = mt5.last_error()
+            print(f"    Chunk {current_start.date()} → {chunk_end.date()}: no data ({err})")
+
+        current_start = chunk_end
+
+    if not all_frames:
+        print(f"✗ No M1 data returned from any chunk")
+        sys.exit(1)
+
+    df = pd.concat(all_frames, ignore_index=True)
+    df = df.drop_duplicates(subset="time").sort_values("time").reset_index(drop=True)
+    df.rename(columns={"real_volume": "volume"}, inplace=True, errors="ignore")
+
+    print(f"✓ Total M1 bars: {len(df):,}")
+    print(f"  M1 range: {df['time'].iloc[0]} → {df['time'].iloc[-1]}")
+
+    # ── Resample M1 → M3 ──
+    print(f"\n→ Resampling M1 → M{RESAMPLE_MINS} …")
+    df.set_index("time", inplace=True)
+
+    resampled = df.resample(f"{RESAMPLE_MINS}min", label="right", closed="right").agg({
+        "open":        "first",
+        "high":        "max",
+        "low":         "min",
+        "close":       "last",
+        "tick_volume":  "sum",
+        "spread":      "last",
+    }).dropna(subset=["open"])
+
+    # Also resample volume if present
+    if "volume" in df.columns:
+        resampled["volume"] = df["volume"].resample(f"{RESAMPLE_MINS}min", label="right", closed="right").sum()
+
+    resampled.reset_index(inplace=True)
+
+    # Ensure required columns
+    required = ["time", "open", "high", "low", "close", "tick_volume", "spread"]
+    for col in required:
+        if col not in resampled.columns:
+            if col == "spread":
+                resampled["spread"] = 0
+            elif col == "tick_volume":
+                resampled["tick_volume"] = 0
+
+    print(f"✓ Resampled to {len(resampled):,} M{RESAMPLE_MINS} bars")
+    print(f"  M3 range: {resampled['time'].iloc[0]} → {resampled['time'].iloc[-1]}")
+    return resampled
+
+
+
+def fetch_spread_from_ticks(symbol: str, days: int) -> float | None:
+    """Fetch recent tick data to compute median spread.  Returns median spread in points."""
+    print(f"\n→ Computing spread from tick data (sampling last 30 days) …")
+
+    utc_now    = datetime.now(timezone.utc)
+    tick_start = utc_now - timedelta(days=min(days, 30))
+
+    ticks = mt5.copy_ticks_range(symbol, tick_start, utc_now, mt5.COPY_TICKS_INFO)
+
+    if ticks is None or len(ticks) == 0:
+        print(f"  ⚠ No tick data available, using bar spread column")
+        return None
+
+    tick_df = pd.DataFrame(ticks)
+    tick_df["time"]       = pd.to_datetime(tick_df["time"], unit="s", utc=True)
+    tick_df["spread_pts"] = (tick_df["ask"] - tick_df["bid"]) / mt5.symbol_info(symbol).point
+
+    avg_spread    = tick_df["spread_pts"].mean()
+    median_spread = tick_df["spread_pts"].median()
+    max_spread    = tick_df["spread_pts"].quantile(0.99)
+
+    print(f"✓ Processed {len(tick_df):,} ticks")
+    print(f"  Avg spread: {avg_spread:.1f} pts | "
+          f"Median: {median_spread:.1f} pts | "
+          f"99th pctl: {max_spread:.1f} pts")
+    return median_spread
+
+
+# ══════════════════════════════════════════════════════════════════════════
+# DATA VALIDATION & CLEANING
+# ══════════════════════════════════════════════════════════════════════════
+def validate_data(df: pd.DataFrame) -> pd.DataFrame:
+    """Validate and clean OHLCV data."""
+    print(f"\n→ Validating data quality …")
+    issues = []
+
+    # 1. NaN
+    nan_count = df[["open", "high", "low", "close"]].isnull().sum().sum()
+    if nan_count > 0:
+        issues.append(f"  ⚠ {nan_count} NaN values in OHLCV — forward-filling")
+        df[["open", "high", "low", "close"]] = df[["open", "high", "low", "close"]].ffill()
+
+    # 2. OHLC integrity
+    bad_hl = (df["high"] < df["low"]).sum()
+    if bad_hl > 0:
+        issues.append(f"  ⚠ {bad_hl} bars where high < low — swapping")
+        mask = df["high"] < df["low"]
+        df.loc[mask, ["high", "low"]] = df.loc[mask, ["low", "high"]].values
+
+    bad_range = ((df["open"] > df["high"]) | (df["open"] < df["low"]) |
+                 (df["close"] > df["high"]) | (df["close"] < df["low"])).sum()
+    if bad_range > 0:
+        issues.append(f"  ⚠ {bad_range} bars where open/close outside H-L — clamping")
+        df["open"]  = df["open"].clip(lower=df["low"], upper=df["high"])
+        df["close"] = df["close"].clip(lower=df["low"], upper=df["high"])
+
+    # 3. Duplicates
+    dups = df["time"].duplicated().sum()
+    if dups > 0:
+        issues.append(f"  ⚠ {dups} duplicate timestamps — keeping last")
+        df = df.drop_duplicates(subset="time", keep="last")
+
+    # 4. Large gaps (> 5 days)
+    time_diff  = df["time"].diff()
+    large_gaps = time_diff[time_diff > pd.Timedelta(days=5)]
+    for idx in large_gaps.index:
+        gap = time_diff.loc[idx]
+        issues.append(f"  ⚠ Large gap: {df['time'].iloc[idx-1]} → {df['time'].iloc[idx]} ({gap})")
+
+    # 5. Sort
+    df = df.sort_values("time").reset_index(drop=True)
+
+    # 6. Remove weekends
+    weekend_mask  = df["time"].dt.dayofweek.isin([5, 6])
+    weekend_count = weekend_mask.sum()
+    if weekend_count > 0:
+        issues.append(f"  ℹ Removed {weekend_count} weekend bars")
+        df = df[~weekend_mask].reset_index(drop=True)
+
+    if issues:
+        for issue in issues:
+            print(issue)
+    else:
+        print("  ✓ Data quality: PASS (no issues found)")
+
+    print(f"\n  Final dataset: {len(df):,} bars")
+    print(f"  Price range: {df['close'].min():.2f} — {df['close'].max():.2f}")
+    print(f"  Avg spread: {df['spread'].mean():.1f} pts")
+    print(f"  Date range: {df['time'].iloc[0].date()} → {df['time'].iloc[-1].date()}")
+    return df
+
+
+# ══════════════════════════════════════════════════════════════════════════
+# MAIN
+# ══════════════════════════════════════════════════════════════════════════
+def main():
+    print("=" * 68)
+    print(f"  MT5 Data Fetcher — {SYMBOL} {TF_LABEL} (1 Year)")
+    print("=" * 68)
+
+    # 1. Connect
+    init_mt5()
+
+    try:
+        # 2. Validate symbol & save info
+        sym_props = validate_symbol(SYMBOL)
+
+        # Save symbol info JSON for Colab / EA consumption
+        with open(OUTPUT_JSON, "w") as f:
+            json.dump(sym_props, f, indent=2, default=str)
+        print(f"\n✓ Symbol info saved: {OUTPUT_JSON}")
+
+        # 3. Fetch OHLCV
+        df = fetch_ohlcv(SYMBOL, LOOKBACK_DAYS)
+
+        # 4. Enhance spread from ticks
+        median_spread = fetch_spread_from_ticks(SYMBOL, LOOKBACK_DAYS)
+        if median_spread is not None:
+            zero_mask = df["spread"] == 0
+            if zero_mask.sum() > 0:
+                df.loc[zero_mask, "spread"] = int(median_spread)
+                print(f"  Filled {zero_mask.sum()} zero-spread bars with median: {median_spread:.0f}")
+
+        # 5. Validate
+        df = validate_data(df)
+
+        # 6. Add metadata columns
+        df["hour"]      = df["time"].dt.hour
+        df["dayofweek"] = df["time"].dt.dayofweek
+        df["returns"]   = np.log(df["close"] / df["close"].shift(1))
+
+        # 7. Save CSV
+        output_cols = [
+            "time", "open", "high", "low", "close",
+            "tick_volume", "spread", "hour", "dayofweek", "returns",
+        ]
+        if "volume" in df.columns and "volume" not in output_cols:
+            output_cols.insert(5, "volume")
+
+        df_out = df[[c for c in output_cols if c in df.columns]]
+        df_out.to_csv(OUTPUT_CSV, index=False)
+
+        print(f"\n{'=' * 68}")
+        print(f"  ✓ SAVED: {OUTPUT_CSV}")
+        print(f"  ✓ Rows: {len(df_out):,} | Columns: {len(df_out.columns)}")
+        print(f"  ✓ File size: {OUTPUT_CSV.stat().st_size / 1024:.0f} KB")
+        print(f"{'=' * 68}")
+
+        print(f"\nSample (first 3 rows):")
+        print(df_out.head(3).to_string(index=False))
+        print(f"\nSample (last 3 rows):")
+        print(df_out.tail(3).to_string(index=False))
+
+    finally:
+        mt5.shutdown()
+        print("\n✓ MT5 connection closed.")
+
+
+if __name__ == "__main__":
+    main()

implementation_plan.md

@@ -0,0 +1,37 @@
+# Implementation Plan - Vector-HaSH Financial Trader
+
+## Objective
+Implement the Vector-HaSH algorithm for predicting pure financial prices (XAUUSD 3-minute timeframe) inside Google Colab (T4 GPU). Evaluate strategy via strict anchored Walk-Forward Optimization (WFO) to eliminate forward-looking bias.
+
+## Proposed Strategy Architecture
+
+### 1. Feature Engineering
+We will rely **ONLY** on pure price transformations.
+- Compute rolling features: Log returns, rolling volatility, and sequence windows of size $W$ (e.g. 15 bars). Let the state at time $t$ be $\mathbf{x}_t \in \mathbb{R}^{W}$.
+- **Discrete Quantization**: To map continuous prices into the discrete elements similar to the visual "sbook" in Vector-HaSH, we will use `flash-kmeans` (with $K$ clusters) to quantize the historical $\mathbf{x}_t$ vectors into discrete sensory classes $\mathbf{s}_t$.
+
+### 2. Vector-HaSH Memory Scaffold
+Instead of a 2D spatial grid, we will use a **1D Continuous Track** (approximating time).
+- **Grid Scaffold ($\mathbf{g}_t$)**: Synthesize multiscale 1D grid cell representations (using sine/cosine waves or cyclic shifts).
+- **Place Cells ($\mathbf{p}_t$)**: Project Grid cells into a sparse higher-dimensional space: $\mathbf{p}_t = \sigma(\mathbf{W}_{pg} \mathbf{g}_t)$.
+- **Hetero-associative Memory**: Train the sensory-to-place map $\mathbf{W}_{sp}$ dynamically using Recursive Least Squares (RLS), mimicking the [pseudotrain_2d_iterative_step](file:///C:/Users/User/Desktop/debugrem/Vector-HaSH-agent-trader/VectorHaSH-main/MTT.py#133-140) seen in [MTT.py](file:///C:/Users/User/Desktop/debugrem/Vector-HaSH-agent-trader/VectorHaSH-main/MTT.py).
+
+### 3. Machine Learning Wrapper (XGBoost)
+- At time $t$, extract the *Memory Recall Error* ($\mathbf{s}_t - \hat{\mathbf{s}}_t$) and the *Place Cell Activations* ($\mathbf{p}_t$).
+- Feed these VectorHaSH embeddings into an XGBoost Classifier/Regressor.
+- Target: Next bar log return $r_{t+1}$ or direction $\text{sign}(r_{t+1})$.
+
+### 4. Anchored Walk-Forward Optimization
+To avoid cheating:
+- Train/Test splits expand over time.
+- Fold 1: Train $[0, T]$, Test $[T, T+H]$.
+- Fold 2: Train $[0, T+H]$, Test $[T+H, T+2H]$.
+- `flash-kmeans`, Vector-HaSH memory construction, and XGBoost fitting will occur **ONLY** on the Training slice of each fold, and act out-of-sample on the Test slice.
+
+### 5. Mono-Script Colab Implementation (`vector_hash_trader.py`)
+- Vectorized using PyTorch (`device='cuda'`) or NumPy (`cuml`/`cupy`/XGBoost-GPU).
+- Plotting module included: cumulative returns, drawdown, WFO heatmaps, and memory collapse analysis.
+
+## Verification
+- Assert strictly positive index lookups when indexing arrays (no `t` to `t+1` leakage before target definition).
+- Verify standard performance metrics: Sharpe Ratio, Sortino Ratio, Max Drawdown.

reference.pdf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:236a8b2612efdb18b213e349062c81db7be02eab115cfb141cf001d928e73b53
+size 6404425

vector_hash_trader_colab.py

@@ -0,0 +1,380 @@
+#!/usr/bin/env python3
+# ==============================================================================
+# ❗ RUN THESE IN A GOOGLE COLAB CELL BEFORE EXECUTING THE SCRIPT:
+# !pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118
+# !pip install xgboost pandas numpy matplotlib seaborn tqdm
+# !pip install git+https://github.com/svg-project/flash-kmeans.git
+# ==============================================================================
+"""
+╔══════════════════════════════════════════════════════════════════════════╗
+║  vector_hash_trader_colab.py — Vector-HaSH Financial Time-Series Trader  ║
+║  Highly optimized monolithic GPU/Vectorized script for Google Colab.     ║
+║  Predicts pure prices via Anchored Walk-Forward Optimization (No Peeking)║
+║  Uses Vector-HaSH biologically plausible Scaffold representations + XGB. ║
+╚══════════════════════════════════════════════════════════════════════════╝
+"""
+import os
+import sys
+import gc
+import time
+import numpy as np
+import pandas as pd
+import matplotlib.pyplot as plt
+import seaborn as sns
+from pathlib import Path
+from tqdm import tqdm
+
+import torch
+import torch.nn as nn
+import torch.nn.functional as F
+
+try:
+    import xgboost as xgb
+except ImportError:
+    print("Running pip install xgboost...")
+    os.system("pip install xgboost")
+    import xgboost as xgb
+
+try:
+    from sklearn.metrics import accuracy_score, classification_report, mean_squared_error
+except ImportError:
+    pass
+
+# Try to import flash_kmeans if installed, else fallback to PyTorch custom KMeans
+try:
+    from flash_kmeans import batch_kmeans_Euclid
+    FLASH_KMEANS_AVAILABLE = True
+    print("[INFO] flash_kmeans is available. We will use Triton-accelerated K-Means!")
+except ImportError:
+    FLASH_KMEANS_AVAILABLE = False
+    print("[WARN] flash_kmeans not installed. Using PyTorch fallback.")
+
+# ══════════════════════════════════════════════════════════════════════════
+# PyTorch Fallback KMeans (if flash_kmeans not installed)
+# ══════════════════════════════════════════════════════════════════════════
+def fast_pytorch_kmeans(x, n_clusters, max_iter=100, tol=1e-4, device='cuda'):
+    """Simple PyTorch KMeans for fallback."""
+    N, D = x.shape
+    # Randomly initialize centers from data points
+    indices = torch.randperm(N, device=device)[:n_clusters]
+    centers = x[indices].clone()
+    
+    for i in range(max_iter):
+        # Compute distances (N, K)
+        dists = torch.cdist(x, centers, p=2)
+        # Assign clusters
+        cluster_ids = torch.argmin(dists, dim=1)
+        
+        # Compute new centers
+        new_centers = torch.zeros_like(centers)
+        counts = torch.bincount(cluster_ids, minlength=n_clusters).float().unsqueeze(1)
+        new_centers.scatter_add_(0, cluster_ids.unsqueeze(1).expand(-1, D), x)
+        
+        # Avoid division by zero
+        new_centers = new_centers / counts.clamp(min=1)
+        
+        # Check convergence
+        center_shift = torch.norm(centers - new_centers, p=2)
+        centers = new_centers
+        if center_shift < tol:
+            break
+            
+    return cluster_ids, centers
+
+# ══════════════════════════════════════════════════════════════════════════
+# Vector-HaSH Scaffold Engine
+# ══════════════════════════════════════════════════════════════════════════
+class VectorHashMemory(nn.Module):
+    """
+    Simulates the Hippocampal/Entorhinal Grid structure for 1D Financial Sequences.
+    g_t: Grid sequence state (Time representation)
+    p_t: Place cells (Sparse projection of grid cells)
+    s_t: Sensory cells (Discretized pure price states/embeddings)
+    W_pg: Fixed random sparse projection from Grid to Place.
+    W_sp: Associative mapping connecting Place to Sensory (RLS trained).
+    """
+    def __init__(self, N_grid=30, N_place=400, N_sensory=64, sparsity=0.1, device='cuda'):
+        super().__init__()
+        self.device = device
+        self.Ng = N_grid
+        self.Np = N_place
+        self.Ns = N_sensory
+        
+        # Grid to Place sparse random projection (Non-trainable but fixed)
+        self.W_pg = torch.randn(self.Np, self.Ng, device=device, dtype=torch.float32)
+        
+        # Apply Sparsity mask (like pruning in MTT.py)
+        mask = (torch.rand(self.Np, self.Ng, device=device) < sparsity).float()
+        self.W_pg = self.W_pg * mask
+        
+        # Sensory Memory Retrieval weights (Trained via Pseudo-inverse / RLS on Train Fold)
+        self.W_sp = torch.zeros(self.Ns, self.Np, device=device, dtype=torch.float32)
+        
+    def generate_grid_scaffold(self, T):
+        """Generates a 1D continuous cyclic ring attractor for time."""
+        # Multi-scale sinusoidal oscillators (phases) corresponding to progression
+        t = torch.arange(T, device=self.device, dtype=torch.float32)
+        g_t = []
+        for i in range(self.Ng // 2):
+            freq = 1.0 / (2.0 ** (i * 0.1)) # Exponential scale
+            g_t.append(torch.sin(t * freq))
+            g_t.append(torch.cos(t * freq))
+        if len(g_t) < self.Ng:
+            g_t.append(torch.zeros_like(t))
+        g_t = torch.stack(g_t, dim=1) # (T, Ng)
+        return g_t
+        
+    def generate_place_cells(self, g_t):
+        """Project grid to place cells and apply ReLU for sparsity."""
+        # (T, Ng) @ (Ng, Np) -> (T, Np)
+        p_t = F.relu(torch.matmul(g_t, self.W_pg.T))
+        return p_t
+        
+    def memorize(self, p_t, s_t):
+        """
+        Calculates W_sp = S * pseudo_inverse(P) using Batched PyTorch SVD.
+        This represents the biological Hetero-Associative memory storage.
+        p_t: (T, Np)
+        s_t: (T, Ns)
+        """
+        # We need pseudo-inverse of P^T, which has shape (Np, T). The inverse will be (T, Np).
+        p_t_inv = torch.linalg.pinv(p_t.T)
+        # W_sp: (Ns, T) @ (T, Np) -> (Ns, Np)
+        self.W_sp = torch.matmul(s_t.T, p_t_inv)
+        
+    def recall(self, p_t):
+        """
+        Returns reconstructed Sensory state.
+        \\hat{S} = P @ W_sp^T
+        """
+        return torch.matmul(p_t, self.W_sp.T)
+
+# ══════════════════════════════════════════════════════════════════════════
+# DATA PROCESSING MODULE
+# ══════════════════════════════════════════════════════════════════════════
+def load_and_prepare_data(csv_path, window_size=16):
+    """Loads XAUUSD M3 pure prices and constructs sequential state matrices."""
+    print(f"→ Loading {csv_path} ...")
+    df = pd.read_csv(csv_path)
+    
+    # We only care about pure price. Use 'close' and calculate 'returns' if missing
+    if 'returns' not in df.columns:
+        df['returns'] = np.log(df['close'] / df['close'].shift(1))
+    
+    df = df.dropna().reset_index(drop=True)
+    
+    # Target: Predict next return
+    df['target_return'] = df['returns'].shift(-1)
+    df['target_class'] = (df['target_return'] > 0).astype(int) # 1 if UP, 0 if DOWN
+    
+    df = df.dropna().reset_index(drop=True)
+    
+    # Create Rolling Winodws representation X_t (using last `window_size` returns)
+    returns_arr = df['returns'].values
+    N_samples = len(returns_arr) - window_size + 1
+    
+    X_seq = np.zeros((N_samples, window_size), dtype=np.float32)
+    for i in range(window_size):
+        X_seq[:, i] = returns_arr[i : N_samples + i]
+        
+    df_aligned = df.iloc[window_size - 1:].reset_index(drop=True)
+    
+    print(f"✓ Data constructed! {N_samples} sequences of shape {window_size}.")
+    return df_aligned, X_seq
+
+# ══════════════════════════════════════════════════════════════════════════
+# ANCHORED WALK-FORWARD OPTIMIZATION STRATEGY
+# ══════════════════════════════════════════════════════════════════════════
+def execute_wfo_strategy(df, X_seq, n_splits=5, device='cuda'):
+    print(f"\n{'='*68}")
+    print(f"  STARTING ANCHORED WALK-FORWARD OPTIMIZATION ({n_splits} folds)")
+    print(f"{'='*68}")
+    
+    N = len(df)
+    fold_size = N // (n_splits + 1)
+    
+    all_predictions = []
+    all_targets = []
+    all_returns = []
+    
+    equity_timestamps = []
+    equity_curve = [1.0] # Starts at 1.0 multiplier
+    
+    for fold in range(n_splits):
+        train_end = fold_size * (fold + 1)
+        test_end = train_end + fold_size
+        if fold == n_splits - 1:
+            test_end = N # Take the rest for the last fold
+            
+        print(f"\n► Fold {fold+1}/{n_splits} | Train: [0 : {train_end}] | Test: [{train_end} : {test_end}]")
+        
+        # 1. Split Data
+        X_train_np = X_seq[:train_end]
+        y_train_np = df['target_class'].iloc[:train_end].values
+        
+        X_test_np = X_seq[train_end:test_end]
+        y_test_np = df['target_class'].iloc[train_end:test_end].values
+        returns_test_np = df['target_return'].iloc[train_end:test_end].values
+        timestamps_test = df['time'].iloc[train_end:test_end].values
+        
+        # Send to Device
+        X_train = torch.tensor(X_train_np, dtype=torch.float32, device=device)
+        X_test = torch.tensor(X_test_np, dtype=torch.float32, device=device)
+        
+        # 2. Flash KMeans Quantization (Sensory Encoding) -> Convert 15D window to K=64 Centroids
+        K_clusters = 64
+        
+        if FLASH_KMEANS_AVAILABLE:
+            # flash-kmeans expects input (Batch, N, Dim), so we add batch dim
+            X_tr_exp = X_train.unsqueeze(0)
+            cluster_ids, centers, _ = batch_kmeans_Euclid(X_tr_exp, n_clusters=K_clusters, tol=1e-4, verbose=False)
+            centers = centers.squeeze(0) # (K, D)
+            
+            # Predict for train
+            dists_tr = torch.cdist(X_train, centers, p=2)
+            c_ids_tr = torch.argmin(dists_tr, dim=1)
+            # Predict for test
+            dists_te = torch.cdist(X_test, centers, p=2)
+            c_ids_te = torch.argmin(dists_te, dim=1)
+            
+        else:
+            c_ids_tr, centers = fast_pytorch_kmeans(X_train, n_clusters=K_clusters, device=device)
+            dists_te = torch.cdist(X_test, centers, p=2)
+            c_ids_te = torch.argmin(dists_te, dim=1)
+            
+        # One-hot encode the sensory targets: (T, K)
+        S_train = F.one_hot(c_ids_tr, num_classes=K_clusters).float()
+        S_test = F.one_hot(c_ids_te, num_classes=K_clusters).float()
+        
+        # 3. Vector-HaSH Memorization
+        print("  → Initializing Vector-HaSH Scaffold & Memorizing...")
+        VH = VectorHashMemory(N_grid=32, N_place=512, N_sensory=K_clusters, sparsity=0.15, device=device)
+        
+        G_train = VH.generate_grid_scaffold(T=train_end)
+        P_train = VH.generate_place_cells(G_train)
+        
+        # Memorize (Hetero-association: Place -> Sensory) using Pseudo-Inverse
+        VH.memorize(P_train, S_train)
+        
+        # Reconstruction Error features
+        S_hat_train = VH.recall(P_train)
+        error_train = (S_train - S_hat_train).detach()
+        
+        # 4. Out-Of-Sample Memory Simulation
+        # For out of sample, we just map time to grid to place, and try to recall.
+        G_test_full = VH.generate_grid_scaffold(T=test_end)
+        G_test = G_test_full[train_end:test_end]
+        P_test = VH.generate_place_cells(G_test)
+        
+        S_hat_test = VH.recall(P_test)
+        error_test = (S_test - S_hat_test).detach()
+        
+        # 5. XGBoost Modeling
+        print("  → Training highly-optimized GPU XGBoost Model...")
+        # Feature Matrix: Concat Raw X_t, Place Cells P_t, Recall Error \epsilon_t
+        F_train = torch.cat([X_train, P_train, error_train], dim=1).cpu().numpy()
+        F_test = torch.cat([X_test, P_test, error_test], dim=1).cpu().numpy()
+        
+        dtrain = xgb.DMatrix(F_train, label=y_train_np)
+        dtest = xgb.DMatrix(F_test, label=y_test_np)
+        
+        params = {
+            'objective': 'binary:logistic',
+            'tree_method': 'hist',
+            'device': 'cuda', # T4 GPU Acceleration
+            'eval_metric': 'logloss',
+            'learning_rate': 0.05,
+            'max_depth': 4,
+            'subsample': 0.8,
+            'colsample_bytree': 0.8
+        }
+        
+        evallist = [(dtrain, 'train'), (dtest, 'eval')]
+        bst = xgb.train(params, dtrain, num_boost_round=100, evals=evallist, verbose_eval=False)
+        
+        # Predict on Test Split!
+        preds_prob = bst.predict(dtest)
+        preds_class = (preds_prob > 0.5).astype(int)
+        
+        acc = accuracy_score(y_test_np, preds_class)
+        print(f"  ✓ Fold {fold+1} completed! Out-of-Sample Accuracy: {acc:.4f}")
+        
+        # Calculate Strategy Returns
+        # Simple strategy: If pred=1, buy. If pred=0, sell.
+        trade_signals = np.where(preds_class == 1, 1, -1)
+        strategy_returns = trade_signals * returns_test_np
+        
+        for ret in strategy_returns:
+            equity_curve.append(equity_curve[-1] * (1 + ret))
+            
+        equity_timestamps.extend(timestamps_test)
+        all_predictions.extend(preds_class)
+        all_targets.extend(y_test_np)
+        all_returns.extend(strategy_returns)
+        
+        # Clear CUDA memory
+        del X_train, X_test, X_tr_exp, G_train, P_train, S_train, S_hat_train, error_train
+        del G_test_full, G_test, P_test, S_test, S_hat_test, error_test, VH
+        torch.cuda.empty_cache()
+        gc.collect()
+
+    print(f"\n{'='*68}")
+    
+    # 6. Evaluation & Plotting
+    overall_acc = accuracy_score(all_targets, all_predictions)
+    print(f"OVERALL OUT-OF-SAMPLE ACCURACY: {overall_acc:.4f}")
+    
+    cum_ret = np.prod([1+r for r in all_returns])
+    print(f"OVERALL CUMULATIVE RETURN (Multiplier): {cum_ret:.4f}x")
+    
+    # Calculate Drawdown
+    eq_array = np.array(equity_curve)
+    peaks = np.maximum.accumulate(eq_array)
+    drawdowns = (eq_array - peaks) / peaks
+    max_dd = np.min(drawdowns) * 100
+    print(f"MAX DRAWDOWN: {max_dd:.2f}%")
+    
+    # Matplotlib Graph Generation
+    plt.style.use('dark_background')
+    fig, axs = plt.subplots(2, 1, figsize=(14, 10), gridspec_kw={'height_ratios': [3, 1]})
+    
+    # Equity Curve
+    axs[0].plot(eq_array, color='cyan', linewidth=1.5, label=f"Strategy Equity (Return: {cum_ret:.2f}x)")
+    axs[0].set_title(f"XAUUSD Vector-HaSH Strategy - Anchored Walking-Forward Equity", fontsize=16, color='white')
+    axs[0].set_ylabel("Portfolio Multiplier", fontsize=12)
+    axs[0].grid(axis='y', linestyle='--', alpha=0.3)
+    axs[0].legend(loc="upper left")
+    
+    # Drawdown Curve
+    axs[1].fill_between(range(len(drawdowns)), drawdowns*100, 0, color='red', alpha=0.5, label="Drawdown (%)")
+    axs[1].set_title(f"Drawdown Profile (Max DD: {max_dd:.2f}%)", fontsize=14, color='white')
+    axs[1].set_ylabel("Drawdown %", fontsize=12)
+    axs[1].set_xlabel("Out-Of-Sample Chronological Steps", fontsize=12)
+    axs[1].grid(axis='y', linestyle='--', alpha=0.3)
+    axs[1].legend(loc="lower left")
+    
+    plt.tight_layout()
+    output_png = "vector_hash_equity_report.png"
+    plt.savefig(output_png, dpi=300, bbox_inches='tight')
+    print(f"✓ Strategy report chart saved to {output_png}!")
+
+# ══════════════════════════════════════════════════════════════════════════
+# EXECUTION SCRIPT
+# ══════════════════════════════════════════════════════════════════════════
+if __name__ == "__main__":
+    device = 'cuda' if torch.cuda.is_available() else 'cpu'
+    print(f"Runtime Device: {device.upper()}")
+    
+    csv_file = Path("XAUUSDc_M3_data.csv")
+    if not csv_file.exists():
+        print(f"ERROR: {csv_file} not found in the current directory.")
+        sys.exit(1)
+        
+    df_data, X_seq_data = load_and_prepare_data(csv_file, window_size=16)
+    
+    # Optional: subset for extremely rapid testing (just uncomment to run faster)
+    # df_data = df_data.iloc[-10000:].reset_index(drop=True)
+    # X_seq_data = X_seq_data[-10000:]
+    
+    execute_wfo_strategy(df_data, X_seq_data, n_splits=5, device=device)
+