feat: add Jetson ARM64 support and improve documentation

- Add Dockerfile.jetson for NVIDIA Jetson Orin Nano/Xavier with L4T base
- Add run-jetson.sh build/run script with GPU runtime support
- Update CLAUDE.md: single test command, hack/ scripts section, GPU notes
- Add test_startrek_similarity.py for topic similarity testing

.gitignore

@@ -68,7 +68,7 @@ venv/
 crossword-app/backend-py/src/services/model_cache/
 hack/model_cache/
 .KARO.md
-CLAUDE.md
+#CLAUDE.md
 crossword-app/backend-py/faiss_cache/
 cache-dir/models--google--flan-t5-base/
 cache-dir/models--google--flan-t5-large/
@@ -79,3 +79,5 @@ cache-dir/.locks/models--google--flan-t5-small/
 cache-dir/embeddings_all-mpnet-base-v2_100000.npy 
 cache-dir/frequencies_100000.pkl 
 cache-dir/vocabulary_100000.pkl
+
+train.jsonl

CLAUDE.md

@@ -7,10 +7,12 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co
 This is a full-stack AI-powered crossword puzzle generator:
 - **Python Backend** (`crossword-app/backend-py/`) - Primary implementation with dynamic word generation
 - **React Frontend** (`crossword-app/frontend/`) - Modern React app with interactive crossword UI
-- **Node.js Backend** (`backend/`) - Legacy implementation (deprecated)
+- **Hack Scripts** (`hack/`) - Experimental/development scripts for testing algorithms
 
 Current deployment uses the Python backend with Docker containerization.
 
+> **Note:** The `backend/` directory contains a deprecated Node.js implementation - do not use or modify.
+
 ## Development Commands
 
 ### Frontend Development
@@ -30,6 +32,7 @@ cd crossword-app/backend-py
 python run_tests.py                                    # Run all tests
 pytest test-unit/ -v                                  # Run unit tests
 pytest test-integration/ -v                           # Run integration tests
+pytest test-unit/test_crossword_generator.py::TestClassName::test_method -v  # Run single test
 python test_integration_minimal.py                    # Quick test without ML deps
 
 # Development server
@@ -41,14 +44,6 @@ python test_softmax_service.py                       # Test word selection logic
 python test_distribution_normalization.py            # Test distribution normalization across topics
 ```
 
-### Backend Development (Node.js - Legacy)
-```bash
-cd backend
-npm install
-npm run dev          # Start Express server on http://localhost:3000
-npm test             # Run tests
-```
-
 ### Docker Deployment
 ```bash
 # Build and run locally
@@ -72,6 +67,13 @@ cd crossword-app/frontend
 npm run lint        # ESLint (if configured)
 ```
 
+### Running hack/ Scripts
+```bash
+# All hack/ scripts should use cache-dir/ from root for model loading
+cd hack
+python test_soft_minimum_quick.py  # Most scripts auto-detect cache-dir
+```
+
 ## Architecture Overview
 
 ### Full-Stack Components
@@ -82,6 +84,7 @@ npm run lint        # ESLint (if configured)
 - Custom hook: `useCrossword.js` manages API calls and puzzle state
 - Interactive crossword grid with cell navigation and solution reveal
 - Debug tab for visualizing word selection process (when enabled)
+- Frontend controls for similarity temperature and difficulty weight tuning
 
 **Python Backend** (`crossword-app/backend-py/` - Primary)
 - FastAPI web framework serving both API and static frontend files
@@ -89,11 +92,7 @@ npm run lint        # ESLint (if configured)
 - No static word files - all words generated on-demand from 100K+ vocabulary
 - WordNet-based clue generation with semantic definitions
 - Comprehensive caching system for models, embeddings, and vocabulary
-
-**Node.js Backend** (`backend/` - Legacy - Deprecated)
-- Express.js with static JSON word files
-- Original implementation, no longer actively maintained
-- Used for comparison and fallback testing only
+- PyTorch tensor support with GPU optimization for embeddings
 
 ### Core Python Backend Components
 
@@ -104,7 +103,7 @@ npm run lint        # ESLint (if configured)
 - Temperature-controlled softmax for balanced word selection randomness
 - 50% word overgeneration strategy for better crossword grid fitting
 - **Multi-topic intersection**: `_compute_multi_topic_similarities()` with vectorized soft minimum, geometric/harmonic means
-- **Adaptive beta mechanism**: Automatically adjusts threshold (0.25→0.175→0.103...) to ensure 15+ word minimum
+- **Adaptive beta mechanism**: Automatically adjusts threshold (10.0→7.0→4.9... via 0.7x decay) to ensure 15+ word minimum
 - **Performance optimized**: 40x speedup through vectorized operations over loop-based approach
 - Key method: `generate_thematic_words()` - Returns words with semantic similarity scores and frequency tiers
 
@@ -143,17 +142,13 @@ npm run lint        # ESLint (if configured)
 
 **Python Backend (Primary):**
 - FastAPI, uvicorn, pydantic (web framework)
-- sentence-transformers, torch (AI word generation)
+- sentence-transformers, torch (AI word generation with GPU support)
 - wordfreq (vocabulary database)
 - nltk (WordNet clue generation)
 - scikit-learn (clustering and similarity)
 - numpy (embeddings and mathematical operations)
 - pytest, pytest-asyncio (testing)
 
-**Node.js Backend (Legacy - Deprecated):**
-- Express.js, cors, helmet
-- JSON file-based word storage
-
 The application requires AI dependencies for core functionality - no fallback to static word lists.
 
 ### API Endpoints
@@ -174,13 +169,13 @@ Python backend provides the following REST API:
 - `test-integration/test_local.py` - End-to-end integration testing
 - `test_integration_minimal.py` - Quick functionality test without heavy ML dependencies
 
-**Multi-Topic Testing & Development Scripts:**
-- `hack/test_soft_minimum_quick.py` - Quick soft minimum method verification
-- `hack/test_optimized_soft_minimum.py` - Performance testing (40x speedup validation)
-- `hack/debug_adaptive_beta_bug.py` - Adaptive beta mechanism debugging
-- `hack/test_adaptive_fix.py` - Full vocabulary testing with adaptive beta
-- `hack/test_simpler_case.py` - Compatible topic testing (animals + nature)
-- All hack/ scripts use shared cache-dir for model loading consistency
+**Multi-Topic Testing & Development Scripts** (`hack/`):
+- `test_soft_minimum_quick.py` - Quick soft minimum method verification
+- `test_optimized_soft_minimum.py` - Performance testing (40x speedup validation)
+- `debug_adaptive_beta_bug.py` - Adaptive beta mechanism debugging
+- `test_adaptive_fix.py` - Full vocabulary testing with adaptive beta
+- `test_simpler_case.py` - Compatible topic testing (animals + nature)
+- All hack/ scripts must use `cache-dir/` from project root for model loading
 
 **Frontend Tests:**
 - Component testing with React Testing Library (if configured)
@@ -293,4 +288,7 @@ VITE_API_BASE_URL=http://localhost:7860  # Points to Python backend
 - ✅ **Debug visualization**: Optional debug tab for development/analysis
 - ✅ **Comprehensive caching**: Models, embeddings, and vocabulary cached for performance
 - ✅ **Modern stack**: FastAPI + React with Docker deployment ready
-- the cache is present in root cache-dir/ folder. every program in hack folder should use this as the cache-dir for loading sentence transformer models
+- ✅ **GPU support**: PyTorch tensor operations with optional CUDA acceleration
+
+### Cache Directory
+The model cache is located at `cache-dir/` in the project root. All `hack/` scripts must use this directory for loading sentence-transformer models to ensure consistency.

Dockerfile.jetson

@@ -0,0 +1,76 @@
+# Dockerfile for NVIDIA Jetson (ARM64) - Jetson Orin Nano, Xavier, etc.
+# Uses NVIDIA's L4T PyTorch container as base for proper GPU support
+
+# Stage 1: Builder
+FROM nvcr.io/nvidia/l4t-pytorch:r36.2.0-pth2.1-py3 AS builder
+
+WORKDIR /app
+
+# Install Node.js for frontend build
+RUN apt-get update && apt-get install -y \
+    curl \
+    git \
+    && curl -fsSL https://deb.nodesource.com/setup_18.x | bash - \
+    && apt-get install -y nodejs \
+    && rm -rf /var/lib/apt/lists/*
+
+# Copy frontend package files and install dependencies
+COPY crossword-app/frontend/package*.json ./frontend/
+RUN cd frontend && npm ci
+
+# Install Python dependencies (PyTorch already included in base image)
+COPY crossword-app/backend-py/requirements.txt ./backend-py/
+RUN pip install --no-cache-dir -r backend-py/requirements.txt
+
+# Copy all source code
+COPY crossword-app/frontend/ ./frontend/
+COPY crossword-app/backend-py/ ./backend-py/
+COPY crossword-app/words/ ./backend-py/words/
+
+# Copy cache directory with pre-built models and NLTK data
+COPY cache-dir/ ./cache-dir/
+RUN chmod -R 755 ./cache-dir/ || true
+
+# Build the React frontend
+RUN cd frontend && npm run build
+
+# Copy built frontend files to Python backend public directory
+RUN mkdir -p backend-py/public && cp -r frontend/dist/* backend-py/public/
+
+# Stage 2: Runtime
+FROM nvcr.io/nvidia/l4t-pytorch:r36.2.0-pth2.1-py3 AS runtime
+
+# Install minimal runtime dependencies
+RUN apt-get update && apt-get install -y \
+    curl \
+    && rm -rf /var/lib/apt/lists/*
+
+# Create non-root user
+RUN useradd -m -u 1000 appuser
+
+WORKDIR /app/backend-py
+
+# Copy Python packages from builder (sentence-transformers, etc.)
+COPY --from=builder /usr/local/lib/python3.10/dist-packages /usr/local/lib/python3.10/dist-packages
+
+# Copy built application files
+COPY --from=builder --chown=appuser:appuser /app/backend-py ./
+
+# Copy cache directory
+COPY --from=builder --chown=appuser:appuser /app/cache-dir /app/backend-py/cache
+
+USER appuser
+
+EXPOSE 7860
+
+ENV NODE_ENV=production
+ENV PORT=7860
+ENV PYTHONPATH=/app/backend-py
+ENV PYTHONUNBUFFERED=1
+ENV PIP_NO_CACHE_DIR=1
+ENV CACHE_DIR=/app/backend-py/cache
+ENV NLTK_DATA=/app/backend-py/cache/nltk_data
+ENV VOCAB_SOURCE=norvig
+ENV NORVIG_VOCAB_PATH=/app/backend-py/words/norvig/count_1w100k.txt
+
+CMD ["python", "-m", "uvicorn", "app:app", "--host", "0.0.0.0", "--port", "7860", "--workers", "1"]

hack/test_startrek_similarity.py

@@ -0,0 +1,63 @@
+#!/usr/bin/env python3
+"""Test semantic similarity between spock/kirk and startrek"""
+
+import sys
+import os
+sys.path.insert(0, os.path.join(os.path.dirname(__file__), '..', 'crossword-app', 'backend-py', 'src'))
+
+from services.thematic_word_service import ThematicWordService
+import torch
+import torch.nn.functional as F
+
+# Initialize service with Norvig vocabulary
+service = ThematicWordService(
+    cache_dir="../cache-dir",
+    vocab_size_limit=100000
+)
+service.initialize()
+
+# Check if startrek is in vocabulary
+if 'startrek' in service.vocabulary:
+    startrek_idx = service.vocabulary.index('startrek')
+    print(f'✅ startrek found at index {startrek_idx} in vocabulary')
+else:
+    print('❌ startrek NOT in vocabulary')
+    sys.exit(1)
+
+# Get startrek embedding
+startrek_embedding = service.vocab_embeddings[startrek_idx]
+print(f'Startrek embedding shape: {startrek_embedding.shape}')
+
+# Encode spock and kirk
+spock_embedding = service.model.encode(['spock'], convert_to_tensor=True)[0]
+kirk_embedding = service.model.encode(['kirk'], convert_to_tensor=True)[0]
+
+# Calculate cosine similarities
+startrek_norm = F.normalize(startrek_embedding.unsqueeze(0), p=2, dim=1)
+spock_norm = F.normalize(spock_embedding.unsqueeze(0), p=2, dim=1)
+kirk_norm = F.normalize(kirk_embedding.unsqueeze(0), p=2, dim=1)
+
+sim_spock_startrek = torch.mm(spock_norm, startrek_norm.T).item()
+sim_kirk_startrek = torch.mm(kirk_norm, startrek_norm.T).item()
+
+print(f'\n📊 Semantic Similarities:')
+print(f'spock → startrek: {sim_spock_startrek:.4f}')
+print(f'kirk → startrek: {sim_kirk_startrek:.4f}')
+
+# Now test with combined "spock kirk" as topic
+combined_embedding = service.model.encode(['spock kirk'], convert_to_tensor=True)[0]
+combined_norm = F.normalize(combined_embedding.unsqueeze(0), p=2, dim=1)
+sim_combined_startrek = torch.mm(combined_norm, startrek_norm.T).item()
+print(f'spock kirk → startrek: {sim_combined_startrek:.4f}')
+
+# Check similarity to some other words for comparison
+test_words = ['enterprise', 'space', 'science', 'fiction', 'television', 'show', 'series', 'star', 'trek']
+print(f'\n📊 Comparison with other words:')
+for word in test_words:
+    if word in service.vocabulary:
+        word_idx = service.vocabulary.index(word)
+        word_embedding = service.vocab_embeddings[word_idx]
+        word_norm = F.normalize(word_embedding.unsqueeze(0), p=2, dim=1)
+        sim_spock = torch.mm(spock_norm, word_norm.T).item()
+        sim_kirk = torch.mm(kirk_norm, word_norm.T).item()
+        print(f'{word:12} → spock: {sim_spock:.4f}, kirk: {sim_kirk:.4f}')

run-jetson.sh

@@ -0,0 +1,65 @@
+#!/bin/bash
+set -e
+
+# Build and run script for NVIDIA Jetson devices (Orin Nano, Xavier, etc.)
+
+show_usage() {
+    echo "Usage: $0 [COMMAND]"
+    echo ""
+    echo "Commands:"
+    echo "  build    - Build the Jetson Docker image"
+    echo "  run      - Run the container with GPU support"
+    echo "  both     - Build and run (default)"
+    echo "  shell    - Run with bash shell for debugging"
+    echo ""
+}
+
+IMAGE_NAME="crossword-py-ai:jetson"
+DOCKER_ARGS="--rm -p 7860:7860 --runtime nvidia \
+    -e ENABLE_DEBUG_TAB=true \
+    -e VOCAB_SOURCE=norvig \
+    -e DIFFICULTY_WEIGHT=0.2"
+
+build_image() {
+    echo "🔨 Building Jetson Docker image..."
+    docker build -f Dockerfile.jetson -t $IMAGE_NAME .
+}
+
+run_container() {
+    echo "🚀 Running on Jetson with GPU..."
+    docker run $DOCKER_ARGS $IMAGE_NAME
+}
+
+run_shell() {
+    echo "🐚 Running shell for debugging..."
+    docker run -it $DOCKER_ARGS $IMAGE_NAME /bin/bash
+}
+
+# Parse command
+COMMAND="${1:-both}"
+
+case "$COMMAND" in
+    build)
+        build_image
+        ;;
+    run)
+        run_container
+        ;;
+    both)
+        build_image
+        run_container
+        ;;
+    shell)
+        build_image
+        run_shell
+        ;;
+    -h|--help|help)
+        show_usage
+        exit 0
+        ;;
+    *)
+        echo "Error: Unknown command '$COMMAND'"
+        show_usage
+        exit 1
+        ;;
+esac