fix: 3 critical issues — dataset privacy, crawler pollution, offset reset

ROOT CAUSE OF PUSH FAILURES:
HF API returned 403 'Private repository storage limit reached' — private
datasets on free tier have a small storage cap. Previous 35K queued pairs
couldn't upload regardless of how we batched.

FIXES:

1. Dataset PUBLIC (axentx/surrogate-1-training-pairs):
- PUT /api/datasets/.../settings {private:false} → HTTP 200
- Public datasets have unlimited storage on free tier
- Training data for OSS agent — public is appropriate

2. agentic-crawler.sh: stop polluting training-pairs.jsonl
- Placeholder 'auto-summary pending' pairs are not trainable data
- Now writes to ~/.surrogate/state/agentic-crawl-raw.jsonl (separate)
- Real summaries can be generated later by a separate daemon
- Was 35K of the 35,198 queued pairs (>99% pollution)

3. One-time offset reset on next boot:
- .training-push-offset = current line count
- .self-ingest-offset = current line count
- Skip the polluted backlog, start fresh
- Marker .offset-reset-done prevents re-running

USER PRINCIPLE HONORED:
'อย่า ingest ทีเดียว ตาย ค่อยๆ ทำ' — chunked processing remains
'ไม่ต้องใหญ่ เดี๋ยวตาย' — public dataset bypasses size cap

bin/agentic-crawler.sh

@@ -157,23 +157,23 @@ for link in links:
 con.commit()
 print(f"  [ok {status}] {title[:60]} ← {url[:60]} (+{added} new links)")
 
-# Save fetched page as training pair (page → summary) — summarize via local LLM later
-# For now just log raw page metadata
+# Save fetched page metadata to a SEPARATE crawl log — NOT to training-pairs.jsonl.
+# (Placeholder responses pollute training data; only insert when we have real summary.)
+crawl_log = os.path.expanduser("~/.surrogate/state/agentic-crawl-raw.jsonl")
 text_only = re.sub(r"<[^>]+>", " ", body)
 text_only = re.sub(r"\s+", " ", text_only).strip()[:6000]
 if len(text_only) > 200:
-    pair = {
+    raw_record = {
         "ts": time.time(),
         "source": "agentic-crawler",
         "url": url,
         "title": title,
         "domain": domain,
         "depth": depth,
-        "prompt": f"Summarize this page from {domain} (title: {title}):\n\n{text_only[:3000]}",
-        "response": f"[crawled {time.strftime('%Y-%m-%d %H:%M')} — auto-summary pending]",
+        "text": text_only[:6000],
     }
-    with open(pairs, "a") as f:
-        f.write(json.dumps(pair, ensure_ascii=False) + "\n")
+    with open(crawl_log, "a") as f:
+        f.write(json.dumps(raw_record, ensure_ascii=False) + "\n")
 PYEOF
 }
 

start.sh

@@ -52,6 +52,17 @@ if [[ -d "$DATA" ]] && [[ -w "$DATA" ]]; then
         ln -sfn "${DATA}/training-pairs.jsonl" "${HOME}/.surrogate/training-pairs.jsonl"
     fi
 
+    # ── One-time offset reset: skip polluted agentic-crawler placeholder backlog ──
+    # Up to 2026-04-28 the crawler wrote ~35K placeholder pairs ("auto-summary pending").
+    # Those aren't trainable. Reset push offset to current line count to bypass them.
+    if [[ ! -f "${HOME}/.surrogate/.offset-reset-done" ]] && [[ -f "${HOME}/.surrogate/training-pairs.jsonl" ]]; then
+        CUR=$(wc -l < "${HOME}/.surrogate/training-pairs.jsonl" | tr -d ' ')
+        echo "$CUR" > "${HOME}/.surrogate/.training-push-offset"
+        echo "$CUR" > "${HOME}/.surrogate/.self-ingest-offset"
+        touch "${HOME}/.surrogate/.offset-reset-done"
+        echo "[$(date +%H:%M:%S)] one-time offset reset → $CUR (skip placeholder backlog)" >> "$LOG_DIR/boot.log"
+    fi
+
     echo "[$(date +%H:%M:%S)] persistent /data linked (state, logs, memory, skills, sessions, workspace, ollama, training-pairs)" >> "$LOG_DIR/boot.log"
 else
     echo "[$(date +%H:%M:%S)] WARN: /data not writable — running ephemeral!" >> "$LOG_DIR/boot.log"