Set LLM_API_KEY fallback default to enabled

.env.example

@@ -132,7 +132,10 @@ LLM_MODEL=anthropic/claude-sonnet-4-5
 # have multiple accounts or want to spread rate-limit quota.
 #
 # Pattern: <PROVIDER>_API_KEYS=key1,key2,key3
-# Fallback order: plural pool → singular key → LLM_API_KEY
+# Fallback order: plural pool → singular key → LLM_API_KEY (optional)
+# Set false only if you want to disable global LLM_API_KEY fallback
+# across providers.
+LLM_API_KEY_FALLBACK_ENABLED=true
 #
 # Uncomment and fill in only the providers you use:
 #

CHANGELOG.md

@@ -2,6 +2,44 @@
 
 All notable changes to this project will be documented in this file.
 
+## [1.5.0] - 2026-05-13
+
+### Added
+
+- **NVIDIA key-rotation support** — added `nvidia-key-rotator.cjs` wiring and startup integration so deployments can rotate NVIDIA credentials similarly to other provider key-rotation flows.
+- **Cloudflare keep-alive automation** — added/expanded `cloudflare-keepalive-setup.py` flow and startup wiring to provision keep-alive through Cloudflare Worker automation instead of the older UptimeRobot-first approach.
+- **Sync metadata marker model** — introduced a structured workspace marker `(file_count, total_size, newest_mtime, metadata_hash)` to support stronger change introspection in sync code.
+
+### Changed
+
+- **Workspace sync script rename finalized** — `workspace-sync.py` flow was migrated to `openclaw-sync.py` in Docker/startup/docs so restore/sync behavior is centralized under one script.
+- **Sync trigger behavior hardened for config churn** — OpenClaw config sync now debounces until JSON settles before immediate sync, reducing false/partial syncs during rapid config writes.
+- **Gateway restart flow now saves state first** — restart path was updated to run a pre-restart one-shot state sync so gateway reloads are less likely to drop recent state.
+- **Shutdown backup now uses a two-step pass** — graceful shutdown now attempts `sync-once-settled` then a final `sync-once` pass to better capture last-second writes.
+- **Telegram allowlist simplified** — consolidated Telegram allowlist into `TELEGRAM_ALLOWED_USERS` and aligned docs/examples.
+- **Plugin startup behavior aligned** — startup-installed plugins are synced into `plugins.allow` before gateway launch so runtime-installed plugins are recognized cleanly.
+- **Cloudflare proxy path matured** — multiple iterations improved fetch/proxy behavior (header handling, endpoint scoping, API root routing, URL parsing, and logging noise reduction), then simplified unstable undici patching paths.
+- **Health dashboard polish** — sync timestamps now show local time, footer credits were corrected, and status rendering/docs were updated for the Cloudflare keep-alive model.
+- **CI workflow churn documented** — GitHub workflow files for HF sync were added/renamed/cleaned multiple times as space/repo naming stabilized.
+
+### Fixed
+
+- **Missed rapid backup updates** — sync logic now relies on content fingerprint checks for no-op decisions so same-second or quick successive changes are less likely to be skipped.
+- **Non-deterministic metadata hashing** — metadata hashing now iterates paths deterministically to avoid hash jitter from traversal order.
+- **Transient file race sync failures** — sync fingerprinting/snapshot copy paths now tolerate transient `OSError` (file rotated/deleted mid-scan) instead of aborting the whole sync pass.
+- **State restore migration edge cases** — restore flow includes migration/cleanup behavior for legacy hidden state paths and stale backup entries.
+- **Startup/env robustness** — fixed shell export formatting/syntax issues (e.g., NVIDIA/XAI lines) and unbound-variable pitfalls in startup scripts.
+- **Proxy runtime errors and noise** — fixed specific proxy runtime issues (including `UND_ERR_INVALID_ARG`, fetch duplex handling, and upstream error visibility) and reduced noisy stdout logs that interfered with clean process output.
+- **HF workflow/repo reference mismatches** — corrected and later cleaned workflow repository references during repo migration/restructure.
+
+### Docs
+
+- README/.env/security docs were refreshed across multiple commits to reflect:
+  - Cloudflare keep-alive replacing UptimeRobot setup path,
+  - updated secrets and startup environment behavior,
+  - provider/key-rotation options,
+  - backup/sync behavior and troubleshooting guidance.
+
 ## [1.4.0] - 2026-04-25
 
 ### Added

README.md

@@ -250,10 +250,10 @@ GEMINI_API_KEYS=AIza-key1,AIza-key2
 **Fallback chain** (per provider):
 1. `{PROVIDER}_API_KEYS` — comma-separated pool *(preferred)*
 2. `{PROVIDER}_API_KEY` — single dedicated key
-3. `LLM_API_KEY` — universal fallback *(default for all providers)*
+3. `LLM_API_KEY` — universal fallback *(enabled by default; disable with `LLM_API_KEY_FALLBACK_ENABLED=false`)*
 
 > [!TIP]
-> If you only set `LLM_API_KEY`, all providers use it as a fallback automatically — no extra config needed. Add per-provider pools only when you need multi-key rotation.
+> By default, `LLM_API_KEY` fallback is enabled for compatibility. Set `LLM_API_KEY_FALLBACK_ENABLED=false` if you want strict provider-only activation.
 
 Supported per-provider variables: `ANTHROPIC_API_KEYS`, `OPENAI_API_KEYS`, `GEMINI_API_KEYS`, `DEEPSEEK_API_KEYS`, `GROQ_API_KEYS`, `MISTRAL_API_KEYS`, `OPENROUTER_API_KEYS`, `XAI_API_KEYS`, `NVIDIA_API_KEYS`, `COHERE_API_KEYS`, `TOGETHER_API_KEYS`, `CEREBRAS_API_KEYS`, and more — see `.env.example` for the full list.
 

multi-provider-key-rotator.cjs

@@ -8,7 +8,7 @@
  *
  * For each provider you can supply a comma-separated pool:
  *   ANTHROPIC_API_KEYS=key1,key2,key3
- * Falls back to the singular env var, then to LLM_API_KEY.
+ * Falls back to the singular env var, and optionally to LLM_API_KEY.
  *
  * Keys are rotated round-robin per provider independently.
  *
@@ -30,7 +30,9 @@ const log = (...args) => console.error(...args);
 // envPlural – env var that holds a comma-separated key pool  (preferred)
 // envSingular – env var that holds a single key              (fallback)
 //
-// LLM_API_KEY is the final fallback for every provider.
+// LLM_API_KEY fallback can be controlled via:
+//   LLM_API_KEY_FALLBACK_ENABLED=true|false
+// Default is enabled for backwards compatibility.
 //
 const PROVIDERS = [
   {
@@ -180,6 +182,8 @@ function normalizeKeys(...inputs) {
 
 // Build per-provider key pools + rotation indices
 const providerState = PROVIDERS.map(p => {
+  const llmFallbackRaw = String(process.env.LLM_API_KEY_FALLBACK_ENABLED || '').trim().toLowerCase();
+  const llmFallbackEnabled = !/^(0|false|no|off)$/.test(llmFallbackRaw);
   const dedicatedKeys = normalizeKeys(
     process.env[p.envPlural]  || '',
     process.env[p.envSingular] || '',
@@ -187,7 +191,11 @@ const providerState = PROVIDERS.map(p => {
   const hasDedicated = dedicatedKeys.length > 0;
   const keys = hasDedicated
     ? dedicatedKeys
-    : normalizeKeys(process.env.LLM_API_KEY || '');
+    : (
+      llmFallbackEnabled
+        ? normalizeKeys(process.env.LLM_API_KEY || '')
+        : []
+    );
 
   if (hasDedicated) {
     log(`[key-rotator] ${p.name}: ${keys.length} key${keys.length === 1 ? '' : 's'}`);
@@ -208,6 +216,8 @@ const fallbackCount = providerState.filter(p => {
 }).length;
 if (fallbackCount > 0) {
   log(`[key-rotator] ${fallbackCount} provider(s) using LLM_API_KEY fallback`);
+} else if (process.env.LLM_API_KEY && /^(0|false|no|off)$/i.test(String(process.env.LLM_API_KEY_FALLBACK_ENABLED || ''))) {
+  log('[key-rotator] LLM_API_KEY fallback disabled (set LLM_API_KEY_FALLBACK_ENABLED=true to re-enable)');
 }
 
 // ─── Runtime helpers ─────────────────────────────────────────────────────────

openclaw-sync.py

@@ -18,6 +18,7 @@ import sys
 import tempfile
 import threading
 import time
+from typing import TypeAlias
 from pathlib import Path
 
 os.environ.setdefault("HF_HUB_DISABLE_PROGRESS_BARS", "1")
@@ -69,6 +70,7 @@ EXCLUDED_STATE_NAMES = {
     "openclaw-app",
     "gateway.log",
     "browser",
+    "npm",
 }
 WHATSAPP_CREDS_DIR = OPENCLAW_HOME / "credentials" / "whatsapp" / "default"
 WHATSAPP_BACKUP_DIR = STATE_DIR / "credentials" / "whatsapp" / "default"
@@ -76,6 +78,7 @@ RESET_MARKER = WORKSPACE / ".reset_credentials"
 HF_API = HfApi(token=HF_TOKEN) if HF_TOKEN else None
 STOP_EVENT = threading.Event()
 _REPO_ID_CACHE: str | None = None
+WorkspaceMarker: TypeAlias = tuple[int, int, int, str]
 
 
 def write_status(status: str, message: str) -> None:
@@ -280,14 +283,15 @@ def file_marker(path: Path) -> tuple[int, int, int]:
     return (1, int(stat.st_size), int(stat.st_mtime_ns))
 
 
-def metadata_marker(root: Path) -> tuple[int, int, int]:
+def metadata_marker(root: Path) -> WorkspaceMarker:
     if not root.exists():
-        return (0, 0, 0)
+        return (0, 0, 0, "")
 
     file_count = 0
     total_size = 0
     newest_mtime = 0
-    for path in root.rglob("*"):
+    metadata_hasher = hashlib.sha256()
+    for path in sorted(root.rglob("*")):
         if not path.is_file():
             continue
         rel = path.relative_to(root).as_posix()
@@ -298,9 +302,17 @@ def metadata_marker(root: Path) -> tuple[int, int, int]:
         except OSError:
             continue
         file_count += 1
-        total_size += int(stat.st_size)
-        newest_mtime = max(newest_mtime, int(stat.st_mtime_ns))
-    return (file_count, total_size, newest_mtime)
+        size = int(stat.st_size)
+        mtime_ns = int(stat.st_mtime_ns)
+        total_size += size
+        newest_mtime = max(newest_mtime, mtime_ns)
+        metadata_hasher.update(rel.encode("utf-8"))
+        metadata_hasher.update(b"\0")
+        metadata_hasher.update(str(size).encode("ascii"))
+        metadata_hasher.update(b"\0")
+        metadata_hasher.update(str(mtime_ns).encode("ascii"))
+        metadata_hasher.update(b"\0")
+    return (file_count, total_size, newest_mtime, metadata_hasher.hexdigest())
 
 
 def fingerprint_dir(root: Path) -> str:
@@ -313,9 +325,16 @@ def fingerprint_dir(root: Path) -> str:
         if _should_exclude(rel, path):
             continue
         hasher.update(rel.encode("utf-8"))
-        with path.open("rb") as handle:
-            for chunk in iter(lambda: handle.read(1024 * 1024), b""):
-                hasher.update(chunk)
+        try:
+            with path.open("rb") as handle:
+                for chunk in iter(lambda: handle.read(1024 * 1024), b""):
+                    hasher.update(chunk)
+        except (FileNotFoundError, IsADirectoryError, NotADirectoryError):
+            # Fingerprint must represent a complete view of the workspace.
+            # Retry next sync pass instead of silently hashing a partial tree.
+            raise RuntimeError(
+                f"Workspace changed while hashing {rel}; retrying next sync pass."
+            )
     return hasher.hexdigest()
 
 
@@ -331,7 +350,13 @@ def create_snapshot_dir(source_root: Path) -> Path:
             target.mkdir(parents=True, exist_ok=True)
             continue
         target.parent.mkdir(parents=True, exist_ok=True)
-        shutil.copy2(path, target)
+        try:
+            shutil.copy2(path, target)
+        except (FileNotFoundError, IsADirectoryError, NotADirectoryError):
+            # Do not upload a partial snapshot; let caller retry on next loop.
+            raise RuntimeError(
+                f"Snapshot changed while copying {rel_posix}; retrying next sync pass."
+            )
     return staging_root
 
 
@@ -396,16 +421,15 @@ def restore_workspace() -> bool:
 
 def _sync_once_unlocked(
     last_fingerprint: str | None = None,
-    last_marker: tuple[int, int, int] | None = None,
-) -> tuple[str, tuple[int, int, int]]:
+    last_marker: WorkspaceMarker | None = None,
+) -> tuple[str, WorkspaceMarker]:
     if not HF_TOKEN:
         write_status("disabled", "HF_TOKEN is not configured.")
-        return (last_fingerprint or "", last_marker or (0, 0, 0))
+        return (last_fingerprint or "", last_marker or (0, 0, 0, ""))
 
     snapshot_state_into_workspace()
     repo_id = ensure_repo_exists()
     current_marker = metadata_marker(WORKSPACE)
-
     if last_marker is not None and current_marker == last_marker:
         write_status("synced", "No workspace changes detected.")
         return (last_fingerprint or "", current_marker)
@@ -444,8 +468,8 @@ def _sync_once_unlocked(
 
 def sync_once(
     last_fingerprint: str | None = None,
-    last_marker: tuple[int, int, int] | None = None,
-) -> tuple[str, tuple[int, int, int]]:
+    last_marker: WorkspaceMarker | None = None,
+) -> tuple[str, WorkspaceMarker]:
     SYNC_LOCK_FILE.parent.mkdir(parents=True, exist_ok=True)
     with SYNC_LOCK_FILE.open("w", encoding="utf-8") as lock_handle:
         fcntl.flock(lock_handle, fcntl.LOCK_EX)

start.sh

@@ -540,10 +540,15 @@ fi
 # ── Trap SIGTERM for graceful shutdown ──
 graceful_shutdown() {
   echo "Shutting down..."
-  if [ -f "/home/node/app/openclaw-sync.py" ]; then
+  if [ -f "/home/node/app/openclaw-sync.py" ] && [ -n "${HF_TOKEN:-}" ]; then
     echo "Saving state before exit..."
+    timeout 8s python3 /home/node/app/openclaw-sync.py sync-once-settled || \
+      echo "Warning: could not complete settled shutdown sync"
+    sleep 1
     python3 /home/node/app/openclaw-sync.py sync-once || \
-      echo "Warning: could not complete shutdown sync"
+      echo "Warning: could not complete final shutdown sync"
+  elif [ -f "/home/node/app/openclaw-sync.py" ]; then
+    echo "HF_TOKEN not set; skipping shutdown backup sync."
   fi
   kill $(jobs -p) 2>/dev/null
   exit 0