Spaces:
Runtime error
feat: CONTINUOUS auto-orchestrate (4 parallel workers, no 20-min gaps)
Browse filesUSER: 'ΰΈΰΈ±ΰΈΰΈΰΈ΅ΰΉ ΰΈΰΈΰΈΰΉΰΈ«ΰΉ dev ΰΈΰΈ₯ΰΈΰΈΰΉΰΈ§ΰΈ₯ΰΈ² ΰΈΰΈ³ΰΉΰΈ‘ΰΈΰΉΰΈΰΈΰΉΰΈΰΈ£ΰΈ 1 AM WTF ΰΈ₯ΰΈ·ΰΈ‘ context ΰΉΰΈ«ΰΈ£ΰΈΰΈΰΈ²ΰΈ’'
CLARIFICATION: '1AM' was just my measurement-window timestamp, not a wait gate.
Dev was ALWAYS running since boot, but on M%20 cron = once every 20 min.
That's 72 fires/day, far less than 'continuous'.
USER REQUEST: dev should run NON-STOP. Fixing now.
NEW: bin/auto-orchestrate-continuous.sh
- Spawns N=4 parallel workers (configurable via ORCHESTRATE_WORKERS env)
- Each worker loops FOREVER:
1. Pick TODO/FIXME via existing auto-orchestrate-loop.sh
2. Run 6-stage pipeline (SA \u2192 architect \u2192 qa \u2192 dev \u2192 verify \u2192 review)
3. APPROVE \u2192 commit + push to GitHub
4. Sleep 10s
5. Next iteration
- Resource guard: pause only if load >80 (was 8 \u2014 too aggressive)
- Workers stagger 3s startup \u2192 don't all hit same TODO
- Existing LOCK_DIR per-task-hash dedup prevents same TODO 2\u00d7
THROUGHPUT:
- Before (cron M%20): 72 cycles/day, 1 at a time = 72 orchestrate runs/day
- After (continuous): 4 workers \u00d7 (3600s/300s avg cycle) \u00d7 24h = ~1100 runs/day
- = ~15\u00d7 more dev work, axentx commits proportionally up
start.sh changes:
- Boot: spawn auto-orchestrate-continuous immediately (alongside other daemons)
- Cron M%20 entry kept as failsafe (skips if continuous already running, via pgrep guard)
Status server: added 'dataset-enrich' and 'auto-orchestrate-continuous' to /logs allowlist
Plus the boot-time enrich kickoff from previous push will fire ON THIS REBUILD
\u2014 96 datasets pull starts immediately, no 60-min cron wait.
- bin/auto-orchestrate-continuous.sh +53 -0
- bin/hermes-status-server.py +1 -1
- start.sh +10 -5
|
@@ -0,0 +1,53 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
#!/usr/bin/env bash
|
| 2 |
+
# Continuous auto-orchestrate worker β replaces cron-fire-every-20min model.
|
| 3 |
+
#
|
| 4 |
+
# Spawns N parallel workers. Each loops forever:
|
| 5 |
+
# pick TODO from random axentx repo β orchestrate pipeline β commit+push if APPROVE
|
| 6 |
+
# β cool-down 5s β next iteration
|
| 7 |
+
#
|
| 8 |
+
# Avoids 'all hit same TODO' race via existing LOCK_DIR per-task hash.
|
| 9 |
+
# Resource guard: only pause if load > 80 (much higher tolerance vs old M%20 fire).
|
| 10 |
+
set -uo pipefail
|
| 11 |
+
set -a; source "$HOME/.hermes/.env" 2>/dev/null; set +a
|
| 12 |
+
|
| 13 |
+
LOG="$HOME/.surrogate/logs/auto-orchestrate-continuous.log"
|
| 14 |
+
mkdir -p "$(dirname "$LOG")"
|
| 15 |
+
|
| 16 |
+
PARALLEL_WORKERS="${ORCHESTRATE_WORKERS:-4}"
|
| 17 |
+
WORKER_COOLDOWN="${WORKER_COOLDOWN:-10}" # seconds between iterations per worker
|
| 18 |
+
|
| 19 |
+
echo "[$(date +%H:%M:%S)] continuous orchestrate start (workers=$PARALLEL_WORKERS, cooldown=${WORKER_COOLDOWN}s)" | tee -a "$LOG"
|
| 20 |
+
|
| 21 |
+
worker_loop() {
|
| 22 |
+
local worker_id="$1"
|
| 23 |
+
local iter=0
|
| 24 |
+
while true; do
|
| 25 |
+
iter=$((iter + 1))
|
| 26 |
+
echo "[$(date +%H:%M:%S)] worker-$worker_id iter=$iter starting" >> "$LOG"
|
| 27 |
+
|
| 28 |
+
# Resource guard β much more lenient than old M%20 cron
|
| 29 |
+
local load
|
| 30 |
+
load=$(uptime | sed -E 's/.*load average[s]?:[[:space:]]*//' | awk -F',' '{print int($1)}')
|
| 31 |
+
load=${load:-0}
|
| 32 |
+
if [[ $load -gt 80 ]]; then
|
| 33 |
+
echo "[$(date +%H:%M:%S)] worker-$worker_id pause: load=$load > 80" >> "$LOG"
|
| 34 |
+
sleep 60
|
| 35 |
+
continue
|
| 36 |
+
fi
|
| 37 |
+
|
| 38 |
+
# Run single orchestrate cycle (existing script does TODO pick + run + push)
|
| 39 |
+
bash "$HOME/.surrogate/bin/auto-orchestrate-loop.sh" >> "$LOG" 2>&1
|
| 40 |
+
local rc=$?
|
| 41 |
+
echo "[$(date +%H:%M:%S)] worker-$worker_id iter=$iter done rc=$rc" >> "$LOG"
|
| 42 |
+
|
| 43 |
+
# Brief cooldown β workers stagger naturally
|
| 44 |
+
sleep "$WORKER_COOLDOWN"
|
| 45 |
+
done
|
| 46 |
+
}
|
| 47 |
+
|
| 48 |
+
# Spawn N workers in parallel
|
| 49 |
+
for i in $(seq 1 "$PARALLEL_WORKERS"); do
|
| 50 |
+
worker_loop "$i" &
|
| 51 |
+
sleep 3 # stagger startup
|
| 52 |
+
done
|
| 53 |
+
wait
|
|
@@ -166,7 +166,7 @@ def log_tail(name: str, lines: int = 100) -> PlainTextResponse:
|
|
| 166 |
"auto-orchestrate-loop", "training-push", "ollama", "discord-bot",
|
| 167 |
"hermes-discord-bot", "surrogate-research-loop", "surrogate-research-apply",
|
| 168 |
"surrogate-dev-loop", "domain-scrape-loop", "github-domain-scrape",
|
| 169 |
-
"qwen-coder", "git-clone", "git-pull", "redis", "hf-dataset-discoverer", "dedup-bootstrap", "github-agentic-crawler", "ollama-pull-granite", "synthetic-data", "self-ingest", "scrape-sre-postmortems", "refresh-cve-feed",
|
| 170 |
"ollama-pull-coder", "ollama-pull-devstral", "ollama-pull-fallback",
|
| 171 |
"ollama-pull-yicoder", "ollama-pull-embed", "ollama-pull-light",
|
| 172 |
}
|
|
|
|
| 166 |
"auto-orchestrate-loop", "training-push", "ollama", "discord-bot",
|
| 167 |
"hermes-discord-bot", "surrogate-research-loop", "surrogate-research-apply",
|
| 168 |
"surrogate-dev-loop", "domain-scrape-loop", "github-domain-scrape",
|
| 169 |
+
"qwen-coder", "git-clone", "git-pull", "redis", "auto-orchestrate-continuous", "dataset-enrich", "hf-dataset-discoverer", "dedup-bootstrap", "github-agentic-crawler", "ollama-pull-granite", "synthetic-data", "self-ingest", "scrape-sre-postmortems", "refresh-cve-feed",
|
| 170 |
"ollama-pull-coder", "ollama-pull-devstral", "ollama-pull-fallback",
|
| 171 |
"ollama-pull-yicoder", "ollama-pull-embed", "ollama-pull-light",
|
| 172 |
}
|
|
@@ -234,12 +234,16 @@ nohup bash ~/.surrogate/bin/github-agentic-crawler.sh > "$LOG_DIR/github-agentic
|
|
| 234 |
echo "[$(date +%H:%M:%S)] github-agentic-crawler started (token pool maximized)" >> "$LOG_DIR/boot.log"
|
| 235 |
|
| 236 |
# ββ 7b3. HF Dataset Discoverer (continuous mega-mix hunt) βββββββββββββββββββ
|
| 237 |
-
# Searches HF Hub across 70+ topic queries every 30 min. Filters license + scores
|
| 238 |
-
# quality. Auto-adds high-confidence permissive picks to dynamic-datasets.json.
|
| 239 |
-
# dataset-enrich reads dynamic list on top of static 89 β infinitely growing corpus.
|
| 240 |
nohup bash ~/.surrogate/bin/hf-dataset-discoverer.sh > "$LOG_DIR/hf-dataset-discoverer.log" 2>&1 &
|
| 241 |
echo "[$(date +%H:%M:%S)] hf-dataset-discoverer started (continuous mega-mix hunt)" >> "$LOG_DIR/boot.log"
|
| 242 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 243 |
# ββ 7c. Skill-synthesis daemon (extract patterns from cloned repos β skills) β
|
| 244 |
nohup bash ~/.surrogate/bin/skill-synthesis-daemon.sh > "$LOG_DIR/skill-synthesis.log" 2>&1 &
|
| 245 |
echo "[$(date +%H:%M:%S)] skill-synthesis daemon started" >> "$LOG_DIR/boot.log"
|
|
@@ -258,8 +262,9 @@ while true; do
|
|
| 258 |
[[ $((M % 5)) -eq 0 ]] && bash ~/.surrogate/bin/work-queue-producer.sh >> "$LOG" 2>&1 &
|
| 259 |
# Every 3 min: training-pair push to HF (drains ~/.surrogate/training-pairs.jsonl)
|
| 260 |
[[ $((M % 3)) -eq 0 ]] && bash ~/.surrogate/bin/push-training-to-hf.sh >> "$LOG" 2>&1 &
|
| 261 |
-
#
|
| 262 |
-
|
|
|
|
| 263 |
# Every 30 min: research-apply (pop queue β orchestrate β ship feature)
|
| 264 |
[[ $((M % 30)) -eq 15 ]] && bash ~/.surrogate/bin/surrogate-research-apply.sh >> "$LOG" 2>&1 &
|
| 265 |
# Every 60 min: keyword tuner (adapts scrape queue based on yields)
|
|
|
|
| 234 |
echo "[$(date +%H:%M:%S)] github-agentic-crawler started (token pool maximized)" >> "$LOG_DIR/boot.log"
|
| 235 |
|
| 236 |
# ββ 7b3. HF Dataset Discoverer (continuous mega-mix hunt) βββββββββββββββββββ
|
|
|
|
|
|
|
|
|
|
| 237 |
nohup bash ~/.surrogate/bin/hf-dataset-discoverer.sh > "$LOG_DIR/hf-dataset-discoverer.log" 2>&1 &
|
| 238 |
echo "[$(date +%H:%M:%S)] hf-dataset-discoverer started (continuous mega-mix hunt)" >> "$LOG_DIR/boot.log"
|
| 239 |
|
| 240 |
+
# ββ 7e. CONTINUOUS auto-orchestrate (4 parallel workers, no cron gap) βββββββ
|
| 241 |
+
# Replaces M%20 cron β was 'fire once every 20 min'. Now: each of 4 workers
|
| 242 |
+
# loops forever, dev work happens nonstop. Picks different TODO/FIXME each iter,
|
| 243 |
+
# uses existing LOCK_DIR for dedup. Result: ~10-20Γ more orchestrate cycles/day.
|
| 244 |
+
nohup bash ~/.surrogate/bin/auto-orchestrate-continuous.sh > "$LOG_DIR/auto-orchestrate-continuous.log" 2>&1 &
|
| 245 |
+
echo "[$(date +%H:%M:%S)] auto-orchestrate-continuous started (4 parallel workers, never sleeps)" >> "$LOG_DIR/boot.log"
|
| 246 |
+
|
| 247 |
# ββ 7c. Skill-synthesis daemon (extract patterns from cloned repos β skills) β
|
| 248 |
nohup bash ~/.surrogate/bin/skill-synthesis-daemon.sh > "$LOG_DIR/skill-synthesis.log" 2>&1 &
|
| 249 |
echo "[$(date +%H:%M:%S)] skill-synthesis daemon started" >> "$LOG_DIR/boot.log"
|
|
|
|
| 262 |
[[ $((M % 5)) -eq 0 ]] && bash ~/.surrogate/bin/work-queue-producer.sh >> "$LOG" 2>&1 &
|
| 263 |
# Every 3 min: training-pair push to HF (drains ~/.surrogate/training-pairs.jsonl)
|
| 264 |
[[ $((M % 3)) -eq 0 ]] && bash ~/.surrogate/bin/push-training-to-hf.sh >> "$LOG" 2>&1 &
|
| 265 |
+
# auto-orchestrate now runs CONTINUOUSLY (4 parallel workers) β see step 7e below.
|
| 266 |
+
# Cron entry retained for legacy single-fire boost (no harm if continuous already up):
|
| 267 |
+
[[ $((M % 20)) -eq 0 ]] && pgrep -f "auto-orchestrate-continuous" >/dev/null || bash ~/.surrogate/bin/auto-orchestrate-loop.sh >> "$LOG" 2>&1 &
|
| 268 |
# Every 30 min: research-apply (pop queue β orchestrate β ship feature)
|
| 269 |
[[ $((M % 30)) -eq 15 ]] && bash ~/.surrogate/bin/surrogate-research-apply.sh >> "$LOG" 2>&1 &
|
| 270 |
# Every 60 min: keyword tuner (adapts scrape queue based on yields)
|