fix: robust gateway restart — force-kill residual processes before restart

ROOT CAUSE: When gateway crashes and watchdog restarts it, the old process
hasn't fully released the Feishu WebSocket lock or port 8642. The new gateway
then fails with:
- 'Another gateway is already using this Feishu app_id (PID xxx)'
- api_server falls back to port 8643 (wrong port)
- entry.py proxy still targets 8642 → complete disconnect

FIX (3 layers):
1. entry.py watchdog: new _kill_all_gateways() function that:
- SIGKILLs ALL hermes gateway processes (not just tracked PID)
- Kills any process holding ports 8642/8643
- Waits for ports to be released
- Cleans ALL lock files including Feishu locks
2. start.sh: force-kill residual gateway processes with kill -9 before
starting fresh, plus clean Feishu lock files
3. entry.py watchdog: increased initial settle time to 15s

entry.py

@@ -666,15 +666,81 @@ def _gateway_watchdog(interval: int = 30):
             pass
         return None
 
-    def _start_gateway():
-        """Start the gateway process."""
-        # Clean stale state
-        for f in (pid_file, lock_file, takeover_file):
+    def _kill_all_gateways():
+        """Force-kill ALL existing gateway processes and wait for ports to free.
+
+        This prevents the 'another gateway is already using this Feishu app_id'
+        error and port conflicts when restarting after a crash.
+        """
+        killed = []
+        # Find ALL hermes gateway processes (not just the one we track)
+        try:
+            for proc in psutil.process_iter(["pid", "cmdline", "create_time"]):
+                try:
+                    cmdline = " ".join(proc.info["cmdline"] or [])
+                    if "hermes_cli.main" in cmdline and "gateway" in cmdline:
+                        pid = proc.info["pid"]
+                        logger.info("[Watchdog] Force-killing residual gateway PID=%d", pid)
+                        proc.kill()  # SIGKILL — no graceful shutdown, just die
+                        killed.append(pid)
+                except (psutil.NoSuchProcess, psutil.AccessDenied):
+                    pass
+        except Exception as e:
+            logger.warning("[Watchdog] Error scanning processes: %s", e)
+
+        # Also kill any lingering aiohttp servers on port 8642/8643
+        for port in (8642, 8643):
+            try:
+                for conn in psutil.net_connections(kind='inet'):
+                    if conn.laddr.port == port and conn.status == 'LISTEN':
+                        try:
+                            owner = psutil.Process(conn.pid)
+                            if conn.pid not in killed:
+                                logger.info("[Watchdog] Force-killing process %d on port %d", conn.pid, port)
+                                owner.kill()
+                        except (psutil.NoSuchProcess, psutil.AccessDenied):
+                            pass
+            except Exception:
+                pass
+
+        # Wait for processes to actually die and ports to be released
+        if killed:
+            time.sleep(3)
+            for pid in killed:
+                try:
+                    p = psutil.Process(pid)
+                    if p.is_running():
+                        logger.warning("[Watchdog] PID %d still running after kill, waiting...", pid)
+                        time.sleep(5)
+                except psutil.NoSuchProcess:
+                    pass
+
+        # Clean ALL stale state files
+        for f in (pid_file, alt_pid_file, lock_file, takeover_file):
             try:
                 os.remove(f)
             except Exception:
                 pass
 
+        # Also clean any .feishu_lock or similar files
+        for pattern in ("feishu*.lock", "*.feishu_lock"):
+            import glob as _glob
+            for lock_f in _glob.glob(os.path.join(HERMES_HOME, pattern)):
+                try:
+                    os.remove(lock_f)
+                    logger.info("[Watchdog] Removed stale lock: %s", lock_f)
+                except Exception:
+                    pass
+
+        return len(killed)
+
+    def _start_gateway():
+        """Start the gateway process with full cleanup of any residual state."""
+        # Force-kill ALL residual gateway processes first
+        killed = _kill_all_gateways()
+        if killed:
+            logger.info("[Watchdog] Killed %d residual gateway process(es) before restart", killed)
+
         log_fh = None
         try:
             log_fh = open(gw_log, "a")

start.sh

@@ -52,12 +52,24 @@ if [ ! -L "$WEBUI_HOME" ]; then
     echo "Symlink: hermes-web-ui -> $WEBUI_DATA"
 fi
 
+# ── Force-kill any residual gateway processes from previous crash ──
+echo "Cleaning up residual gateway processes..."
+# Kill any lingering hermes gateway processes (prevents Feishu lock conflict)
+for pid in $(pgrep -f "hermes_cli.main.*gateway" 2>/dev/null); do
+    echo "  Killing residual gateway PID=$pid"
+    kill -9 "$pid" 2>/dev/null
+done
+sleep 2  # Wait for processes and ports to be fully released
+
 # ── Clean up stale PID/lock files from previous crash ──
 echo "Cleaning up stale state..."
 rm -f "$HERMES_HOME/gateway.pid" 2>/dev/null
 rm -f "$HERMES_HOME/.gateway_runtime_lock" 2>/dev/null
 rm -f "$HERMES_HOME/.gateway_takeover" 2>/dev/null
 rm -f /tmp/hermes-gateway.pid 2>/dev/null
+# Clean Feishu lock files
+rm -f "$HERMES_HOME"/feishu*.lock 2>/dev/null
+rm -f "$HERMES_HOME"/*.feishu_lock 2>/dev/null
 echo "Stale state cleaned."
 
 # Initialize MemPalace if not already