probe: docker sdk bucket mount test (UID 1000 per HF docs)

Dockerfile

@@ -0,0 +1,31 @@
+# Follows https://huggingface.co/docs/hub/spaces-sdks-docker#permissions exactly.
+# The ONLY change from the official example is the final CMD.
+FROM python:3.12-slim
+
+# Set up a new user named "user" with user ID 1000
+RUN useradd -m -u 1000 user
+
+# Switch to the "user" user
+USER user
+
+# Set home to the user's home directory
+ENV HOME=/home/user \
+    PATH=/home/user/.local/bin:$PATH
+
+# Set the working directory to the user's home directory
+WORKDIR $HOME/app
+
+# Copy the current directory contents into the container at $HOME/app
+# setting the owner to the user
+COPY --chown=user . $HOME/app
+
+# Try and chown /data so it's writable by the user. This is EXACTLY what
+# the docs suggest for writable runtime directories. It will be overridden
+# at runtime when the bucket mount lands on top — that's the bug we're
+# demonstrating.
+USER root
+RUN mkdir -p /data && chown -R user:user /data && chmod -R 777 /data
+USER user
+
+EXPOSE 7860
+CMD ["python", "-u", "app.py"]

README.md

@@ -1,10 +1,80 @@
 ---
-title: Bucket Sqlite Probe Docker
-emoji: 🌍
-colorFrom: indigo
-colorTo: yellow
+title: bucket-sqlite-probe-docker
+emoji: 🐳
+colorFrom: red
+colorTo: gray
 sdk: docker
+app_port: 7860
 pinned: false
+tags:
+ - bucket
+ - sqlite
+ - probe
+ - reference
 ---
 
-Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference
+# Bucket mount × SQLite probe — Docker SDK Space (UID 1000)
+
+**This is the failing half of a matched pair.** The Gradio SDK half is at
+[`davanstrien/bucket-sqlite-probe-gradio`](https://huggingface.co/spaces/davanstrien/bucket-sqlite-probe-gradio) and all its probes pass. This Space runs the *same probe code* against the *same bucket*, but inside a Docker SDK container that follows the official [`spaces-sdks-docker#permissions`](https://huggingface.co/docs/hub/spaces-sdks-docker#permissions) guidance — creating a `user` account with `uid=1000` and switching to it via `USER user` in the Dockerfile. Its write probes fail with `Permission denied` / `unable to open database file`.
+
+The Dockerfile is intentionally as close to the official permissions example as possible — the only deviations are the `CMD` (runs `app.py`) and an explicit `chown/chmod 777 /data` step to prove that build-time permissions are overridden by the runtime mount.
+
+## What it demonstrates
+
+With bucket `davanstrien/search-v2-chroma` attached R/W at `/data`:
+
+- The container runs as `uid=1000(user)`, exactly as `spaces-sdks-docker#permissions` recommends.
+- `/data` is mounted by `hf-mount` with `idmapped,user_id=0,group_id=0,default_permissions`. That id-mapping pins the mount's writable UID to **0 (root)**, not the container's UID 1000.
+- `ls -lan /data` shows `drwxr-xr-x 3 65534 65534` (nobody:nogroup, mode 755). The `chmod 777` in the Dockerfile is silently overridden — the runtime mount replaces the build-time directory entirely.
+- Write probes all fail:
+  ```
+  FAIL touch /data/docker_probe_touch: PermissionError: [Errno 13] Permission denied: '/data/docker_probe_touch'
+  FAIL sqlite3 connect + CREATE + INSERT: OperationalError: unable to open database file
+  FAIL sqlite3 journal_mode=DELETE: OperationalError: unable to open database file
+  FAIL fcntl.flock LOCK_EX|LOCK_NB: PermissionError: [Errno 13] Permission denied: '/data/docker_probe.lock'
+  ```
+- The control probe (`touch $HOME/app/control_probe` on the container's build-time writable dir) succeeds — so the container is healthy and UID 1000 can write *somewhere*, just not to the bucket mount.
+
+## Why this matters
+
+1. The Docker Spaces docs explicitly tell you to run as UID 1000. There's no note that this is incompatible with Storage Bucket mounts.
+2. The Storage Buckets blog post ([buckets as working layer](https://huggingface.co/blog/davanstrien/buckets-as-working-layer)) implies buckets are a drop-in R/W volume for Spaces.
+3. These two bits of guidance are silently incompatible today because the FUSE mount is provisioned with `user_id=0,group_id=0`. A root container sees the mount as writable; a UID 1000 container does not.
+4. Any SQLite-backed tool (ChromaDB, DuckDB persistent, LMDB, RocksDB) built on a Docker SDK Space following the official permissions guidance will silently fail to open its database on a bucket mount. `huggingface-datasets-search-v2` hit this; [trackio](https://github.com/gradio-app/trackio) doesn't because Gradio SDK Spaces run as root.
+
+## The fix (almost certainly)
+
+The mount provisioning layer should either:
+
+1. Mount with `user_id=1000,group_id=1000` for Docker SDK Spaces (the conventional UID), or
+2. Mount with the Space's runtime UID dynamically (cleanest), or
+3. Chmod the mount root to `0777` at provisioning time (hacky but works for any UID).
+
+All three are infra-side changes. The container user can't fix this — any `chown` / `chmod` in the Dockerfile is overridden when the mount is attached at runtime.
+
+## Reproducing
+
+1. Fork or duplicate this Space.
+2. Create or choose a Storage Bucket you own.
+3. Attach it R/W at `/data` via **Space settings → Volumes** (UI) or via:
+   ```python
+   from huggingface_hub import HfApi, Volume  # requires huggingface_hub >= 1.9.1
+   HfApi().set_space_volumes(
+       "your-namespace/bucket-sqlite-probe-docker",
+       volumes=[Volume(type="bucket", source="your-namespace/your-bucket", mount_path="/data")],
+   )
+   ```
+4. Restart. Probe output appears in startup logs and at the Space URL.
+
+## Related
+
+- Matched Gradio SDK probe Space (passing): https://huggingface.co/spaces/davanstrien/bucket-sqlite-probe-gradio
+- `gradio-app/trackio#465` — trackio's PR switching to the bucket backend (works because Gradio SDK ⇒ root)
+- `huggingface/huggingface_hub#4054` — `set_space_volumes` payload fix (required for this Space to attach the bucket)
+- Docker Spaces permissions docs: https://huggingface.co/docs/hub/spaces-sdks-docker#permissions
+- Blog post: [Buckets as working layer](https://huggingface.co/blog/davanstrien/buckets-as-working-layer)
+
+---
+
+Throwaway investigative Space. Kept public as a reference example. Do not rely on for production.

app.py

@@ -0,0 +1,191 @@
+"""Minimal bucket mount probe for a Docker SDK Space following
+https://huggingface.co/docs/hub/spaces-sdks-docker#permissions (UID 1000).
+
+Runs the probe once at import time so results appear in startup logs,
+then serves the output via a tiny http.server on port 7860 so the Space
+has something to show. No Gradio dependency — keeps the container minimal
+and the probe behavior identical to the Gradio SDK probe Space for an
+apples-to-apples comparison.
+"""
+
+from __future__ import annotations
+
+import http.server
+import os
+import pwd
+import socketserver
+import sqlite3
+import subprocess
+import traceback
+
+try:
+    import fcntl
+except ImportError:
+    fcntl = None
+
+
+def _run_probe() -> str:
+    lines = []
+    lines.append("=" * 60)
+    lines.append("BUCKET MOUNT PROBE — Docker SDK Space (UID 1000)")
+    lines.append("=" * 60)
+
+    try:
+        uid = os.getuid()
+        user = pwd.getpwuid(uid).pw_name
+        lines.append(f"uid={uid} user={user} euid={os.geteuid()} gid={os.getgid()}")
+    except Exception as e:
+        lines.append(f"uid/user lookup failed: {e}")
+
+    lines.append(f"SYSTEM={os.environ.get('SYSTEM')!r}")
+    lines.append(f"SPACE_ID={os.environ.get('SPACE_ID')!r}")
+    lines.append(f"HOME={os.environ.get('HOME')!r}")
+
+    lines.append("")
+    lines.append("--- ls -lan /data ---")
+    try:
+        out = subprocess.run(
+            ["ls", "-lan", "/data"], capture_output=True, text=True, timeout=10
+        )
+        lines.append(out.stdout or "(empty)")
+        if out.stderr:
+            lines.append(f"stderr: {out.stderr}")
+    except Exception as e:
+        lines.append(f"ls failed: {e}")
+
+    lines.append("--- stat /data ---")
+    try:
+        out = subprocess.run(
+            ["stat", "/data"], capture_output=True, text=True, timeout=10
+        )
+        lines.append(out.stdout or "(empty)")
+    except Exception as e:
+        lines.append(f"stat failed: {e}")
+
+    lines.append("--- mount | grep /data ---")
+    try:
+        out = subprocess.run(
+            "mount | grep /data || true",
+            shell=True,
+            capture_output=True,
+            text=True,
+            timeout=10,
+        )
+        lines.append(out.stdout or "(no match)")
+    except Exception as e:
+        lines.append(f"mount failed: {e}")
+
+    lines.append("")
+    lines.append("--- write probes ---")
+
+    def probe(label: str, fn):
+        try:
+            fn()
+            lines.append(f"OK   {label}")
+        except Exception as e:
+            lines.append(f"FAIL {label}: {type(e).__name__}: {e}")
+
+    probe(
+        "touch /data/docker_probe_touch",
+        lambda: open("/data/docker_probe_touch", "w").close(),
+    )
+
+    def _sqlite_create():
+        conn = sqlite3.connect("/data/docker_probe.db", timeout=5.0)
+        conn.execute("CREATE TABLE IF NOT EXISTS t(x INTEGER)")
+        conn.execute("INSERT INTO t VALUES (1)")
+        conn.commit()
+        conn.close()
+
+    probe("sqlite3 connect + CREATE + INSERT", _sqlite_create)
+
+    def _sqlite_delete_journal():
+        conn = sqlite3.connect("/data/docker_probe_delete.db", timeout=5.0)
+        conn.execute("PRAGMA journal_mode = DELETE")
+        conn.execute("CREATE TABLE IF NOT EXISTS t(x INTEGER)")
+        conn.commit()
+        conn.close()
+
+    probe("sqlite3 journal_mode=DELETE", _sqlite_delete_journal)
+
+    if fcntl is not None:
+
+        def _flock():
+            f = open("/data/docker_probe.lock", "w")
+            fcntl.flock(f.fileno(), fcntl.LOCK_EX | fcntl.LOCK_NB)
+            fcntl.flock(f.fileno(), fcntl.LOCK_UN)
+            f.close()
+
+        probe("fcntl.flock LOCK_EX|LOCK_NB", _flock)
+    else:
+        lines.append("SKIP fcntl.flock: fcntl not importable")
+
+    # Control: can we write to $HOME/app (the chown'd build-time dir)?
+    # This should always work — it's the "normal" writable path for UID 1000.
+    lines.append("")
+    lines.append("--- control: write to $HOME/app ---")
+    probe(
+        "touch $HOME/app/control_probe",
+        lambda: open(os.path.expanduser("~/app/control_probe"), "w").close(),
+    )
+
+    lines.append("=" * 60)
+    return "\n".join(lines)
+
+
+try:
+    STARTUP_PROBE = _run_probe()
+except Exception:
+    STARTUP_PROBE = "startup probe crashed:\n" + traceback.format_exc()
+
+print(STARTUP_PROBE, flush=True)
+
+_HTML = """<!doctype html>
+<html><head><meta charset="utf-8"><title>bucket-sqlite-probe-docker</title>
+<style>
+  body {{ font-family: ui-monospace, SFMono-Regular, Menlo, monospace;
+          max-width: 900px; margin: 2em auto; padding: 0 1em;
+          background: #0b0f17; color: #dfe6f0; }}
+  h1 {{ color: #ff8f6e; }}
+  pre {{ background: #121826; padding: 1em; border-radius: 6px;
+         overflow-x: auto; font-size: 13px; line-height: 1.45; }}
+  a {{ color: #7cc4ff; }}
+  .note {{ color: #9aa7ba; font-size: 14px; }}
+</style>
+</head><body>
+<h1>bucket-sqlite-probe-docker</h1>
+<p class="note">
+Matched pair with
+<a href="https://huggingface.co/spaces/davanstrien/bucket-sqlite-probe-gradio">
+bucket-sqlite-probe-gradio</a>.
+This half runs as <code>UID 1000</code> per
+<a href="https://huggingface.co/docs/hub/spaces-sdks-docker#permissions">
+spaces-sdks-docker#permissions</a>. Output below captured at container startup.
+</p>
+<pre>{body}</pre>
+</body></html>
+"""
+
+
+class _Handler(http.server.BaseHTTPRequestHandler):
+    def do_GET(self):  # noqa: N802
+        import html
+
+        body = _HTML.format(body=html.escape(STARTUP_PROBE))
+        data = body.encode("utf-8")
+        self.send_response(200)
+        self.send_header("Content-Type", "text/html; charset=utf-8")
+        self.send_header("Content-Length", str(len(data)))
+        self.end_headers()
+        self.wfile.write(data)
+
+    def log_message(self, fmt, *args):  # noqa: N802
+        # Quiet the access log; keep stderr clean for probe output visibility.
+        return
+
+
+if __name__ == "__main__":
+    port = int(os.environ.get("PORT", "7860"))
+    print(f"Serving probe output on 0.0.0.0:{port}", flush=True)
+    with socketserver.TCPServer(("0.0.0.0", port), _Handler) as srv:
+        srv.serve_forever()