seriffic Claude Opus 4.7 (1M context) commited on
Commit
89c4f83
Β·
1 Parent(s): d43cf2b

feat: enable EO specialists via remote ML on HF Space

Browse files

Prithvi-NYC-Pluvial, TerraMind LULC, TerraMind Buildings, and the
eo_chip fetch were all returning 'deps unavailable on this
deployment' on the HF Space because their deps gates required
terratorch (which doesn't fit the HF build sandbox per the four
prior failed attempts documented in the Dockerfile). But these
specialists already have remote-inference paths to the AMD MI300X
via app/inference.py β€” terratorch is only needed for *local*
inference, not for the chip-fetch + remote-call path that the HF
Space actually uses.

Refactor:

- Add planetary-computer, pystac-client, rioxarray, xarray, einops
to requirements.txt. These are pure-Python or thin numpy/rasterio
wrappers (~10 MB combined) β€” small enough to fit the HF build
budget that terratorch's 250 MB transitive dep cone overflows.

- prithvi_live._has_required_deps splits into two tiers: chip-fetch
deps (Tier 1, always required) and local-inference deps (Tier 2,
only required when remote inference is unavailable). When remote
is configured (RIPRAP_ML_BACKEND=remote + RIPRAP_ML_BASE_URL set,
which is the default on HF), Tier 2 is skipped and the specialist
runs via the existing app.inference.prithvi_pluvial() path.

terramind_lulc and terramind_buildings inherit the fix transparently:
they consume the upstream eo_chip, and terramind_nyc._run already
tries remote first before checking local deps.

terramind_synthesis is left unchanged β€” it has no remote path
(DEM-driven LULC synthesis is not in services/riprap-models/main.py)
and continues to skip cleanly on HF.

Verified locally with RIPRAP_ML_BACKEND=remote: prithvi_live and
eo_chip_cache report _DEPS_OK=True.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

app/context/terramind_synthesis.py CHANGED
@@ -75,15 +75,14 @@ LULC_CLASSES = [
75
 
76
 
77
  def _has_required_deps() -> tuple[bool, str | None]:
78
- """Probe deps. Distinguishes a *truly missing* package
79
- (ModuleNotFoundError) from a *transient race* (other ImportError β€”
80
- typically sklearn's "partially initialized module" from concurrent
81
- imports inside the parallel-fanout block).
82
-
83
- Truly missing returns (False, names). Transient race returns
84
- (True, None) β€” let the caller try again, the import will resolve
85
- on the next attempt once the racing thread finishes.
86
- """
87
  missing = []
88
  for name in ("terratorch", "rasterio"):
89
  try:
@@ -91,8 +90,6 @@ def _has_required_deps() -> tuple[bool, str | None]:
91
  except ModuleNotFoundError:
92
  missing.append(name)
93
  except ImportError:
94
- # sklearn-style partial-init race; treat as available and
95
- # let _ensure_model retry. Logged but not surfaced as missing.
96
  log.debug("terramind: import race on %s, will retry on demand", name)
97
  return (not missing, ", ".join(missing) if missing else None)
98
 
 
75
 
76
 
77
  def _has_required_deps() -> tuple[bool, str | None]:
78
+ """Probe deps. terramind_synthesis runs only locally (no remote path
79
+ in app/inference.py for DEM-driven synthesis), so it always needs
80
+ terratorch. On the HF Space terratorch isn't installed, so this
81
+ specialist returns a clean `skipped: deps unavailable` outcome.
82
+
83
+ Distinguishes a *truly missing* package (ModuleNotFoundError) from
84
+ a *transient race* (other ImportError β€” typically sklearn's
85
+ "partially initialized module" from concurrent imports)."""
 
86
  missing = []
87
  for name in ("terratorch", "rasterio"):
88
  try:
 
90
  except ModuleNotFoundError:
91
  missing.append(name)
92
  except ImportError:
 
 
93
  log.debug("terramind: import race on %s, will retry on demand", name)
94
  return (not missing, ", ".join(missing) if missing else None)
95
 
app/flood_layers/prithvi_live.py CHANGED
@@ -63,27 +63,48 @@ _INIT_LOCK = threading.Lock() # serializes lazy load if multiple threads
63
 
64
 
65
  def _has_required_deps() -> tuple[bool, str | None]:
66
- """Heavy-EO deps (terratorch / planetary_computer / rioxarray /
67
- pystac-client / xarray / einops) live in requirements-experiments.txt
68
- only β€” they don't fit Riprap's HF Spaces' Py3.10 dep cone alongside
69
- transformers<5 / hf_hub<1 / granite-tsfm<0.3.4 / mellea<0.4.
70
-
71
- Probe each importable name once at module load. If any are missing,
72
- fetch() returns a clean `skipped: deps_unavailable` outcome instead
73
- of crashing with a noisy ModuleNotFoundError in the trace. Local
74
- dev + AMD path have these installed and the specialist runs."""
75
- missing = []
76
- for name in ("terratorch", "planetary_computer", "pystac_client",
77
- "rioxarray", "xarray", "einops"):
78
- try:
79
- __import__(name)
80
- except ImportError:
81
- missing.append(name)
 
 
 
 
82
  if missing:
83
  return False, ", ".join(missing)
 
 
 
 
 
 
 
 
 
84
  return True, None
85
 
86
 
 
 
 
 
 
 
 
 
87
  _DEPS_OK, _DEPS_MISSING = _has_required_deps()
88
 
89
 
 
63
 
64
 
65
  def _has_required_deps() -> tuple[bool, str | None]:
66
+ """Probe deps in two tiers.
67
+
68
+ Tier 1 β€” chip fetching (planetary_computer / pystac_client / rioxarray
69
+ / xarray / einops) is always required: prithvi_live always pulls a
70
+ Sentinel-2 chip from Microsoft Planetary Computer regardless of where
71
+ inference runs.
72
+
73
+ Tier 2 β€” local inference (terratorch) is only required when remote
74
+ inference is unavailable. On the HF Space we have remote inference
75
+ on the AMD MI300X via app/inference.py, so terratorch is not needed
76
+ even though chip-fetch is.
77
+
78
+ Returns (False, missing) if any required dep is missing. Splitting
79
+ the gate this way lets the HF Space deployment fetch chips and run
80
+ remote inference even though it doesn't fit terratorch's transitive
81
+ dep cone (~250 MB) in the HF build sandbox."""
82
+ chip_deps = ("planetary_computer", "pystac_client",
83
+ "rioxarray", "xarray", "einops")
84
+ missing = [n for n in chip_deps
85
+ if not _has_module(n)]
86
  if missing:
87
  return False, ", ".join(missing)
88
+ # Tier 2: only need terratorch if we'd run inference locally.
89
+ try:
90
+ from app import inference as _inf
91
+ if _inf.remote_enabled():
92
+ return True, None
93
+ except Exception:
94
+ pass
95
+ if not _has_module("terratorch"):
96
+ return False, "terratorch (local inference)"
97
  return True, None
98
 
99
 
100
+ def _has_module(name: str) -> bool:
101
+ try:
102
+ __import__(name)
103
+ return True
104
+ except ImportError:
105
+ return False
106
+
107
+
108
  _DEPS_OK, _DEPS_MISSING = _has_required_deps()
109
 
110
 
requirements.txt CHANGED
@@ -74,10 +74,21 @@ gliner>=0.2.13
74
  # 30-second pip budget.
75
  #
76
  # On HF Spaces the lazy-import path returns clean `skipped: deps
77
- # unavailable on this deployment` for both prithvi_live and
78
- # terramind_synthesis steps; the other 14 specialists run normally.
79
- # - terratorch / torchgeo / pystac-client / planetary-computer
80
- # - rioxarray / xarray / einops
 
 
 
 
 
 
 
 
 
 
 
81
 
82
  # Burr FSM
83
  burr>=0.40
 
74
  # 30-second pip budget.
75
  #
76
  # On HF Spaces the lazy-import path returns clean `skipped: deps
77
+ # unavailable on this deployment` for terramind_synthesis (which has
78
+ # no remote-inference path); the other EO specialists (prithvi_live,
79
+ # terramind_lulc, terramind_buildings) work via app/inference.py
80
+ # routing to the AMD MI300X droplet, provided we have the chip-fetch
81
+ # deps below β€” they're small (pure-Python or thin wrappers around
82
+ # numpy/rasterio which we already have) and don't pull terratorch or
83
+ # torchvision binaries.
84
+ # - planetary-computer / pystac-client: STAC search at Microsoft PC
85
+ # - rioxarray / xarray: COG band reads
86
+ # - einops: tensor reshape used by prithvi_live._build_chip
87
+ planetary-computer>=1.0
88
+ pystac-client>=0.7
89
+ rioxarray>=0.15
90
+ xarray>=2024.1
91
+ einops>=0.7
92
 
93
  # Burr FSM
94
  burr>=0.40