Add files using upload-large-folder tool

LICENSE

@@ -0,0 +1,28 @@
+BSD 3-Clause License
+
+Copyright (c) 2025, Princeton Vision & Learning Lab
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions are met:
+
+1. Redistributions of source code must retain the above copyright notice, this
+   list of conditions and the following disclaimer.
+
+2. Redistributions in binary form must reproduce the above copyright notice,
+   this list of conditions and the following disclaimer in the documentation
+   and/or other materials provided with the distribution.
+
+3. Neither the name of the copyright holder nor the names of its
+   contributors may be used to endorse or promote products derived from
+   this software without specific prior written permission.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
+FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
+SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
+CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

PHASE0_EVALMDE_HANDOFF.md

@@ -0,0 +1,245 @@
+# Phase 0 EvalMDE Adaptation — Handoff
+
+**Date:** 2026-05-14
+**Status:** EvalMDE workspace bootstrapped; main eval script + sbatch still to write.
+
+---
+
+## Goal
+
+Run the 7 MoGe-Phase-0 models on **Infinigen 95 scenes** under the **EvalMDE protocol**
+(raw native input, no homography warp), producing **RelNormal + SAWA-H + standard metrics**.
+
+EvalMDE and MoGe are independent workflows. EvalMDE workspace is at `/home/ywan0794/EvalMDE/`.
+Model wrappers are *copied* from MoGe (single source of truth still in MoGe/baselines/),
+because the wrappers' `infer(image, intrinsics)` API doesn't depend on MoGe's eval pipeline.
+
+---
+
+## What's done
+
+### 1. EvalMDE env (Python 3.10) — built and verified
+`evalmde` conda env has: torch 2.7.0+cu126, opencv, scipy, utils3d, pipeline, evalmde package,
+bpy 4.0 (Blender python, for textureless-relighting visualization).
+Sample run `python compute_metrics_example.py` outputs `sawa_h=1.268, rel_normal=0.390` ✓.
+
+### 2. 7 baselines (model wrappers) — copied from MoGe + verified
+`/home/ywan0794/EvalMDE/baselines/`:
+- `depth_pro.py`   →  emits `depth_metric` (+ `intrinsics` from FOV head)
+- `marigold.py`    →  emits `depth_affine_invariant`   (paper: scale_inv+shift_inv → affine)
+- `lotus.py`       →  emits `disparity_affine_invariant`  when `--disparity` set
+- `depthmaster.py` →  emits `depth_affine_invariant`
+- `ppd.py`         →  emits `depth_affine_invariant`   (training quantile normalization)
+- `da3_mono.py`    →  emits `depth_scale_invariant`
+- `fe2e.py`        →  emits `depth_affine_invariant`   (Lpred clamped to [0,1])
+
+`MGEBaselineInterface` copied to `/home/ywan0794/EvalMDE/test/baseline.py`.
+
+### 3. EvalMDE-native dataloader skeleton — written
+`/home/ywan0794/EvalMDE/scripts/dataloader.py` (`EvalMDELoaderPipeline`):
+- Reads `<scene>/rgb.png` + `<scene>/gt_depth.npz` (keys: `depth (H,W)`, `intr (4,) [fx,fy,cx,cy]px`, `valid (H,W) bool`)
+- Pixel intrinsics → 3×3 normalized matrix `[fx/W, fy/H, cx/W, cy/H]` (MoGe convention)
+- Computes 3D pointmap from depth + native pixel intrinsics
+- NaN/invalid pixels replaced with `1.0` (matches `evalmde/utils/depth.py:load_data` convention)
+- Returns dict with: `image [3,H,W] float [0,1]`, `depth`, `depth_mask`, `intrinsics (3,3)`,
+  `points (H,W,3)`, `is_metric=True`, `_intr_px (4,)` (for EvalMDE metrics raw npz)
+
+### 4. Infinigen download — IN PROGRESS (background)
+- Source: Princeton GDrive `1amzb6KyF2USFQ5W4CeYKFCh1F-yOQsmp`
+- Target: `/home/ywan0794/EvalMDE/data/infinigen/`
+- Log: `/tmp/dl_infinigen.log`
+- Estimated 50-100 GB
+- Check state: `du -sh /home/ywan0794/EvalMDE/data/infinigen/`
+
+### 5. Production MoGe-protocol eval — independent track, already running
+- `sbatch eval_scripts/eval_all_slurm.sh` submitted earlier (job 12110 etc.)
+- 5 models pending (Marigold/Lotus/DepthMaster/PPD/FE2E), 2 already done (DA3-Mono/Depth Pro)
+- Results in `/home/ywan0794/MoGe/eval_output/<model>_<TS>.json`
+- **EvalMDE adaptation is a separate effort, doesn't block production MoGe eval.**
+
+---
+
+## TODO (was 4 items, now 2 remain)
+
+### ✅ TODO-1: Fix baseline imports — SUPERSEDED by sys.path approach in run_inference.py
+`EvalMDE/baselines/*.py` still have `from moge.test.baseline import MGEBaselineInterface`.
+**Resolved via Option A**: `scripts/run_inference.py` does `sys.path.insert(0, '/home/ywan0794/MoGe')`
+so baselines still resolve their interface from MoGe. No sed needed.
+
+### ✅ TODO-2 (inference driver): `scripts/run_inference.py` — WRITTEN
+`/home/ywan0794/EvalMDE/scripts/run_inference.py`:
+- Click CLI with `--baseline /path/to/baselines/<m>.py --data-root <infinigen> --output-root <out> --model-name <name>`
+- Passes remaining click args through to baseline's `load.main(ctx.args)`
+- For each scene with `rgb.png + gt_depth.npz`: loads rgb, builds normalized 3×3 K from GT pixel intr,
+  calls `baseline.infer_for_evaluation(image, K_norm)`, picks depth in priority order
+  (`depth_metric > depth_scale_invariant > depth_affine_invariant > 1/disparity_affine_invariant`),
+  writes `<out>/<model>/<scene>/pred_depth.npz` with EvalMDE keys `{depth, intr (4,) px, valid}`
+- For pred intrinsics: uses model-predicted intr if present (Depth Pro), else GT intr
+
+### ❗ Original TODO-2 (script/eval.py) was REWORKED into 2 stages: inference + metric.
+This is cleaner: inference runs in per-model env, metric runs in evalmde env.
+
+### TODO-3: Write `scripts/compute_metrics.py` (run in evalmde env)
+Reads each model's pred_depth.npz + GT gt_depth.npz, computes EvalMDE metrics + standard MDE metrics.
+
+Pseudocode:
+```python
+import sys, json, click
+from pathlib import Path
+import numpy as np
+
+from evalmde.utils.depth import load_data
+from evalmde.metrics.rel_normal import compute_rel_normal
+from evalmde.metrics.sawa_h     import compute_sawa_h
+
+@click.command()
+@click.option('--gt-root',   required=True, type=click.Path())  # Infinigen root
+@click.option('--pred-root', required=True, type=click.Path())  # output of run_inference.py
+@click.option('--model-name', required=True, type=str)
+@click.option('--output',    required=True, type=click.Path())
+def main(gt_root, pred_root, model_name, output):
+    gt_root = Path(gt_root); pred_root = Path(pred_root) / model_name
+    scenes = sorted(d.name for d in pred_root.iterdir() if (d / 'pred_depth.npz').exists())
+
+    results = []
+    for scene in scenes:
+        gt_d,  gt_intr,  gt_v = load_data(gt_root  / scene / 'gt_depth.npz')
+        pr_d,  pr_intr,  pr_v = load_data(pred_root / scene / 'pred_depth.npz')
+
+        # SAWA-H aligns internally (affine via least-squares). RelNormal uses surface normals
+        # which are invariant to scale but NOT to shift — for affine-invariant preds, the
+        # shift will skew normals at far depths. Acceptable caveat in Phase 0; document it.
+        sawa  = compute_sawa_h    (pr_d, pr_intr, pr_v, gt_d, gt_intr, gt_v)
+        rnorm = compute_rel_normal(pr_d, pr_intr, pr_v, gt_d, gt_intr, gt_v)
+
+        # Standard AbsRel + δ1 after affine alignment (re-implement, ~10 lines):
+        mask  = gt_v & pr_v
+        gtm, prm = gt_d[mask], pr_d[mask]
+        # fit y = a*x + b on (prm, gtm)
+        A = np.stack([prm, np.ones_like(prm)], axis=-1)
+        a, b = np.linalg.lstsq(A, gtm, rcond=None)[0]
+        aligned = pr_d * a + b
+        am = aligned[mask]
+        abs_rel = np.mean(np.abs(am - gtm) / np.maximum(gtm, 1e-6))
+        delta1  = np.mean(np.maximum(am/gtm, gtm/am) < 1.25)
+
+        results.append({'scene': scene, 'sawa_h': float(sawa), 'rel_normal': float(rnorm),
+                        'abs_rel': float(abs_rel), 'delta1': float(delta1)})
+
+    # Per-scene + aggregate mean
+    summary = {'per_scene': results,
+               'mean': {k: float(np.mean([r[k] for r in results])) for k in ['sawa_h','rel_normal','abs_rel','delta1']}}
+    json.dump(summary, open(output, 'w'), indent=2)
+
+
+if __name__ == '__main__':
+    main()
+```
+
+**Note on alignment**: `compute_sawa_h` aligns internally (via `align_depth_least_square` + `align_affine_lstsq`),
+so passing RAW pred (affine-invariant) is correct. `compute_rel_normal` does NOT align — its
+inputs should be in a comparable depth scale. For Phase 0 simplicity, pass raw pred; document
+the affine-shift caveat in the analysis. For stricter eval, pre-affine-align before RelNormal.
+
+### TODO-4: Scene list / config
+Once Infinigen download succeeds (currently blocked, see issue below), `run_inference.py`
+auto-discovers all scene dirs under `--data-root`. If a subset is wanted, write
+`scenes.txt` and add filtering in run_inference.py (~3 lines).
+
+### TODO-5: sbatch `eval_scripts/eval_evalmde_all_slurm.sh`
+Same pattern as MoGe's `sanity_all_slurm.sh`: single sbatch, single H100, serial per-model.
+For each of 7 models: `conda activate <env>; python scripts/run_inference.py --baseline baselines/<m>.py ...`
+Then after all 7 inferences done: `conda activate evalmde; for m in ...; do python scripts/compute_metrics.py --model-name $m ...; done`
+
+Each per-model env needs `evalmde` pip-installed so it can `from evalmde.metrics...` — actually
+**no, this is wrong**: per-model envs only run inference (which needs torch + model wrapper deps,
+no evalmde). Only the metric-aggregation stage runs in evalmde env. So envs need no extra install.
+
+### TODO-3: Scene list / config
+Once Infinigen download finishes, inspect actual layout:
+```bash
+ls /home/ywan0794/EvalMDE/data/infinigen/ | head -20
+```
+If scenes are `scene_001/`, `scene_002/`, ...: dataloader auto-discovers them.
+If grouped under sub-folders or different naming: may need a manual `scenes.txt` split file.
+
+### TODO-4: sbatch `EvalMDE/eval_scripts/eval_evalmde_all_slurm.sh`
+Mirror MoGe's `sanity_all_slurm.sh` structure:
+- Single sbatch, single H100, serial per-model
+- For each model: activate model's conda env, run `python scripts/eval.py --baseline baselines/<m>.py --data-root data/infinigen --output results/<m>.json`
+- After all inference done, optionally re-aggregate in evalmde env for cross-model summary
+
+Per-model env mapping same as MoGe:
+| model | env |
+|---|---|
+| depth_pro | depth-pro |
+| marigold | marigold |
+| lotus | lotus |
+| depthmaster | depthmaster |
+| ppd | ppd |
+| da3_mono | da3 |
+| fe2e | fe2e |
+
+Plus: each env needs `evalmde` package installed (`pip install -e /home/ywan0794/EvalMDE`)
+so `from evalmde.metrics.* import compute_rel_normal, compute_sawa_h` works inside model envs.
+
+---
+
+## Paper-canonical inference parameters (locked, confirmed against each repo)
+
+| Model | Args | Source |
+|---|---|---|
+| Depth Pro | `--precision fp32` | `create_model_and_transforms()` default |
+| Marigold | v1-1 + `--denoise_steps 4 --ensemble_size 1` | (user decision: balanced speed) |
+| Lotus | g-v2-1-disparity + `--mode generation --disparity --timestep 999 --fp16 --seed 42` | `Lotus/eval.sh` |
+| DepthMaster | `--processing_res 768` | `DepthMaster/scripts/infer.sh` |
+| PPD | `--semantics_model MoGe2 --semantics_pth checkpoints/moge2.pt --model_pth checkpoints/ppd_moge.pth --sampling_steps 4` | `PPD/ppd/configs/eval.yaml` |
+| DA3-Mono | `--hf_id depth-anything/DA3MONO-LARGE` | DA3 README |
+| FE2E | `--prompt_type empty --single_denoise --cfg_guidance 6.0 --size_level 768` | `FE2E/README.md` eval block |
+
+---
+
+## Key insights to preserve
+
+1. **EvalMDE protocol uses raw native input, no homography warp.** MoGe's eval pipeline
+   does aggressive canonical-view warping (`dataloader.py:_process_instance:119-180`).
+   That is MoGe-paper-specific; EvalMDE explicitly uses raw inputs (see `compute_metrics_example.py`).
+
+2. **Output key contract** (per MGEBaselineInterface):
+   - `depth_metric` → metric depth in meters (Depth Pro)
+   - `depth_scale_invariant` → scale-invariant relative depth (DA3-Mono)
+   - `depth_affine_invariant` → affine-invariant depth (Marigold/DepthMaster/PPD/FE2E)
+   - `disparity_affine_invariant` → affine-invariant disparity (Lotus disparity ckpts)
+
+3. **Pre-alignment for SAWA-H/RelNormal**: SAWA-H itself does affine alignment internally
+   (`evalmde/metrics/sawa_h.py:compute_sawa_h` uses `align_depth_least_square` + `align_affine_lstsq`),
+   so you can pass RAW pred depth to SAWA-H. RelNormal works on normals which are
+   scale-invariant in the limit, but **shift in depth space WILL skew normals at far depths** —
+   so for affine-invariant pred models, do an affine align before passing to `compute_rel_normal`.
+
+4. **MoGe's eval can run in parallel with EvalMDE work.** Production `eval_all_slurm.sh`
+   already running. Don't disturb.
+
+5. **Lotus disparity ckpt inversion was numerically unstable** (1/disp blows up near
+   disparity=0). For EvalMDE, only emit `disparity_affine_invariant` from Lotus, then
+   convert: `aligned_disp = scale*disp + shift` (fit in disp space), `aligned_depth = 1/aligned_disp.clamp(1/gt_depth_max)`.
+   Reference: `moge/test/metrics.py:202-218` disparity_affine_invariant block.
+
+---
+
+## Resume instructions
+
+1. `cd /home/ywan0794/EvalMDE`
+2. Check Infinigen download: `du -sh data/infinigen; tail /tmp/dl_infinigen.log`
+3. Fix imports (TODO-1):
+   ```bash
+   sed -i 's|from moge.test.baseline|from test.baseline|g' baselines/*.py
+   ```
+4. Write `scripts/eval.py` (TODO-2) using the pseudocode above.
+5. Test on 1 scene with depth_pro: `python scripts/eval.py --baseline baselines/depth_pro.py --data-root data/infinigen --output /tmp/test.json --repo /home/ywan0794/EvalMDE/ml-depth-pro --checkpoint /home/ywan0794/EvalMDE/ml-depth-pro/checkpoints/depth_pro.pt`
+6. Inspect `/tmp/test.json`. If sane (rel_normal in [0, 1] rad, sawa_h plausible),
+   proceed to write sbatch (TODO-4).
+
+---
+
+**End of handoff.**

README.md

@@ -0,0 +1,90 @@
+# Toward A Better Understanding of Monocular Depth Evaluation
+This repository contains the source code for our paper:
+
+[Toward A Better Understanding of Monocular Depth Evaluation](https://arxiv.org/abs/2510.19814)<br/>
+[Siyang Wu](https://nj-wusiyang.github.io/), Jack Nugent, Willow Yang, [Jia Deng](https://www.cs.princeton.edu/~jiadeng/)
+
+```
+@article{wu2025evaluate,
+  title={Toward A Better Understanding of Monocular Depth Evaluation},
+  author={Wu, Siyang and Nugent, Jack and Yang, Willow and Deng, Jia},
+  journal={arXiv preprint arXiv:2510.19814},
+  year={2025}
+}
+```
+
+## Installation Instructions
+Under `EvalMDE`, run:
+```bash
+conda create -n evalmde python=3.10 -y
+conda activate evalmde
+
+pip install torch==2.7.0 torchvision==0.22.0 torchaudio==2.7.0 --index-url https://download.pytorch.org/whl/cu126
+pip install -e .
+pip install bpy==4.0.0 --extra-index-url https://download.blender.org/pypi/
+```
+
+## Data Format
+### Depth map file
+This repository takes depth map file in `*.npz` format, with keys: `depth, intr, valid`.
++ `depth`: `(H,W)`-shaped numpy array that stores depth value;
++ `intr`: `(4,)`-shaped numpy array that stores camera intrinsics `[fx, fy, cx, cy]`, where units are pixels;
++ `valid`: `(H,W)`-shaped boolean numpy array that stores whether the depth value of a pixel is valid (e.g. a pixel of `inf,nan` or extreme depth value is invalid).
+
+`sample_data/gt_depth.npz`, `sample_data/curv_low_freq__0.200_10.0.npz`, `sample_data_2/gt_depth.npz`, `sample_data_2/depthpro_gt_focal.npz` provide examples of depth map files.
+### Valid triangle (Required For Textureless Relighting Visualization)
+In textureless relighting, we induce a textureless mesh from depth map and camera intrinsics. The mesh consists of triangle faces of vertices `(i,j), (i+1,j), (i,j+1)` and `(i+1,j+1), (i+1,j), (i,j+1)`.
+Triangle faces across occlusion boundaries should be excluded.
+
+`valid_triangle.npz` specifies which triangles are included (`True`) and which are not (`False`). 
+It has keys: `valid_triangle`, which is a `(H-1,W-1,2)` shaped boolean numpy array, where `valid_triangle[i,j,0/1]` stands for whether `(i,j), (i+1,j), (i,j+1)` and `(i+1,j+1), (i+1,j), (i,j+1)` are included (`True`) or not included (`False`).
+`sample_data/valid_triangle.npz` provides an example.
+
+**Induce valid triangle from ground truth depth.** Valid triangle can be induced from ground truth depth by inducing occlusion boundaries by some heuristic.
+In `induce_valid_triangle_from_gt_depth.py`, we provide an example script which detects occlusion boundaries by relative depth between neighboring pixels and set triangles across occlusion boundaries as invalid.
+Running `python induce_valid_triangle_from_gt_depth.py` generates `sample_data_2/valid_triangle.npz`. 
+
+## Compute Metric
+Please refer to `compute_metrics_example.py`
+
+## Visualization
+### Projected Contours
+
+<img src="images/projected_contours.png">
+
+```bash
+ROOT=sample_data  # Path to directory where rgb.png is located
+# ROOT=sample_data_2
+DEPTH_F=gt_depth.npz  # Path to depth map to draw visualization, relative to $ROOT 
+# DEPTH_F=curv_low_freq__0.200_10.0.npz  # when ROOT=sample_data
+# DEPTH_F=depthpro_gt_focal.npz  # when ROOT=sample_data_2
+python evalmde/visualization/render_contour_line.py $ROOT --depth_f $DEPTH_F
+```
+Running the above command generates projected contours visualization under `sample_data/contour_line` or `sample_data_2/contour_line`.
+Projected contours of different densities along different axes are generated.
+### Textureless Relighting
+
+<img src="images/textureless_relighting.png">
+
+```bash
+ROOT=sample_data  # Path to directory where rgb.png is located
+# ROOT=sample_data_2
+DEPTH_F=gt_depth.npz  # Path to depth map to draw visualization, relative to $ROOT 
+# DEPTH_F=curv_low_freq__0.200_10.0.npz  # when ROOT=sample_data
+# DEPTH_F=depthpro_gt_focal.npz  # when ROOT=sample_data_2
+LIGHT_L=0  # specifies light direction
+LIGHT_R=5  # specifies light direction
+python evalmde/visualization/render_textureless_relighting.py $ROOT --depth_f $DEPTH_F --light_l $LIGHT_L --light_r $LIGHT_R
+```
+Running the above command generates textureless relighting visualization under `sample_data/visualization`.
+By default, the script renders visualization using GPU. Add `--cpu` to run everything in cpu.
+
+`ROT_LIGHT_NUM_LIGHT,ROT_LIGHT_NUM_LOOP` in `evalmde/visualization/__init__.py` specifies the light configuration. 
+`ROT_LIGHT_NUM_LIGHT` locations of the source of directional light are equally spaced along the path that spirals up from `(0,0,-1)` to `(0,0,1)` along the surface of a unit sphere, rotating around `z`-axis for `ROT_LIGHT_NUM_LOOP` times.
+Textureless mesh under the `i`-th source of directional light (`LIGHT_L<=i<LIGHT_R`) are rendered in the above command.
+
+## Dataset
+Dataset can be accessed [here](https://drive.google.com/drive/folders/1amzb6KyF2USFQ5W4CeYKFCh1F-yOQsmp?usp=sharing).
+
+## Acknowledgments
+This repository uses open source projects. We specially thank authors of [MoGe](https://github.com/microsoft/MoGe), [Marigold](https://github.com/prs-eth/Marigold), [DepthPro](https://github.com/apple/ml-depth-pro).

compute_metrics_example.py

@@ -0,0 +1,12 @@
+from evalmde.utils.depth import load_data
+# gt_depth, gt_intr, gt_valid = load_data('sample_data/gt_depth.npz')
+# pr_depth, pr_intr, pr_valid = load_data('sample_data/curv_low_freq__0.200_10.0.npz')
+gt_depth, gt_intr, gt_valid = load_data('sample_data_2/gt_depth.npz')
+pr_depth, pr_intr, pr_valid = load_data('sample_data_2/depthpro_gt_focal.npz')
+
+
+from evalmde.metrics.rel_normal import compute_rel_normal
+from evalmde.metrics.sawa_h import compute_sawa_h
+sawa_h = compute_sawa_h(pr_depth, pr_intr, pr_valid, gt_depth, gt_intr, gt_valid)
+rel_normal = compute_rel_normal(pr_depth, pr_intr, pr_valid, gt_depth, gt_intr, gt_valid)
+print(f'{sawa_h=}, {rel_normal=}')

evalmde/metrics/boundary.py

@@ -0,0 +1,346 @@
+# source https://github.com/apple/ml-depth-pro/blob/main/src/depth_pro/eval/boundary_metrics.py
+
+from typing import List, Tuple
+
+import numpy as np
+import torch
+
+
+def connected_component(r: np.ndarray, c: np.ndarray) -> List[List[int]]:
+    """Find connected components in the given row and column indices.
+
+    Args:
+    ----
+        r (np.ndarray): Row indices.
+        c (np.ndarray): Column indices.
+
+    Yields:
+    ------
+        List[int]: Indices of connected components.
+
+    """
+    indices = [0]
+    for i in range(1, r.size):
+        if r[i] == r[indices[-1]] and c[i] == c[indices[-1]] + 1:
+            indices.append(i)
+        else:
+            yield indices
+            indices = [i]
+    yield indices
+
+
+def nms_horizontal(ratio: np.ndarray, threshold: float) -> np.ndarray:
+    """Apply Non-Maximum Suppression (NMS) horizontally on the given ratio matrix.
+
+    Args:
+    ----
+        ratio (np.ndarray): Input ratio matrix.
+        threshold (float): Threshold for NMS.
+
+    Returns:
+    -------
+        np.ndarray: Binary mask after applying NMS.
+
+    """
+    mask = np.zeros_like(ratio, dtype=bool)
+    r, c = np.nonzero(ratio > threshold)
+    if len(r) == 0:
+        return mask
+    for ids in connected_component(r, c):
+        values = [ratio[r[i], c[i]] for i in ids]
+        mi = np.argmax(values)
+        mask[r[ids[mi]], c[ids[mi]]] = True
+    return mask
+
+
+def nms_vertical(ratio: np.ndarray, threshold: float) -> np.ndarray:
+    """Apply Non-Maximum Suppression (NMS) vertically on the given ratio matrix.
+
+    Args:
+    ----
+        ratio (np.ndarray): Input ratio matrix.
+        threshold (float): Threshold for NMS.
+
+    Returns:
+    -------
+        np.ndarray: Binary mask after applying NMS.
+
+    """
+    return np.transpose(nms_horizontal(np.transpose(ratio), threshold))
+
+
+def fgbg_depth(
+    d: np.ndarray, t: float
+) -> Tuple[np.ndarray, np.ndarray, np.ndarray, np.ndarray]:
+    """Find foreground-background relations between neighboring pixels.
+
+    Args:
+    ----
+        d (np.ndarray): Depth matrix.
+        t (float): Threshold for comparison.
+
+    Returns:
+    -------
+        Tuple[np.ndarray, np.ndarray, np.ndarray, np.ndarray]: Four matrices indicating
+        left, top, right, and bottom foreground-background relations.
+
+    """
+    right_is_big_enough = (d[..., :, 1:] / d[..., :, :-1]) > t
+    left_is_big_enough = (d[..., :, :-1] / d[..., :, 1:]) > t
+    bottom_is_big_enough = (d[..., 1:, :] / d[..., :-1, :]) > t
+    top_is_big_enough = (d[..., :-1, :] / d[..., 1:, :]) > t
+    return (
+        left_is_big_enough,
+        top_is_big_enough,
+        right_is_big_enough,
+        bottom_is_big_enough,
+    )
+
+
+def fgbg_depth_thinned(
+    d: np.ndarray, t: float
+) -> Tuple[np.ndarray, np.ndarray, np.ndarray, np.ndarray]:
+    """Find foreground-background relations between neighboring pixels with Non-Maximum Suppression.
+
+    Args:
+    ----
+        d (np.ndarray): Depth matrix.
+        t (float): Threshold for NMS.
+
+    Returns:
+    -------
+        Tuple[np.ndarray, np.ndarray, np.ndarray, np.ndarray]: Four matrices indicating
+        left, top, right, and bottom foreground-background relations with NMS applied.
+
+    """
+    right_is_big_enough = nms_horizontal(d[..., :, 1:] / d[..., :, :-1], t)
+    left_is_big_enough = nms_horizontal(d[..., :, :-1] / d[..., :, 1:], t)
+    bottom_is_big_enough = nms_vertical(d[..., 1:, :] / d[..., :-1, :], t)
+    top_is_big_enough = nms_vertical(d[..., :-1, :] / d[..., 1:, :], t)
+    return (
+        left_is_big_enough,
+        top_is_big_enough,
+        right_is_big_enough,
+        bottom_is_big_enough,
+    )
+
+
+def fgbg_binary_mask(
+    d: np.ndarray,
+) -> Tuple[np.ndarray, np.ndarray, np.ndarray, np.ndarray]:
+    """Find foreground-background relations between neighboring pixels in binary masks.
+
+    Args:
+    ----
+        d (np.ndarray): Binary depth matrix.
+
+    Returns:
+    -------
+        Tuple[np.ndarray, np.ndarray, np.ndarray, np.ndarray]: Four matrices indicating
+        left, top, right, and bottom foreground-background relations in binary masks.
+
+    """
+    assert d.dtype == bool
+    right_is_big_enough = d[..., :, 1:] & ~d[..., :, :-1]
+    left_is_big_enough = d[..., :, :-1] & ~d[..., :, 1:]
+    bottom_is_big_enough = d[..., 1:, :] & ~d[..., :-1, :]
+    top_is_big_enough = d[..., :-1, :] & ~d[..., 1:, :]
+    return (
+        left_is_big_enough,
+        top_is_big_enough,
+        right_is_big_enough,
+        bottom_is_big_enough,
+    )
+
+
+def edge_recall_matting(pr: np.ndarray, gt: np.ndarray, t: float) -> float:
+    """Calculate edge recall for image matting.
+
+    Args:
+    ----
+        pr (np.ndarray): Predicted depth matrix.
+        gt (np.ndarray): Ground truth binary mask.
+        t (float): Threshold for NMS.
+
+    Returns:
+    -------
+        float: Edge recall value.
+
+    """
+    assert gt.dtype == bool
+    ap, bp, cp, dp = fgbg_depth_thinned(pr, t)
+    ag, bg, cg, dg = fgbg_binary_mask(gt)
+    return 0.25 * (
+        np.count_nonzero(ap & ag) / max(np.count_nonzero(ag), 1)
+        + np.count_nonzero(bp & bg) / max(np.count_nonzero(bg), 1)
+        + np.count_nonzero(cp & cg) / max(np.count_nonzero(cg), 1)
+        + np.count_nonzero(dp & dg) / max(np.count_nonzero(dg), 1)
+    )
+
+
+def _boundary_f1(
+    pr: np.ndarray,
+    gt: np.ndarray,
+    t: float,
+    return_p: bool = False,
+    return_r: bool = False,
+) -> float:
+    """Calculate Boundary F1 score.
+
+    Args:
+    ----
+        pr (np.ndarray): Predicted depth matrix.
+        gt (np.ndarray): Ground truth depth matrix.
+        t (float): Threshold for comparison.
+        return_p (bool, optional): If True, return precision. Defaults to False.
+        return_r (bool, optional): If True, return recall. Defaults to False.
+
+    Returns:
+    -------
+        float: Boundary F1 score, or precision, or recall depending on the flags.
+
+    """
+    ap, bp, cp, dp = fgbg_depth(pr, t)
+    ag, bg, cg, dg = fgbg_depth(gt, t)
+
+    r = 0.25 * (
+        np.count_nonzero(ap & ag) / max(np.count_nonzero(ag), 1)
+        + np.count_nonzero(bp & bg) / max(np.count_nonzero(bg), 1)
+        + np.count_nonzero(cp & cg) / max(np.count_nonzero(cg), 1)
+        + np.count_nonzero(dp & dg) / max(np.count_nonzero(dg), 1)
+    )
+    p = 0.25 * (
+        np.count_nonzero(ap & ag) / max(np.count_nonzero(ap), 1)
+        + np.count_nonzero(bp & bg) / max(np.count_nonzero(bp), 1)
+        + np.count_nonzero(cp & cg) / max(np.count_nonzero(cp), 1)
+        + np.count_nonzero(dp & dg) / max(np.count_nonzero(dp), 1)
+    )
+    if r + p == 0:
+        return 0.0
+    if return_p:
+        return p
+    if return_r:
+        return r
+    return 2 * (r * p) / (r + p)
+
+
+def get_thresholds_and_weights(
+    t_min: float, t_max: float, N: int
+) -> Tuple[np.ndarray, np.ndarray]:
+    """Generate thresholds and weights for the given range.
+
+    Args:
+    ----
+        t_min (float): Minimum threshold.
+        t_max (float): Maximum threshold.
+        N (int): Number of thresholds.
+
+    Returns:
+    -------
+        Tuple[np.ndarray, np.ndarray]: Array of thresholds and corresponding weights.
+
+    """
+    thresholds = np.linspace(t_min, t_max, N)
+    weights = thresholds / thresholds.sum()
+    return thresholds, weights
+
+
+def invert_depth(depth: np.ndarray, eps: float = 1e-6) -> np.ndarray:
+    """Inverts a depth map with numerical stability.
+
+    Args:
+    ----
+        depth (np.ndarray): Depth map to be inverted.
+        eps (float): Minimum value to avoid division by zero (default is 1e-6).
+
+    Returns:
+    -------
+    np.ndarray: Inverted depth map.
+
+    """
+    inverse_depth = 1.0 / depth.clip(min=eps)
+    return inverse_depth
+
+
+def SI_boundary_F1(
+    predicted_depth: np.ndarray,
+    target_depth: np.ndarray,
+    t_min: float = 1.05,
+    t_max: float = 1.25,
+    N: int = 10,
+) -> float:
+    """Calculate Scale-Invariant Boundary F1 Score for depth-based ground-truth.
+
+    Args:
+    ----
+        predicted_depth (np.ndarray): Predicted depth matrix.
+        target_depth (np.ndarray): Ground truth depth matrix.
+        t_min (float, optional): Minimum threshold. Defaults to 1.05.
+        t_max (float, optional): Maximum threshold. Defaults to 1.25.
+        N (int, optional): Number of thresholds. Defaults to 10.
+
+    Returns:
+    -------
+        float: Scale-Invariant Boundary F1 Score.
+
+    """
+    assert predicted_depth.ndim == target_depth.ndim == 2
+    thresholds, weights = get_thresholds_and_weights(t_min, t_max, N)
+    f1_scores = np.array(
+        [
+            _boundary_f1(invert_depth(predicted_depth), invert_depth(target_depth), t)
+            for t in thresholds
+        ]
+    )
+    return np.sum(f1_scores * weights)
+
+
+def SI_boundary_Recall(
+    predicted_depth: np.ndarray,
+    target_mask: np.ndarray,
+    t_min: float = 1.05,
+    t_max: float = 1.25,
+    N: int = 10,
+    alpha_threshold: float = 0.1,
+) -> float:
+    """Calculate Scale-Invariant Boundary Recall Score for mask-based ground-truth.
+
+    Args:
+    ----
+        predicted_depth (np.ndarray): Predicted depth matrix.
+        target_mask (np.ndarray): Ground truth binary mask.
+        t_min (float, optional): Minimum threshold. Defaults to 1.05.
+        t_max (float, optional): Maximum threshold. Defaults to 1.25.
+        N (int, optional): Number of thresholds. Defaults to 10.
+        alpha_threshold (float, optional): Threshold for alpha masking. Defaults to 0.1.
+
+    Returns:
+    -------
+        float: Scale-Invariant Boundary Recall Score.
+
+    """
+    assert predicted_depth.ndim == target_mask.ndim == 2
+    thresholds, weights = get_thresholds_and_weights(t_min, t_max, N)
+    thresholded_target = target_mask > alpha_threshold
+
+    recall_scores = np.array(
+        [
+            edge_recall_matting(
+                invert_depth(predicted_depth), thresholded_target, t=float(t)
+            )
+            for t in thresholds
+        ]
+    )
+    weighted_recall = np.sum(recall_scores * weights)
+    return weighted_recall
+
+def boundary_f1(pred, target, mask):
+    # set masked values to NaN
+    pred = torch.where(mask, pred, torch.nan)
+    target = torch.where(mask, target, torch.nan)
+
+    f1 = SI_boundary_F1(pred.cpu().numpy(), target.cpu().numpy())
+
+    return None, (1 - f1).item()
+
+

evalmde/metrics/rel_normal.py

@@ -0,0 +1,231 @@
+from typing import List
+from math import floor
+
+import numpy as np
+import torch
+import torch.nn.functional as F
+
+from evalmde.utils.torch import get_angle_between, reformat_as_torch_tensor
+from evalmde.utils.downsample import downsample
+from evalmde.utils.proj import depth_to_xyz
+
+DEFAULT_CONFIG={
+    'scales': [1, 2, 4, 8],
+    'num_sample': int(1e6),
+    'radius': 32,
+    'min_radius': 3,
+    'invalid': 'penalty',
+}
+
+@torch.no_grad()
+def _fetch_pixel_val(x: torch.Tensor, vertex_slice):
+    '''
+    :param x: shape (H, W, ...)
+    :param vertex_slice:
+    :return: shape (H - 1, W - 1, ...)
+    '''
+    return x[vertex_slice[0], vertex_slice[1]]
+
+
+@torch.no_grad()
+def get_triangle_valid(valid: torch.Tensor):
+    '''
+    a triangle is valid if all vertices are valid
+    :param valid: shape (H, W)
+    :return: triangle_valid
+        triangle_valid: shape (H - 1, W - 1, NUM_TRIANGLE)
+    '''
+    H, W = valid.shape
+    device = valid.device
+    ret = torch.empty((H - 2, W - 2, NUM_TRIANGLE), dtype=torch.bool, device=device)
+    for i, TRIANGLE_SLICE in enumerate(TRIANGLE_SLICES):
+        ret[..., i] = _fetch_pixel_val(valid, TRIANGLE_SLICE[0]) & \
+                      _fetch_pixel_val(valid, TRIANGLE_SLICE[1]) & \
+                      _fetch_pixel_val(valid, TRIANGLE_SLICE[2])
+    return ret
+
+TRIANGLE_SLICES=((
+    (slice(None, -2), slice(None, -2)),
+    (slice(2, None), slice(None, -2)),
+    (slice(None, -2), slice(2, None)),
+),)
+NUM_TRIANGLE = 1
+@torch.no_grad()
+def get_triangle_normal(xyz: torch.Tensor):
+    '''
+    Normal computation method 2: 2-pixel spacing
+    :param xyz: shape (H, W, 3)
+    :return: normal, normal_valid
+        normal: shape (H - 2, W - 2, NUM_TRIANGLE_2, 3)
+        normal_valid: shape (H - 2, W - 2, NUM_TRIANGLE_2)
+    '''
+    H, W = xyz.shape[:2]
+    device = xyz.device
+    dtype = xyz.dtype
+    normal = torch.empty((H - 2, W - 2, 1, 3), dtype=dtype, device=device)
+    normal_valid = torch.empty((H - 2, W - 2, 1), dtype=torch.bool, device=device)
+    for i, TRIANGLE_SLICE in enumerate(TRIANGLE_SLICES):
+        normal[..., i, :] = torch.linalg.cross(
+            F.normalize(_fetch_pixel_val(xyz, TRIANGLE_SLICE[1]) - _fetch_pixel_val(xyz, TRIANGLE_SLICE[0]), dim=-1),
+            F.normalize(_fetch_pixel_val(xyz, TRIANGLE_SLICE[2]) - _fetch_pixel_val(xyz, TRIANGLE_SLICE[0]), dim=-1),
+            dim=-1
+        )
+        vec_norm = torch.norm(normal[..., i, :], dim=-1)
+        normal_valid[..., i] = vec_norm > 1e-5
+        normal[..., i, :] /= vec_norm.clamp(min=1e-5).unsqueeze(-1)
+    return normal, normal_valid
+
+@torch.no_grad()
+def get_triangle_normal_and_valid(xyz: torch.Tensor, valid: torch.Tensor, flatten: bool = True):
+    '''
+    if gt_d and depth_layer are not None, filter out triangle across depth layers
+    :param xyz:
+    :param valid:
+    :param flatten:
+    :return: normal, valid
+    '''
+    normal, normal_valid = get_triangle_normal(xyz)
+    tri_valid = get_triangle_valid(valid)
+    valid = normal_valid & tri_valid
+    if flatten:
+        normal = normal.reshape(-1, 3)
+        valid = valid.reshape(-1)
+    return normal, valid
+
+
+@torch.no_grad()
+def get_angle_between(n1: torch.Tensor, n2: torch.Tensor) -> torch.Tensor:
+    '''
+    :param n1: shape (..., 3), norm > 0
+    :param n2: shape (..., 3), norm > 0
+    :return: shape (...)
+    '''
+    return torch.acos((F.normalize(n1, dim=-1) * F.normalize(n2, dim=-1)).sum(dim=-1).clamp(-1, 1))
+
+@torch.no_grad()
+def get_pair_pxl(H: int, W: int, num_sample: int, radius: int, device):
+    radius = min(radius, max(H, W))
+    i1 = torch.empty((num_sample,), dtype=torch.long, device=device)
+    j1 = torch.empty((num_sample,), dtype=torch.long, device=device)
+    i2 = torch.empty((num_sample,), dtype=torch.long, device=device)
+    j2 = torch.empty((num_sample,), dtype=torch.long, device=device)
+
+    n = 0
+    s = torch.quasirandom.SobolEngine(4)
+    while n < num_sample:
+        samples = s.draw(floor(num_sample * 1.1)).to(device)
+        samples[:,0] *= H
+        samples[:,1] *= W
+        samples[:,2] *= radius * 2
+        samples[:,2] -= radius
+        samples[:,3] *= radius * 2
+        samples[:,3] -= radius
+        points = torch.cat([samples[:,:2], samples[:,:2] + samples[:,2:]], dim=1)
+        points = torch.floor(points)
+
+        valid = (points[:,[0,2]] < H).all(dim=-1) & (points[:,[1,3]] < W).all(dim=-1) & (0 <= points[:,[0,2]]).all(dim=-1) & (0 <= points[:,[1,3]]).all(dim=-1)
+        points = points[valid]
+        m = min(len(points), num_sample - n)
+        i1[n:n+m] = points[:m,0]
+        j1[n:n+m] = points[:m,1]
+        i2[n:n+m] = points[:m,2]
+        j2[n:n+m] = points[:m,3]
+        n += m
+    
+    return i1, j1, i2, j2
+
+
+@torch.no_grad()
+def get_rel_normal_err_heatmap_idx(gt_xyz: torch.Tensor, gt_valid: torch.Tensor,
+                               pred_xyz: torch.Tensor, pred_valid: torch.Tensor,
+                               num_sample: int, radius: int):
+    '''
+    :param gt_xyz:
+    :param gt_valid:
+    :param pred_xyz:
+    :param pred_valid:
+    :param num_sample:
+    :param radius:
+    :return: rel_normal_err, gt_pair_valid, pred_pair_valid
+        rel_normal_err: shape (-1,)
+        gt_pair_valid: shape (-1,)
+        pred_pair_valid: shape (-1,)
+    '''
+    gt_normal, gt_normal_valid = get_triangle_normal_and_valid(gt_xyz, gt_valid, flatten=False)
+    pred_normal, pred_normal_valid = get_triangle_normal_and_valid(pred_xyz, pred_valid, flatten=False)
+
+    H, W = gt_normal.shape[:2]
+    i1, j1, i2, j2 = get_pair_pxl(H, W, num_sample, radius, gt_xyz.device)
+
+    gt_rel_normal = get_angle_between(gt_normal[i1, j1], gt_normal[i2, j2])
+    gt_pair_valid = gt_normal_valid[i1, j1] & gt_normal_valid[i2, j2]
+    pred_rel_normal = get_angle_between(pred_normal[i1, j1], pred_normal[i2, j2])
+    pred_pair_valid = pred_normal_valid[i1, j1] & pred_normal_valid[i2, j2]
+    rel_normal_err = torch.abs(gt_rel_normal - pred_rel_normal)  # [0, pi]
+    return rel_normal_err, gt_pair_valid, pred_pair_valid, (i1,j1,i2,j2)
+
+
+
+def get_multi_scale_rel_normal_err(gt_xyz: torch.Tensor, gt_valid: torch.Tensor,
+                                   pred_xyz: torch.Tensor, pred_valid: torch.Tensor,
+                                   scales: List[int], num_sample: int, radius: int, min_radius: int, invalid):
+    '''
+    :param gt_xyz:
+    :param gt_valid:
+    :param pred_xyz:
+    :param pred_valid:
+    :param scales: list of down-sample scales
+    :param num_sample:
+    :param radius:
+    :param min_radius:
+    :return: list of avg relative normal errors under each scale
+    '''
+    ret = []
+    for sc in scales:
+        ds_gt_valid, ds_gt_xyz, ds_pred_valid, ds_pred_xyz = downsample(sc, gt_valid, [gt_xyz, pred_valid, pred_xyz])
+        err, gt_pair_valid, pred_pair_valid, _ = get_rel_normal_err_heatmap_idx(ds_gt_xyz, ds_gt_valid, ds_pred_xyz, ds_pred_valid, num_sample, max(radius // sc, min_radius))
+        match invalid:
+            case 'penalty':
+                err = torch.where(gt_pair_valid & ~pred_pair_valid, torch.pi, err)
+                err = err[gt_pair_valid]
+            case 'ignore':
+                err = err[gt_pair_valid & pred_pair_valid]
+            case _:
+                raise ValueError()
+
+        if err.shape[0] > 0:
+            scalar_err = err.mean().item()
+            ret.append(scalar_err)
+    if len(ret) == 0:
+        ret = [0]
+    return ret
+
+
+def rel_normal(gt_xyz, gt_valid, pred_xyz, pred_valid, cfg=None, **kwargs):
+    if cfg is None:
+        cfg = DEFAULT_CONFIG | kwargs
+    device_args = {k:v for k,v in cfg.items() if k == 'device'}
+    cfg.pop('device', None)
+    gt_xyz = reformat_as_torch_tensor(gt_xyz, **device_args)
+    gt_valid = reformat_as_torch_tensor(gt_valid, **device_args)
+    pred_xyz = reformat_as_torch_tensor(pred_xyz, **device_args)
+    pred_valid = reformat_as_torch_tensor(pred_valid, **device_args)
+    return np.mean(get_multi_scale_rel_normal_err(gt_xyz, gt_valid, pred_xyz, pred_valid, **cfg))
+
+
+def compute_rel_normal(pred_depth: np.ndarray, pred_intr: np.ndarray, pred_valid: np.ndarray,
+                       gt_depth: np.ndarray, gt_intr: np.ndarray, gt_valid: np.ndarray) -> float:
+    '''
+    :param pred_depth: shape (H, W)
+    :param pred_intr: shape (4,), [fx, fy, cx, cy]
+    :param pred_valid: shape (H, W), dtype: np.bool_
+    :param gt_depth: shape (H, W)
+    :param gt_intr: shape (4,), [fx, fy, cx, cy]
+    :param gt_valid: shape (H, W), dtype: np.bool_
+    :return: SAWA-H value
+    '''
+    err = rel_normal(
+        depth_to_xyz(gt_intr, gt_depth), gt_valid,
+        depth_to_xyz(pred_intr, pred_depth), pred_valid,
+    )
+    return err

evalmde/metrics/sawa_h.py

@@ -0,0 +1,45 @@
+import numpy as np
+
+from evalmde.utils.proj import depth_to_xyz
+from evalmde.utils.depth import align
+from evalmde.utils.torch import reformat_as_torch_tensor
+from evalmde.metrics.standard import rel_depth, delta0125
+from evalmde.metrics.boundary import boundary_f1
+from evalmde.metrics.rel_normal import rel_normal as rel_normal_func
+
+
+def compute_sawa_h(pred_depth: np.ndarray, pred_intr: np.ndarray, pred_valid: np.ndarray,
+                   gt_depth: np.ndarray, gt_intr: np.ndarray, gt_valid: np.ndarray) -> float:
+    '''
+    :param pred_depth: shape (H, W)
+    :param pred_intr: shape (4,), [fx, fy, cx, cy]
+    :param pred_valid: shape (H, W), dtype: np.bool_
+    :param gt_depth: shape (H, W)
+    :param gt_intr: shape (4,), [fx, fy, cx, cy]
+    :param gt_valid: shape (H, W), dtype: np.bool_
+    :return: SAWA-H value
+    '''
+    wkdr__no_align = 1 - rel_depth(pred_depth, gt_depth, gt_valid)[1]
+    delta0125__disparity_af_clip_by_0 = 1 - delta0125(align(
+        1 / reformat_as_torch_tensor(pred_depth),
+        reformat_as_torch_tensor(gt_depth),
+        reformat_as_torch_tensor(gt_valid),
+        'disparity_affine_clip_by_0'
+    ), gt_depth, gt_valid)[1]
+    delta0125__depth_af_lst_sq_clip_by_0 = 1 - delta0125(align(
+        reformat_as_torch_tensor(pred_depth),
+        reformat_as_torch_tensor(gt_depth),
+        reformat_as_torch_tensor(gt_valid),
+        'depth_affine_lst_sq_clip_by_0'
+    ), gt_depth, gt_valid)[1]
+    boundary__no_align = boundary_f1(
+        reformat_as_torch_tensor(pred_depth),
+        reformat_as_torch_tensor(gt_depth),
+        reformat_as_torch_tensor(gt_valid)
+    )[1]
+    rel_normal = rel_normal_func(
+        depth_to_xyz(gt_intr, gt_depth), gt_valid,
+        depth_to_xyz(pred_intr, pred_depth), pred_valid,
+    )
+    err = 3.65 * wkdr__no_align + 0.18 * delta0125__disparity_af_clip_by_0 + 0.01 * delta0125__depth_af_lst_sq_clip_by_0 + 0.20 * boundary__no_align + 1.94 * rel_normal
+    return err

evalmde/metrics/standard.py

@@ -0,0 +1,214 @@
+# source: https://github.com/YvanYin/Metric3D/blob/main/mono/utils/avg_meter.py
+import torch
+
+
+def reformat_input(x):
+    if not isinstance(x, torch.Tensor):
+        x = torch.from_numpy(x)
+    x = x.to(torch.float)
+    device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
+    x = x.to(device)
+    return x
+
+
+def absrel_pnt(pred, target, mask):
+    pred, target, mask = reformat_input(pred), reformat_input(target), reformat_input(mask)
+    assert pred.dim() == 3 and target.dim() == 3 and mask.dim() == 2
+    if mask.sum() == 0:
+        return None, None
+
+    dist_gt = torch.norm(target, dim=-1)
+    dist_err = torch.norm(pred - target, dim=-1)
+    err_heatmap = dist_err / (dist_gt + (1e-10)) * mask
+    err_heatmap[mask < .5] = 0
+    err = err_heatmap.sum() / mask.sum()
+    return err_heatmap.cpu().numpy(), err.item()
+
+
+def rel_depth(pred, target, mask):
+    pred, target, mask = reformat_input(pred), reformat_input(target), reformat_input(mask)
+    assert pred.dim() == 2 and target.dim() == 2 and mask.dim() == 2
+    if mask.sum() == 0:
+        return None, None
+    mask = mask > .5
+    p, t = pred[mask], target[mask]
+    device = p.device
+    N = p.shape[0]
+    M = int(1e7)
+    i = torch.randint(0, N, (M,), device=device, dtype=torch.long)
+    j = torch.randint(0, N, (M,), device=device, dtype=torch.long)
+    correct = (p[i] < p[j]) == (t[i] < t[j])
+    return None, correct.float().mean().item()
+
+
+def absrel(pred, target, mask):
+    pred, target, mask = reformat_input(pred), reformat_input(target), reformat_input(mask)
+    assert pred.dim() == 2 and target.dim() == 2 and mask.dim() == 2
+    if mask.sum() == 0:
+        return None, None
+
+    t_m = target * mask
+    p_m = pred * mask
+    t_m[mask < .5] = 0
+    p_m[mask < .5] = 0
+
+    err_heatmap = torch.abs(t_m - p_m) / (t_m + 1e-10)  # (H, W)
+    err = err_heatmap.sum() / mask.sum()
+    return err_heatmap.cpu().numpy(), err.item()
+
+
+def rmse(pred, target, mask):
+    pred, target, mask = reformat_input(pred), reformat_input(target), reformat_input(mask)
+    assert pred.dim() == 2 and target.dim() == 2 and mask.dim() == 2
+    if mask.sum() == 0:
+        return None, None
+
+    t_m = target * mask
+    p_m = pred * mask
+    t_m[mask < .5] = 0
+    p_m[mask < .5] = 0
+
+    err_heatmap = (t_m - p_m) ** 2  # (H, W)
+    err = torch.sqrt(err_heatmap.sum() / mask.sum())
+    return err_heatmap.cpu().numpy(), err.item()
+
+
+def rmse_log(pred, target, mask):
+    pred, target, mask = reformat_input(pred), reformat_input(target), reformat_input(mask)
+    assert pred.dim() == 2 and target.dim() == 2 and mask.dim() == 2
+    if mask.sum() == 0:
+        return None, None
+
+    t_m = target * mask
+    p_m = pred * mask
+    t_m[mask < .5] = 0
+    p_m[mask < .5] = 0
+
+    err_heatmap = ((torch.log10(p_m+1e-10) - torch.log10(t_m+1e-10)) * mask) ** 2  # (H, W)
+    err = torch.sqrt(err_heatmap.sum() / mask.sum())
+    return err_heatmap.cpu().numpy(), err.item()
+
+
+def delta1(pred, target, mask):
+    pred, target, mask = reformat_input(pred), reformat_input(target), reformat_input(mask)
+    assert pred.dim() == 2 and target.dim() == 2 and mask.dim() == 2
+    if mask.sum() == 0:
+        return None, (None, None, None)
+
+    t_m = target * mask
+    p_m = pred
+
+    gt_pred = t_m / (p_m + 1e-10)  # (H, W)
+    pred_gt = p_m / (t_m + 1e-10)  # (H, W)
+    gt_pred_gt = torch.stack([gt_pred, pred_gt], dim=-1)  # (H, W, 2)
+    ratio_max = torch.amax(gt_pred_gt, dim=-1)  # (H, W)
+    err_heatmap = (ratio_max - 1) * mask  # (H, W)
+    ratio_max[mask < .5] = 99999
+
+    delta_1_sum = torch.sum(ratio_max < 1.25)
+    delta_2_sum = torch.sum(ratio_max < 1.25 ** 2)
+    delta_3_sum = torch.sum(ratio_max < 1.25 ** 3)
+    return err_heatmap.cpu().numpy(), (delta_1_sum / mask.sum()).item()
+
+
+def delta0125(pred, target, mask):
+    pred, target, mask = reformat_input(pred), reformat_input(target), reformat_input(mask)
+    assert pred.dim() == 2 and target.dim() == 2 and mask.dim() == 2
+    if mask.sum() == 0:
+        return None, (None, None, None)
+
+    t_m = target * mask
+    p_m = pred
+
+    gt_pred = t_m / (p_m + 1e-10)  # (H, W)
+    pred_gt = p_m / (t_m + 1e-10)  # (H, W)
+    gt_pred_gt = torch.stack([gt_pred, pred_gt], dim=-1)  # (H, W, 2)
+    ratio_max = torch.amax(gt_pred_gt, dim=-1)  # (H, W)
+    err_heatmap = (ratio_max - 1) * mask  # (H, W)
+    ratio_max[mask < .5] = 99999
+
+    delta_sum = torch.sum(ratio_max < 1.25 ** 0.125)
+    return err_heatmap.cpu().numpy(), (delta_sum / mask.sum()).item()
+
+
+def delta2(pred, target, mask):
+    pred, target, mask = reformat_input(pred), reformat_input(target), reformat_input(mask)
+    assert pred.dim() == 2 and target.dim() == 2 and mask.dim() == 2
+    if mask.sum() == 0:
+        return None, (None, None, None)
+
+    t_m = target * mask
+    p_m = pred
+
+    gt_pred = t_m / (p_m + 1e-10)  # (H, W)
+    pred_gt = p_m / (t_m + 1e-10)  # (H, W)
+    gt_pred_gt = torch.stack([gt_pred, pred_gt], dim=-1)  # (H, W, 2)
+    ratio_max = torch.amax(gt_pred_gt, dim=-1)  # (H, W)
+    err_heatmap = (ratio_max - 1) * mask  # (H, W)
+    ratio_max[mask < .5] = 99999
+
+    delta_1_sum = torch.sum(ratio_max < 1.25)
+    delta_2_sum = torch.sum(ratio_max < 1.25 ** 2)
+    delta_3_sum = torch.sum(ratio_max < 1.25 ** 3)
+    return err_heatmap.cpu().numpy(), (delta_2_sum / mask.sum()).item()
+
+
+def delta3(pred, target, mask):
+    pred, target, mask = reformat_input(pred), reformat_input(target), reformat_input(mask)
+    assert pred.dim() == 2 and target.dim() == 2 and mask.dim() == 2
+    if mask.sum() == 0:
+        return None, (None, None, None)
+
+    t_m = target * mask
+    p_m = pred
+
+    gt_pred = t_m / (p_m + 1e-10)  # (H, W)
+    pred_gt = p_m / (t_m + 1e-10)  # (H, W)
+    gt_pred_gt = torch.stack([gt_pred, pred_gt], dim=-1)  # (H, W, 2)
+    ratio_max = torch.amax(gt_pred_gt, dim=-1)  # (H, W)
+    err_heatmap = (ratio_max - 1) * mask  # (H, W)
+    ratio_max[mask < .5] = 99999
+
+    delta_1_sum = torch.sum(ratio_max < 1.25)
+    delta_2_sum = torch.sum(ratio_max < 1.25 ** 2)
+    delta_3_sum = torch.sum(ratio_max < 1.25 ** 3)
+    return err_heatmap.cpu().numpy(), (delta_3_sum / mask.sum()).item()
+
+
+def log10(pred, target, mask):
+    pred, target, mask = reformat_input(pred), reformat_input(target), reformat_input(mask)
+    assert pred.dim() == 2 and target.dim() == 2 and mask.dim() == 2
+    if mask.sum() == 0:
+        return None, None
+
+    t_m = target * mask
+    p_m = pred * mask
+    t_m[mask < .5] = 0
+    p_m[mask < .5] = 0
+
+    err_heatmap = torch.abs((torch.log10(p_m+1e-10) - torch.log10(t_m+1e-10)) * mask)
+    err = err_heatmap.sum() / mask.sum()
+    return err_heatmap.cpu().numpy(), err.item()
+
+
+def rmse_log_si(pred, target, mask):  # RMSE (log, scale-invariant)
+    # https://github.com/prs-eth/Marigold/blob/main/src/util/metric.py#L175
+    depth_pred, depth_gt, valid_mask = reformat_input(pred), reformat_input(target), reformat_input(mask)
+    assert depth_pred.dim() == 2 and depth_gt.dim() == 2 and valid_mask.dim() == 2
+    if valid_mask.sum() == 0:
+        return None, None
+
+    valid_mask = valid_mask > .5
+    diff = torch.log(depth_pred) - torch.log(depth_gt)
+    if valid_mask is not None:
+        diff[~valid_mask] = 0
+        n = valid_mask.sum((-1, -2))
+    else:
+        n = depth_gt.shape[-2] * depth_gt.shape[-1]
+
+    diff2 = torch.pow(diff, 2)
+
+    first_term = torch.sum(diff2, (-1, -2)) / n
+    second_term = torch.pow(torch.sum(diff, (-1, -2)), 2) / (n**2)
+    loss = torch.sqrt(torch.mean(first_term - second_term))
+    return None, loss.item()

evalmde/metrics/triangle.py

@@ -0,0 +1,93 @@
+import torch
+import torch.nn.functional as F
+
+
+'''
+VERTEX_SLICES:
+0 2
+1 3
+'''
+VERTEX_SLICES = [
+    (slice(None, -1), slice(None, -1)),
+    (slice(1, None), slice(None, -1)),
+    (slice(None, -1), slice(1, None)),
+    (slice(1, None), slice(1, None)),
+]
+TRIANGLE_SLICES = [
+    [VERTEX_SLICES[0], VERTEX_SLICES[1], VERTEX_SLICES[2]],
+    [VERTEX_SLICES[2], VERTEX_SLICES[0], VERTEX_SLICES[3]],
+    [VERTEX_SLICES[0], VERTEX_SLICES[1], VERTEX_SLICES[3]],
+    [VERTEX_SLICES[2], VERTEX_SLICES[1], VERTEX_SLICES[3]],
+]
+NUM_TRIANGLE = len(TRIANGLE_SLICES)
+
+
+@torch.no_grad()
+def _fetch_pixel_val(x: torch.Tensor, vertex_slice):
+    '''
+    :param x: shape (H, W, ...)
+    :param vertex_slice:
+    :return: shape (H - 1, W - 1, ...)
+    '''
+    return x[vertex_slice[0], vertex_slice[1]]
+
+
+@torch.no_grad()
+def get_triangle_valid(valid: torch.Tensor):
+    '''
+    a triangle is valid if all vertices are valid
+    :param valid: shape (H, W)
+    :return: triangle_valid
+        triangle_valid: shape (H - 1, W - 1, NUM_TRIANGLE)
+    '''
+    H, W = valid.shape
+    device = valid.device
+    ret = torch.empty((H - 1, W - 1, NUM_TRIANGLE), dtype=torch.bool, device=device)
+    for i, TRIANGLE_SLICE in enumerate(TRIANGLE_SLICES):
+        ret[..., i] = _fetch_pixel_val(valid, TRIANGLE_SLICE[0]) & \
+                      _fetch_pixel_val(valid, TRIANGLE_SLICE[1]) & \
+                      _fetch_pixel_val(valid, TRIANGLE_SLICE[2])
+    return ret
+
+
+@torch.no_grad()
+def get_triangle_normal(xyz: torch.Tensor):
+    '''
+    :param xyz: shape (H, W, 3)
+    :return: normal, normal_valid
+        normal: shape (H - 1, W - 1, NUM_TRIANGLE, 3)
+        normal_valid: shape (H - 1, W - 1, NUM_TRIANGLE)
+    '''
+    H, W = xyz.shape[:2]
+    device = xyz.device
+    dtype = xyz.dtype
+    normal = torch.empty((H - 1, W - 1, NUM_TRIANGLE, 3), dtype=dtype, device=device)
+    normal_valid = torch.empty((H - 1, W - 1, NUM_TRIANGLE), dtype=torch.bool, device=device)
+    for i, TRIANGLE_SLICE in enumerate(TRIANGLE_SLICES):
+        normal[..., i, :] = torch.linalg.cross(
+            F.normalize(_fetch_pixel_val(xyz, TRIANGLE_SLICE[1]) - _fetch_pixel_val(xyz, TRIANGLE_SLICE[0]), dim=-1),
+            F.normalize(_fetch_pixel_val(xyz, TRIANGLE_SLICE[2]) - _fetch_pixel_val(xyz, TRIANGLE_SLICE[0]), dim=-1),
+            dim=-1
+        )
+        vec_norm = torch.norm(normal[..., i, :], dim=-1)  # (H - 1, W - 1)
+        normal_valid[..., i] = vec_norm > 1e-5
+        normal[..., i, :] /= vec_norm.clamp(min=1e-5).unsqueeze(-1)
+    return normal, normal_valid
+
+
+@torch.no_grad()
+def get_triangle_normal_and_valid(xyz: torch.Tensor, valid: torch.Tensor, flatten: bool = True):
+    '''
+    if gt_d and depth_layer are not None, filter out triangle across depth layers
+    :param xyz:
+    :param valid:
+    :param flatten:
+    :return: normal, valid
+    '''
+    normal, normal_valid = get_triangle_normal(xyz)
+    tri_valid = get_triangle_valid(valid)
+    valid = normal_valid & tri_valid
+    if flatten:
+        normal = normal.reshape(-1, 3)
+        valid = valid.reshape(-1)
+    return normal, valid

evalmde/utils/blender.py

@@ -0,0 +1,213 @@
+import os
+import shutil
+
+import bpy
+import mathutils
+import numpy as np
+import OpenEXR
+import Imath
+from scipy.spatial.transform import Rotation as scipy_Rotation
+
+from evalmde.utils.constants import VALID_DEPTH_LB, VALID_DEPTH_UB
+from evalmde.utils.common import pathlib_file
+from evalmde.utils.depth import get_depth_valid
+from evalmde.utils.image import imread_rgb, imwrite_rgb
+
+
+def bpy_set_tmp_dir(tmp_dir):
+    tmp_dir = pathlib_file(tmp_dir)
+    tmp_dir.mkdir(parents=True, exist_ok=True)
+    bpy.context.preferences.filepaths.temporary_directory = str(tmp_dir)
+
+
+def bpy_create_cam(cam_name, cam_pose, fx, fy, cx, cy, w, h):
+    cam_data = bpy.data.cameras.new(name=cam_name)
+    cam_data.sensor_height = cam_data.sensor_width * h / w
+    cam_data.lens = (fx + fy) / 2 * cam_data.sensor_width / w
+    cam_data.shift_x = (w / 2 - cx) / w
+    cam_data.shift_y = (cy - h / 2) / h
+
+    cam_object = bpy.data.objects.new(cam_name, cam_data)
+    bpy.context.collection.objects.link(cam_object)
+    cam_object.matrix_world = mathutils.Matrix([cam_pose[0], cam_pose[1], cam_pose[2], cam_pose[3]])
+    return cam_object
+
+
+def bpy_add_ambient_light(energy=1.0):
+    world = bpy.data.worlds.new("AmbientWorld")
+    bpy.context.scene.world = world
+    world.use_nodes = True
+    bg = world.node_tree.nodes["Background"]
+    bg.inputs[0].default_value = (1, 1, 1, 1)
+    bg.inputs[1].default_value = energy
+
+
+def bpy_enable_gpu(device_type="CUDA"):
+    prefs = bpy.context.preferences
+    cprefs = prefs.addons['cycles'].preferences
+    cprefs.compute_device_type = device_type  # "CUDA", "OPTIX", "METAL", "HIP"
+    cprefs.get_devices()  # Initialize devices
+    for device in cprefs.devices:
+        device.use = True
+    bpy.context.scene.cycles.device = 'GPU'
+
+
+def bpy_setup_rgb_render():
+    bpy.context.scene.use_nodes = True
+    tree = bpy.context.scene.node_tree
+    tree.nodes.clear()
+    render_layers = tree.nodes.new(type='CompositorNodeRLayers')
+    rgb_output = tree.nodes.new(type='CompositorNodeOutputFile')
+    rgb_output.label = 'RGB Output'
+    rgb_output.base_path = ''
+    rgb_output.format.file_format = 'PNG'
+    rgb_output.file_slots[0].use_node_format = True
+    rgb_output.file_slots[0].save_as_render = True
+    tree.links.new(render_layers.outputs['Image'], rgb_output.inputs[0])
+    return rgb_output
+
+
+def bpy_setup_depth_render():
+    bpy.context.scene.use_nodes = True
+    tree = bpy.context.scene.node_tree
+    tree.nodes.clear()
+    render_layers = tree.nodes.new(type='CompositorNodeRLayers')
+    depth_output = tree.nodes.new(type='CompositorNodeOutputFile')
+    depth_output.label = 'Depth Output'
+    depth_output.base_path = ''
+    depth_output.format.file_format = 'OPEN_EXR'
+    depth_output.file_slots[0].use_node_format = True
+    depth_output.file_slots[0].save_as_render = True
+    bpy.context.view_layer.use_pass_z = True
+    bpy.context.scene.view_layers["ViewLayer"].use_pass_z = True
+    tree.links.new(render_layers.outputs['Depth'], depth_output.inputs[0])
+    return depth_output
+
+
+def bpy_setup_rgbd_render():
+    bpy.context.scene.use_nodes = True
+    tree = bpy.context.scene.node_tree
+    tree.nodes.clear()
+
+    # Add Render Layers node to get passes
+    render_layers = tree.nodes.new(type='CompositorNodeRLayers')
+
+    # Add Output File node to save the EXR
+    depth_output = tree.nodes.new(type='CompositorNodeOutputFile')
+    depth_output.label = 'Depth Output'
+    depth_output.base_path = ''
+    depth_output.format.file_format = 'OPEN_EXR'
+    depth_output.file_slots[0].use_node_format = True
+    depth_output.file_slots[0].save_as_render = True
+
+    # Output for RGB
+    rgb_output = tree.nodes.new(type='CompositorNodeOutputFile')
+    rgb_output.label = 'RGB Output'
+    rgb_output.base_path = ''
+    rgb_output.format.file_format = 'PNG'
+    rgb_output.file_slots[0].use_node_format = True
+    rgb_output.file_slots[0].save_as_render = True
+
+    # Enable the Z (Depth) pass
+    bpy.context.view_layer.use_pass_z = True
+    bpy.context.scene.view_layers["ViewLayer"].use_pass_z = True
+
+    # Link the depth pass output from the render layers node
+    tree.links.new(render_layers.outputs['Depth'], depth_output.inputs[0])
+    tree.links.new(render_layers.outputs['Image'], rgb_output.inputs[0])
+
+    return depth_output, rgb_output
+
+
+def save_depth_from_exr(filepath, h, w):
+    exr_file = OpenEXR.InputFile(filepath)
+
+    dw = exr_file.header()['dataWindow']
+    size = (dw.max.x - dw.min.x + 1, dw.max.y - dw.min.y + 1)
+    assert size == (w, h), f"Expected {(w, h)}, got {size}"
+
+    pt = Imath.PixelType(Imath.PixelType.FLOAT)
+    channels = exr_file.channels(["R", "G", "B"], pt)
+    depth = [np.frombuffer(c, dtype=np.float32).reshape(size[1], size[0]) for c in channels]
+    assert np.all(depth[0] == depth[1]) and np.all(depth[0] == depth[2])
+    return depth[0]
+
+
+def bpy_create_directional_light(src: np.ndarray, dst: np.ndarray, energy=5.0, name='Sun'):
+    light_data = bpy.data.lights.new(name=name, type='SUN')
+    light_data.energy = energy
+    light_obj = bpy.data.objects.new(name=name, object_data=light_data)
+    bpy.context.collection.objects.link(light_obj)
+    light_obj.location = (float(src[0]), float(src[1]), float(src[2]))
+
+    direction = dst - src
+    rot_axis = np.cross(np.array([0, 0, -1.]), direction)
+    if np.linalg.norm(rot_axis) < 1e-5:
+        rot_axis = np.array([1., 0, 0])
+    rot_axis /= np.linalg.norm(rot_axis)
+    rot_ang = np.arccos(np.clip(((direction / np.linalg.norm(direction)) * np.array([0, 0, -1.])).sum(), -1, 1))
+    rot_euler = scipy_Rotation.from_rotvec(rot_ang * rot_axis, degrees=False).as_euler('xyz', degrees=False)
+    light_obj.rotation_euler = (float(rot_euler[0]), float(rot_euler[1]), float(rot_euler[2]))
+
+
+def bpy_render_rgb(cam_object, h, w, num_sample, rgb_node, output_root, out_name):
+    bpy.context.scene.cycles.samples = num_sample
+    bpy.context.scene.render.resolution_x = w
+    bpy.context.scene.render.resolution_y = h
+
+    bpy.context.scene.camera = cam_object
+    rgb_node.base_path = str(output_root)
+    rgb_node.file_slots[0].path = f"image_{out_name}-"
+    bpy.ops.render.render(write_still=True)
+
+
+def bpy_render_rgbd(cam_object, h, w, num_sample, depth_node, rgb_node, output_root, out_name):
+    bpy.context.scene.cycles.samples = num_sample
+    bpy.context.scene.render.resolution_x = w
+    bpy.context.scene.render.resolution_y = h
+
+    bpy.context.scene.camera = cam_object
+    depth_node.base_path = str(output_root)
+    depth_node.file_slots[0].path = f"depth_{out_name}_"
+    rgb_node.base_path = str(output_root)
+    rgb_node.file_slots[0].path = f"image_{out_name}-"
+    bpy.ops.render.render(write_still=True)
+    exr_path = os.path.join(str(output_root), f"depth_{out_name}_0001.exr")
+    depth_np = save_depth_from_exr(exr_path, h, w)
+    np.save(os.path.join(str(output_root), f"depth_{out_name}.npy"), depth_np)
+    os.remove(exr_path)
+
+
+def bpy_render_rgb_and_filter_invalid(cam_object, h, w, num_sample, depth_node, rgb_node, output_root, out_name, bkg_color, valid_depth_lb=VALID_DEPTH_LB, valid_depth_ub=VALID_DEPTH_UB, save_depth=False):
+    '''
+    :param cam_object:
+    :param h:
+    :param w:
+    :param num_sample:
+    :param depth_node:
+    :param rgb_node:
+    :param output_root:
+    :param out_name:
+    :param bkg_color: list of 3 integers, [0, 255]
+    :param valid_depth_lb:
+    :param valid_depth_ub:
+    :param save_depth:
+    :return:
+    '''
+    bpy_render_rgbd(cam_object, h, w, num_sample, depth_node, rgb_node, output_root, out_name)
+
+    img_f = pathlib_file(os.path.join(str(output_root), f"image_{out_name}-0001.png"))
+    depth_f = pathlib_file(os.path.join(str(output_root), f"depth_{out_name}.npy"))
+    output_root = pathlib_file(output_root)
+    tmp_dir = output_root.parent / f'{output_root.name}__tmp'
+    tmp_dir.mkdir(parents=True, exist_ok=True)
+
+    img = imread_rgb(img_f)
+    depth = np.load(depth_f)
+    shutil.move(img_f, tmp_dir / img_f.name)
+    if not save_depth:
+        shutil.move(depth_f, tmp_dir / depth_f.name)
+    img[~get_depth_valid(depth, valid_depth_lb, valid_depth_ub)] = np.array(bkg_color)
+    imwrite_rgb(output_root / f'image_{out_name}.png', img)
+    os.remove(tmp_dir / img_f.name)
+    os.remove(tmp_dir / depth_f.name)

evalmde/utils/common.py

@@ -0,0 +1,60 @@
+from pathlib import Path
+from datetime import datetime
+
+import shortuuid
+from omegaconf import DictConfig
+
+
+def flatten_dict_cfg(cfg):  # [dict | DictConfig]) -> DictConfig:
+    ret = {}
+    if isinstance(cfg, dict):
+        cfg = DictConfig(cfg)
+    for k, v in cfg.items():
+        if isinstance(v, DictConfig):
+            ret_v = flatten_dict_cfg(v)
+            for _k, _v in ret_v.items():
+                ret[f'{k}_{_k}'] = _v
+        else:
+            ret[k] = v
+    return DictConfig(ret)
+
+
+def current_time():
+    current_time = datetime.now()
+    readable_time = current_time.strftime("%Y-%m-%d-%H:%M:%S")
+    return readable_time
+
+
+def uuid(length=8):
+    """
+    https://github.com/wandb/client/blob/master/wandb/util.py#L677
+    """
+
+    # ~3t run ids (36**8)
+    run_gen = shortuuid.ShortUUID(alphabet=list("0123456789abcdefghijklmnopqrstuvwxyz"))
+    return run_gen.random(length)
+
+
+def pathlib_file(file_name):
+    if isinstance(file_name, str):
+        file_name = Path(file_name)
+    elif not isinstance(file_name, Path):
+        raise TypeError(f'Please check the type of the filename:{file_name}')
+    return file_name
+
+
+def assign_item_to_dict(d: dict, ks: list, v):
+    '''
+    run d[ks[0]][ks[1]]...[ks[-1]] = v with filling empty keys
+    :param d:
+    :param ks:
+    :param v:
+    :return:
+    '''
+    k = ks[0]
+    if len(ks) == 1:
+        d[k] = v
+    else:
+        if k not in d:
+            d[k] = dict()
+        assign_item_to_dict(d[k], ks[1:], v)

evalmde/utils/constants.py

@@ -0,0 +1,2 @@
+VALID_DEPTH_LB = 1e-2
+VALID_DEPTH_UB = 1e4

evalmde/utils/depth.py

@@ -0,0 +1,132 @@
+from typing import Tuple
+
+import numpy as np
+import torch
+
+from evalmde.utils.constants import VALID_DEPTH_LB, VALID_DEPTH_UB
+from evalmde.utils.torch import reformat_as_torch_tensor
+
+
+def align_depth_least_square(
+    gt_arr: np.ndarray,
+    pred_arr: np.ndarray,
+    valid_mask_arr: np.ndarray,
+    return_scale_shift=True,
+    max_resolution=None,
+):
+    # https://github.com/prs-eth/Marigold/blob/62413d56099d36573b2de1eb8/src/util/alignment.py#L8
+    ori_shape = pred_arr.shape  # input shape
+
+    gt = gt_arr.squeeze()  # [H, W]
+    pred = pred_arr.squeeze()
+    valid_mask = valid_mask_arr.squeeze()
+
+    # Downsample
+    if max_resolution is not None:
+        scale_factor = np.min(max_resolution / np.array(ori_shape[-2:]))
+        if scale_factor < 1:
+            downscaler = torch.nn.Upsample(scale_factor=scale_factor, mode="nearest")
+            gt = downscaler(torch.as_tensor(gt).unsqueeze(0)).numpy()
+            pred = downscaler(torch.as_tensor(pred).unsqueeze(0)).numpy()
+            valid_mask = (
+                downscaler(torch.as_tensor(valid_mask).unsqueeze(0).float())
+                .bool()
+                .numpy()
+            )
+
+    assert (
+        gt.shape == pred.shape == valid_mask.shape
+    ), f"{gt.shape}, {pred.shape}, {valid_mask.shape}"
+
+    gt_masked = gt[valid_mask].reshape((-1, 1))
+    pred_masked = pred[valid_mask].reshape((-1, 1))
+
+    # numpy solver
+    _ones = np.ones_like(pred_masked)
+    A = np.concatenate([pred_masked, _ones], axis=-1)
+    X = np.linalg.lstsq(A, gt_masked, rcond=None)[0]
+    scale, shift = X
+
+    aligned_pred = pred_arr * scale + shift
+
+    # restore dimensions
+    aligned_pred = aligned_pred.reshape(ori_shape)
+
+    if return_scale_shift:
+        return aligned_pred, scale, shift
+    else:
+        return aligned_pred
+
+
+def align_affine_lstsq(x: torch.Tensor, y: torch.Tensor, w: torch.Tensor = None) -> Tuple[torch.Tensor, torch.Tensor]:
+    # https://github.com/microsoft/MoGe/blob/a8c37341bc0325ca99b9d57981cc3bb2bd3e255b/moge/utils/alignment.py#L399
+    """
+    Solve `min sum_i w_i * (a * x_i + b - y_i ) ^ 2`, where `a` and `b` are scalars, with respect to `a` and `b` using least squares.
+
+    ### Parameters:
+    - `x: torch.Tensor` of shape (..., N)
+    - `y: torch.Tensor` of shape (..., N)
+    - `w: torch.Tensor` of shape (..., N)
+
+    ### Returns:
+    - `a: torch.Tensor` of shape (...,)
+    - `b: torch.Tensor` of shape (...,)
+    """
+    w_sqrt = torch.ones_like(x) if w is None else w.sqrt()
+    A = torch.stack([w_sqrt * x, torch.ones_like(x)], dim=-1)
+    B = (w_sqrt * y)[..., None]
+    a, b = torch.linalg.lstsq(A, B)[0].squeeze(-1).unbind(-1)
+    return a, b
+
+
+def get_depth_valid(depth, valid_depth_lb=VALID_DEPTH_LB, valid_depth_ub=VALID_DEPTH_UB):
+    if isinstance(depth, np.ndarray):
+        return (~np.isnan(depth)) & (~np.isinf(depth)) & (depth >= valid_depth_lb) & (depth <= valid_depth_ub)
+    elif isinstance(depth, torch.Tensor):
+        return (~torch.isnan(depth)) & (~torch.isinf(depth)) & (depth >= valid_depth_lb) & (depth <= valid_depth_ub)
+    else:
+        raise ValueError(f'{type(depth)=}')
+
+
+def load_data(depth_f, as_torch=False):
+    data = np.load(depth_f)
+    depth, intr, valid = data['depth'], data['intr'], data['valid']
+    depth[~valid] = 1
+    if as_torch:
+        depth = reformat_as_torch_tensor(depth)
+        intr = reformat_as_torch_tensor(intr)
+        valid = reformat_as_torch_tensor(valid)
+    return depth, intr, valid
+
+
+def align(pred, gt, gt_valid, method, return_align_param=False, eps=1e-4):
+    if method == 'no':
+        if return_align_param:
+            return pred, None
+        return pred
+
+    if method == 'depth_affine_lst_sq_clip_by_0':
+        # pred: affine-invariant depth
+        # gt: gt depth
+        # return: aligned depth
+        ret, scale, shift = align_depth_least_square(gt.cpu().numpy(), pred.cpu().numpy(), gt_valid.cpu().numpy())
+        ret = torch.from_numpy(ret).to(device=pred.device, dtype=pred.dtype).clamp_min(eps)
+        if return_align_param:
+            return ret, (float(scale), float(shift))
+        return ret
+
+    if method in ['disparity_affine', 'disparity_affine_clip_by_0']:
+        # pred: predicted affine-invariant disparity
+        # gt: gt depth
+        # return: aligned depth
+        scale, shift = align_affine_lstsq(pred[gt_valid], 1 / gt[gt_valid])
+        pred_disp = pred * scale + shift
+        if method == 'disparity_affine':
+            ret = 1 / pred_disp.clamp_min(1 / gt[gt_valid].max().item())
+        else:
+            ret = 1 / pred_disp.clamp_min(eps)
+        if return_align_param:
+            return ret, (float(scale), float(shift))
+        return ret
+
+    raise NotImplementedError(f'{method=}')

evalmde/utils/depth_to_mesh.py

@@ -0,0 +1,150 @@
+import numpy as np
+import open3d as o3d
+import trimesh
+
+from evalmde.utils.proj import depth_to_xyz, apply_SE3
+
+
+def gen_triangle_v_idx(H, W):
+    pxl_idx = np.arange(H * W).reshape(H, W)
+    triangle_v_idx = np.stack([
+        np.stack([pxl_idx[:-1, :-1], pxl_idx[1:, :-1], pxl_idx[:-1, 1:]], axis=-1),  # (H - 1, W - 1, 3)
+        np.stack([pxl_idx[1:, 1:], pxl_idx[:-1, 1:], pxl_idx[1:, :-1]], axis=-1),  # (H - 1, W - 1, 3)
+    ], axis=-2)  # (H - 1, W - 1, 2, 3)
+    return triangle_v_idx
+
+
+def gen_trimesh_mesh(vs, cs, triangles):
+    mesh_vertices = o3d.utility.Vector3dVector(vs.reshape(-1, 3))
+    mesh_faces = o3d.utility.Vector3iVector(triangles)
+    mesh = o3d.geometry.TriangleMesh(mesh_vertices, mesh_faces)
+    mesh.compute_vertex_normals()
+
+    trimesh_mesh = trimesh.Trimesh(
+        vertices=np.asarray(mesh.vertices),
+        faces=np.asarray(mesh.triangles),
+        vertex_normals=np.asarray(mesh.vertex_normals),
+        vertex_colors=cs.reshape(-1, 3),
+        process=False
+    )
+    material = trimesh.visual.material.PBRMaterial(
+        vertexColors=True,
+        doubleSided=True
+    )
+    trimesh_mesh.visual.material = material
+    return trimesh_mesh
+
+
+def concatenate_mesh_data(mesh_datas):
+    n = 0
+    vs, cs, fs = [], [], []
+    for v, c, f in mesh_datas:
+        vs.append(v)
+        cs.append(c)
+        fs.append(f + n)
+        n += v.shape[0]
+    return np.concatenate(vs, axis=0), np.concatenate(cs, axis=0), np.concatenate(fs, axis=0)
+
+
+def gen_mesh_and_pcd(intr, depth, depth_valid, SE3=np.eye(4), rgb=None, valid_triangle=None, crop_region=None):
+    '''
+    :param intr: shape (4,)
+    :param depth: shape (H, W)
+    :param SE3: shape (4, 4), points coords: apply_SE3(SE3, depth_to_xyz(intr, depth))
+    :param rgb:
+        if rgb.dtype == np.uint8:
+            use rgb / 255
+        else:
+            assert rgb.dtype == np.float32
+            use rgb
+    :param valid_triangle:
+    :param crop_region: [lb_i, ub_i, lb_j, ub_j]
+    :return:
+    '''
+    depth = depth.astype(np.float32)
+    SE3 = SE3.astype(np.float32)
+
+    H, W = depth.shape
+
+    if crop_region is not None and len(crop_region) > 0:
+        lb_i, ub_i, lb_j, ub_j = crop_region
+        region_valid = np.zeros_like(depth_valid)
+        region_valid[lb_i:ub_i, lb_j:ub_j] = True
+        depth_valid = depth_valid & region_valid
+
+    xyz = apply_SE3(SE3, depth_to_xyz(intr, depth))
+
+    # create triangles
+    triangle_v_idx = gen_triangle_v_idx(H, W)
+
+    # compute validity based on xyz validity
+    valid_flattened = depth_valid.reshape(-1)
+    xyz_flattened = xyz.reshape(-1, 3)
+    valid_triangle_vertex = \
+        valid_flattened[triangle_v_idx[..., 0]] & \
+        valid_flattened[triangle_v_idx[..., 1]] & \
+        valid_flattened[triangle_v_idx[..., 2]]  # (H - 1, W - 1, 2)
+    if valid_triangle is None:
+        valid_triangle = valid_triangle_vertex
+    else:
+        valid_triangle = valid_triangle_vertex & valid_triangle
+
+    if rgb is None:
+        vertex_colors = .7 * np.ones_like(xyz_flattened)
+    else:
+        if rgb.dtype == np.uint8:
+            vertex_colors = rgb.reshape(-1, 3).astype(np.float32) / 255.
+        else:
+            assert rgb.dtype == np.float32
+            vertex_colors = rgb.reshape(-1, 3)
+
+    pxl_displayed = np.zeros((H, W), dtype=np.bool_)
+    pxl_displayed[:-1, :-1] |= valid_triangle[..., 0]
+    pxl_displayed[1:, :-1] |= valid_triangle[..., 0]
+    pxl_displayed[:-1, 1:] |= valid_triangle[..., 0]
+    pxl_displayed[1:, 1:] |= valid_triangle[..., 1]
+    pxl_displayed[1:, :-1] |= valid_triangle[..., 1]
+    pxl_displayed[:-1, 1:] |= valid_triangle[..., 1]
+    invisible_to_display = depth_valid & (~pxl_displayed)
+
+    def get_up_xyz(depth):
+        fx, fy, cx, cy = intr[0], intr[1], intr[2], intr[3]
+        v, u = np.meshgrid(np.arange(depth.shape[0]), np.arange(depth.shape[1]), indexing='ij')
+        up_xyz = apply_SE3(SE3, np.stack([
+            np.stack([((u - 1) - cx) / fx * depth, ((v - 1) - cy) / fy * depth, depth], axis=-1),
+            np.stack([((u + 1) - cx) / fx * depth, ((v - 1) - cy) / fy * depth, depth], axis=-1),
+            np.stack([((u - 1) - cx) / fx * depth, ((v + 1) - cy) / fy * depth, depth], axis=-1),
+            np.stack([((u + 1) - cx) / fx * depth, ((v + 1) - cy) / fy * depth, depth], axis=-1),
+        ], axis=-2).reshape(H, W, 2, 2, 3))
+        return up_xyz
+
+    depth_range = 1 / (.5 * (intr[0] + intr[1]))
+    up_xyz_fnt = get_up_xyz((1 - depth_range) * depth)
+    up_xyz_bck = get_up_xyz((1 + depth_range) * depth)
+
+    up_xyz = np.stack([up_xyz_fnt, up_xyz_bck], axis=2).reshape(H, W, 8, 3)  # (H, W, 8, 3)
+    up_vertex_idx = np.arange(H * W * 8).reshape(H, W, 8)
+    up_triangles_to_stack = []
+    for v1, v2, v3, v4 in [
+        [0, 2, 3, 1],
+        [0, 4, 6, 2],
+        [0, 1, 5, 4],
+        [7, 5, 1, 3],
+        [7, 3, 2, 6],
+        [7, 6, 4, 5],
+    ]:
+        up_triangles_to_stack.append(up_vertex_idx[..., [v1, v2, v3]])
+        up_triangles_to_stack.append(up_vertex_idx[..., [v3, v4, v1]])
+    up_triangles = np.stack(up_triangles_to_stack, axis=-2)  # (H, W, -1, 3)
+    up_vertex_colors = np.repeat(vertex_colors.reshape(H, W, 1, 3), 8, axis=-2).reshape(-1, 3)
+
+    xyz_flattened[~valid_flattened] = 0
+    up_xyz[~depth_valid] = 0
+
+    trimesh_mesh = gen_trimesh_mesh(*concatenate_mesh_data([
+        (xyz_flattened, vertex_colors, triangle_v_idx[valid_triangle]),
+        (up_xyz.reshape(-1, 3), up_vertex_colors, up_triangles[invisible_to_display].reshape(-1, 3))
+    ]))
+
+    pcd = gen_trimesh_mesh(up_xyz, up_vertex_colors, up_triangles[depth_valid].reshape(-1, 3))
+    return trimesh_mesh, pcd

evalmde/utils/downsample.py

@@ -0,0 +1,72 @@
+from typing import List
+
+import torch
+import torch.nn.functional as F
+
+from evalmde.utils.proj import th_uv_grid
+
+
+def pad(x: torch.Tensor, sc: int) -> torch.Tensor:
+    '''
+    pad x to bottom and right with 0, so that H % sc == 0 and W % sc == 0
+    :param x: shape (H, W, ...)
+    :param sc: int
+    :return: pad_x
+    '''
+    H, W, C_shape = x.shape[0], x.shape[1], x.shape[2:]
+    x = x.reshape(H, W, -1).permute(2, 0, 1)  # (-1, H, W)
+    pad_H = (sc - H % sc) % sc
+    pad_W = (sc - W % sc) % sc
+    x = F.pad(x, (0, pad_W, 0, pad_H), value=0)  # (-1, H', W')
+    return x.permute(1, 2, 0).reshape((x.shape[-2], x.shape[-1]) + C_shape)
+
+
+def patchify(x: torch.Tensor, sc: int):
+    '''
+    reshape (H, W, ...) to (sc, sc, H / sc, W / sc, ...)
+    :param x: shape (H, W, ...)
+    :param sc: int
+    :return: patched_x
+    '''
+    H, W, C_shape = x.shape[0], x.shape[1], x.shape[2:]
+    assert H % sc == 0 and W % sc == 0, f'can\'t patchify ({x.shape=}, {sc=})'
+    _H, _W = H // sc, W // sc
+    x = x.reshape(_H, sc, _W, sc, -1).permute(1, 3, 0, 2, 4)
+    return x.reshape((sc, sc, _H, _W) + C_shape)
+
+
+def gather(x: torch.Tensor, idx: torch.Tensor):
+    '''
+    :param x: shape (sc, sc, H / sc, W / sc, ...)
+    :param idx: shape (H / sc, W / sc)
+    :return: x[idx[i,j] // sc, idx[i,j] % sc, i, j, ...]
+    '''
+    sc, _, H, W, C_shape = x.shape[0], x.shape[1], x.shape[2], x.shape[3], x.shape[4:]
+    x = x.reshape(sc * sc, H, W, -1)
+    idx = idx[None, :, :, None].repeat(1, 1, 1, x.shape[-1])  # (1, H / sc, W / sc, -1)
+    return torch.gather(x, 0, idx).reshape((H, W) + C_shape)
+
+
+def downsample(ds_sc: int, valid: torch.Tensor, tensors: List[torch.Tensor]) -> List[torch.Tensor]:
+    '''
+    :param ds_sc: downsample scale
+    :param valid: (H, W), dtype: torch.bool
+    :param tensors: list of tensors of shape (H, W, ...)
+    :return: [ds_valid, *ds_tensors]
+        ds_valid: (ds_H, ds_W)
+        ds_tensors: list of tensors of shape (ds_H, ds_W, ...)
+    '''
+    tensor_kwargs = dict(device=valid.device, dtype=torch.float)
+    H, W = valid.shape
+    uv = th_uv_grid(H, W, **tensor_kwargs)  # (H, W, 2)
+    uv = patchify(pad(uv, ds_sc), ds_sc)  # (sc, sc, H / sc, W / sc, 2)
+    ds_H, ds_W = uv.shape[2], uv.shape[3]
+    patch_center = th_uv_grid(ds_H, ds_W, **tensor_kwargs) * ds_sc + .5 * (ds_sc - 1)  # (H / sc, W / sc, 2)
+    valid = patchify(pad(valid, ds_sc), ds_sc)  # (sc, sc, H / sc, W / sc)
+    uv_dst = (uv - patch_center[None, None]).norm(dim=-1)  # (sc, sc, H / sc, W / sc)
+    uv_dst[~valid] = torch.inf  # mask out invalid pixels
+    uv_dst = uv_dst.reshape(-1, uv_dst.shape[-2], uv_dst.shape[-1])  # (sc * sc, H / sc, W / sc)
+    ds_pxl = torch.argmin(uv_dst, dim=0)  # (H / sc, W / sc)
+    valid = gather(valid, ds_pxl)
+    tensors = [gather(patchify(pad(x, ds_sc), ds_sc), ds_pxl) for x in tensors]
+    return [valid] + tensors

evalmde/utils/image.py

@@ -0,0 +1,45 @@
+import cv2
+
+from evalmde.utils.common import pathlib_file
+
+
+def imread_rgb(img_f):
+    return cv2.imread(str(pathlib_file(img_f)))[..., ::-1].copy()
+
+
+def imwrite_rgb(img_f, img, verbose=False):
+    img_f = pathlib_file(img_f)
+    img_f.parent.mkdir(parents=True, exist_ok=True)
+    cv2.imwrite(str(img_f), img[..., ::-1])
+    if verbose:
+        print(f'Saved to {img_f.resolve()}')
+
+
+def resize(img, H=None, W=None, interpolation=cv2.INTER_NEAREST, return_sc=False):
+    '''
+    if both H and W are specified, resize to smaller one while keeping aspect ratio
+    :param img:
+    :param H:
+    :param W:
+    :param interpolation:
+    :param return_sc:
+    :return:
+    '''
+    cur_H, cur_W = img.shape[:2]
+    if (H is not None) and (W is not None):
+        H = int(H)
+        W = int(W)
+        if H / cur_H < W / cur_W:
+            W = None
+        else:
+            H = None
+    if H is not None:
+        H = int(H)
+        img = cv2.resize(img, (int(img.shape[1] / img.shape[0] * H), H), interpolation=interpolation)
+    if W is not None:
+        W = int(W)
+        img = cv2.resize(img, (W, int(img.shape[0] / img.shape[1] * W)), interpolation=interpolation)
+    if return_sc:
+        sc = img.shape[0] / cur_H
+        return img, sc
+    return img

evalmde/utils/np_and_th.py

@@ -0,0 +1,27 @@
+import numpy as np
+import torch
+
+
+def get_shifted_data(data, di, dj):
+    H, W = data.shape
+    shifted_data = data[max(di, 0): H + min(di, 0), max(dj, 0): W + min(dj, 0)]
+    if isinstance(data, np.ndarray):
+        if di < 0:
+            shifted_data = np.concatenate([np.zeros_like(shifted_data[di:]), shifted_data], axis=0)
+        if di > 0:
+            shifted_data = np.concatenate([shifted_data, np.zeros_like(shifted_data[:di])], axis=0)
+        if dj < 0:
+            shifted_data = np.concatenate([np.zeros_like(shifted_data[:, dj:]), shifted_data], axis=1)
+        if dj > 0:
+            shifted_data = np.concatenate([shifted_data, np.zeros_like(shifted_data[:, :dj])], axis=1)
+    elif isinstance(data, torch.Tensor):
+        shifted_data = data[max(di, 0): H + min(di, 0), max(dj, 0): W + min(dj, 0)]
+        if di < 0:
+            shifted_data = torch.cat([torch.zeros_like(shifted_data[di:]), shifted_data], dim=0)
+        if di > 0:
+            shifted_data = torch.cat([shifted_data, torch.zeros_like(shifted_data[:di])], dim=0)
+        if dj < 0:
+            shifted_data = torch.cat([torch.zeros_like(shifted_data[:, dj:]), shifted_data], dim=1)
+        if dj > 0:
+            shifted_data = torch.cat([shifted_data, torch.zeros_like(shifted_data[:, :dj])], dim=1)
+    return shifted_data

evalmde/utils/proj.py

@@ -0,0 +1,41 @@
+import numpy as np
+import torch
+import torch.nn.functional as F
+
+
+def th_uv_grid(H: int, W: int, **tensor_kwargs) -> torch.Tensor:
+    '''
+    :param H: int
+    :param W: int
+    :param tensor_kwargs:
+    :return: (H, W, 2)
+    '''
+    v, u = torch.meshgrid(torch.arange(H).to(**tensor_kwargs), torch.arange(W).to(**tensor_kwargs))
+    return torch.stack([u, v], dim=-1)
+
+
+def depth_to_xyz(intr, depth):
+    '''
+    :param intr: shape (4,)
+    :param depth: shape (H, W)
+    :return: shape (H, W, 3)
+    '''
+    fx, fy, cx, cy = intr[0], intr[1], intr[2], intr[3]
+    if isinstance(depth, np.ndarray):
+        v, u = np.meshgrid(np.arange(depth.shape[0]), np.arange(depth.shape[1]), indexing='ij')
+        x = (u - cx) / fx * depth
+        y = (v - cy) / fy * depth
+        return np.stack([x, y, depth], axis=-1)
+    elif isinstance(depth, torch.Tensor):
+        tensor_kwargs = dict(device=depth.device, dtype=depth.dtype)
+        v, u = torch.meshgrid(torch.arange(depth.shape[0]).to(**tensor_kwargs), torch.arange(depth.shape[1]).to(**tensor_kwargs))
+        x = (u - cx) / fx * depth
+        y = (v - cy) / fy * depth
+        return torch.stack([x, y, depth], dim=-1)
+    else:
+        raise ValueError(f'{type(depth)=}')
+
+
+def apply_SE3(SE3, pnt):
+    assert SE3.shape == (4, 4) and pnt.shape[-1] == 3
+    return (SE3[:3, :3] @ pnt[..., None])[..., 0] + SE3[:3, -1]

evalmde/utils/torch.py

@@ -0,0 +1,26 @@
+from typing import List
+
+import numpy as np
+import torch
+import torch.nn.functional as F
+
+
+@torch.no_grad()
+def get_angle_between(n1: torch.Tensor, n2: torch.Tensor) -> torch.Tensor:
+    '''
+    :param n1: shape (..., 3), norm > 0
+    :param n2: shape (..., 3), norm > 0
+    :return: shape (...), in radius
+    '''
+    return torch.acos((F.normalize(n1, dim=-1) * F.normalize(n2, dim=-1)).sum(dim=-1).clamp(-1, 1))
+
+
+def reformat_as_torch_tensor(x, device=torch.device('cuda') if torch.cuda.is_available() else torch.device('cpu')):
+    if isinstance(x, List):
+        return torch.tensor(x, device=device)
+    elif isinstance(x, np.ndarray):
+        return torch.from_numpy(x).to(device=device)
+    elif isinstance(x, torch.Tensor):
+        return x.to(device=device)
+    else:
+        raise ValueError(f'Unsupported type: {type(x)}')

evalmde/visualization/__init__.py

@@ -0,0 +1,14 @@
+import numpy as np
+
+
+ROT_LIGHT_NUM_LIGHT = 30
+ROT_LIGHT_NUM_LOOP = 3
+
+
+def gen_rot_light__light_pos(num_light, num_loop):
+    theta = np.linspace(0, np.pi, num_light)
+    phi = np.linspace(0, 2 * np.pi * num_loop, num_light)
+    x = np.sin(theta) * np.cos(phi)
+    z = np.sin(theta) * np.sin(phi)
+    y = np.cos(theta)
+    return np.stack([x, y, z], axis=-1)

evalmde/visualization/cfg.py

@@ -0,0 +1,54 @@
+import numpy as np
+
+from evalmde.utils.common import uuid, pathlib_file
+from evalmde.utils.image import imread_rgb
+
+
+def get_intermediate_mesh_f(args):
+    if args.mesh_dir:
+        return args.mesh_dir / f'mesh_{uuid(12)}.glb'
+    return args.root / f'mesh_{uuid(12)}.glb'
+
+
+def get_vis_root(args):
+    root = args.root
+    valid_triangle_name = 'none'
+    if args.valid_triangle_f:
+        valid_triangle_name = str((root / args.valid_triangle_f).resolve().relative_to(root.resolve()))
+        if args.filter_quad:
+            valid_triangle_name = valid_triangle_name + '--filter_quad'
+    return pathlib_file(root) / 'visualization' / valid_triangle_name[:-4] / str((root / args.depth_f).resolve().relative_to(root.resolve()))[:-4].replace('/', '_')
+
+
+def get_crop_region(args):
+    if len(args.crop_region) == 0:
+        return []
+    elif len(args.crop_region) == 4:
+        return args.crop_region
+    else:
+        print(f'Warning: invalid length of crop region (expected 4), {args.crop_region=}. Using [] instead.')
+        return []
+
+
+def get_mesh_vertex_col(args, img_shape):
+    '''
+    :param args:
+    :return: in [0, 1]
+    '''
+    if getattr(args, 'rgb_f', None):
+        rgb = imread_rgb(args.root / args.rgb_f).astype(np.float32) / 255
+    else:
+        rgb = .7 * np.ones(img_shape + (3,), dtype=np.float32)
+        print('no rgb, use gray')
+    return rgb
+
+
+def get_valid_triangle(args, img_shape):
+    if getattr(args, 'valid_triangle_f', None):
+        ret = np.load(args.root / args.valid_triangle_f)['valid_triangle']
+        if args.filter_quad:
+            ret[..., 0] &= ret[..., 1]
+            ret[..., 1] &= ret[..., 0]
+        return ret
+    else:
+        return np.ones((img_shape[0] - 1, img_shape[1] - 1, 2), dtype=np.bool_)

evalmde/visualization/render_contour_line.py

@@ -0,0 +1,256 @@
+import argparse
+import math
+from pathlib import Path
+import json
+
+from PIL import Image
+import cv2
+import torch
+from torchvision import transforms as torch_trans
+import numpy as np
+
+from evalmde.utils.proj import depth_to_xyz
+from evalmde.utils.common import assign_item_to_dict, pathlib_file
+from evalmde.utils.image import resize
+from evalmde.utils.image import imread_rgb
+from evalmde.utils.depth import load_data
+from evalmde.utils.np_and_th import get_shifted_data
+
+
+@torch.no_grad()
+def compute_grid_lb_ub(data, i, j):
+    '''
+    .           .           .
+
+          -------------
+          |(0,0)|(0,1)|
+    .     ------.------     .
+          |(1,0)|(1,1)|
+          -------------
+
+    .           .           .
+    '''
+    if i == 0 and j == 0:
+        x00 = .25 * (data[:-1, :-1] + data[:-1, 1:] + data[1:, :-1] + data[1:, 1:])
+        x01 = .5 * (data[:-1, 1:] + data[1:, 1:])
+        x10 = .5 * (data[1:, :-1] + data[1:, 1:])
+        x11 = 1. * data[1:, 1:]
+    elif i == 0 and j == 1:
+        x00 = .5 * (data[:-1, :-1] + data[1:, :-1])
+        x01 = .25 * (data[:-1, :-1] + data[:-1, 1:] + data[1:, :-1] + data[1:, 1:])
+        x10 = 1. * data[1:, :-1]
+        x11 = .5 * (data[1:, :-1] + data[1:, 1:])
+    elif i == 1 and j == 0:
+        x00 = .5 * (data[:-1, :-1] + data[:-1, 1:])
+        x01 = 1. * data[:-1, 1:]
+        x10 = .25 * (data[:-1, :-1] + data[:-1, 1:] + data[1:, :-1] + data[1:, 1:])
+        x11 = .5 * (data[:-1, 1:] + data[1:, 1:])
+    else:
+        x00 = 1. * data[:-1, :-1]
+        x01 = .5 * (data[:-1, :-1] + data[:-1, 1:])
+        x10 = .5 * (data[:-1, :-1] + data[1:, :-1])
+        x11 = .25 * (data[:-1, :-1] + data[:-1, 1:] + data[1:, :-1] + data[1:, 1:])
+    x = torch.stack([x00, x01, x10, x11], dim=-1)
+    lb, ub = x.min(dim=-1).values, x.max(dim=-1).values  # (H - 1, W - 1), (H - 1, W - 1)
+    del x
+    return lb, ub
+
+
+@torch.no_grad()
+def compute_high_res_idx(high_res_shape, data_low_res, valid, valid_high_res, gap, val_lb):
+    '''
+    :param high_res_shape: (Hu, Wu)
+    :param data_low_res: shape (Hl, Wl)
+    :param valid_high_res: shape (Hl, Wl)
+    :param gap:
+    :param val_lb:
+    :return: res_high_res
+        res_high_res: shape (Hu, Wu)
+    '''
+    Hu, Wu = high_res_shape
+
+    # fill invalid pixels with neighbor means
+    data_low_res = data_low_res.clone()
+    nb_data_sum = torch.zeros_like(data_low_res)
+    nb_data_cnt = torch.zeros_like(data_low_res)
+    for di in [-1, 0, 1]:
+        for dj in [-1, 0, 1]:
+            nb_valid = get_shifted_data(valid, di, dj)
+            nb_data = get_shifted_data(data_low_res, di, dj)
+            nb_data_sum[nb_valid] += nb_data[nb_valid]
+            nb_data_cnt[nb_valid] += 1
+    nb_data_sum[nb_data_cnt < .5] = 0
+    data_low_res[~valid] = (nb_data_sum / nb_data_cnt.clamp(min=1))[~valid]
+
+    data_high_res = torch_trans.functional.resize(data_low_res[None], (Hu, Wu), torch_trans.InterpolationMode.BILINEAR)[0]
+    res_high_res = -torch.ones((Hu, Wu), dtype=torch.int32, device=data_high_res.device)
+
+    for i in range(2):
+        for j in range(2):
+            lb, ub = compute_grid_lb_ub(data_high_res, i, j)
+            lb_i = torch.clip(torch.ceil((lb - val_lb) / gap), min=0).to(res_high_res.dtype)
+            ub_i = torch.clip(torch.floor((ub - val_lb) / gap), max=2e9).to(res_high_res.dtype)
+
+            multi_line_mask = (lb_i < ub_i) | ((lb_i == ub_i) & (res_high_res[1 - i: Hu - i, 1 - j: Wu - j] != -1))
+            single_line_mask = (lb_i == ub_i) & (res_high_res[1 - i: Hu - i, 1 - j: Wu - j] == -1)
+
+            res_high_res[1 - i: Hu - i, 1 - j: Wu - j][single_line_mask] = lb_i[single_line_mask]
+
+            multi_line_upd_idx = torch.clip(torch.round((data_high_res[1 - i: Hu - i, 1 - j: Wu - j] - val_lb) / gap), min=0, max=2e9).to(res_high_res.dtype)
+            multi_line_upd_idx = torch.where(multi_line_upd_idx < lb_i, lb_i, multi_line_upd_idx)
+            multi_line_upd_idx = torch.where(multi_line_upd_idx > ub_i, ub_i, multi_line_upd_idx)
+            multi_line_upd_mask = ((res_high_res[1 - i: Hu - i, 1 - j: Wu - j] == -1) | (
+                torch.abs(data_high_res[1 - i: Hu - i, 1 - j: Wu - j] - (res_high_res[1 - i: Hu - i, 1 - j: Wu - j] * gap + val_lb)) >
+                torch.abs(data_high_res[1 - i: Hu - i, 1 - j: Wu - j] - (multi_line_upd_idx * gap + val_lb))
+            )) & multi_line_mask
+            res_high_res[1 - i: Hu - i, 1 - j: Wu - j][multi_line_upd_mask] = multi_line_upd_idx[multi_line_upd_mask]
+    res_high_res[~valid_high_res] = -1
+    return res_high_res
+
+
+def get_contour_line_gap(data: torch.Tensor, valid: torch.Tensor, num_gap, qt):
+    if not valid.any():
+        return 1
+    qt_lb = data[valid].quantile(qt).item()
+    qt_ub = data[valid].quantile(1 - qt).item()
+    gap = (qt_ub - qt_lb) / (num_gap * (1 - qt * 2))
+    return gap
+
+
+@torch.no_grad()
+def gen_contour_line(rgb_high_res, data, valid, valid_high_res, is_z, num_gap, shift, thickness=0, qt=0.05, colormap=cv2.COLORMAP_JET):
+    device = 'cuda' if torch.cuda.is_available() else 'cpu'
+    data = torch.from_numpy(data).to(device)
+    valid = torch.from_numpy(valid).to(device)
+    valid_high_res = torch.from_numpy(valid_high_res).to(device)
+    if is_z:
+        data = 1 / data
+
+    gap = get_contour_line_gap(data, valid, num_gap, qt)
+    data_lb = data[valid].min().item() if valid.any() else 0
+    val_lb = data_lb + gap * shift
+
+    high_res_shape = rgb_high_res.shape[:2]
+    res_high_res = compute_high_res_idx(high_res_shape, data, valid, valid_high_res, gap, val_lb)
+
+    res = res_high_res.clone()
+    dlt_rng = int(math.floor(thickness))
+    for di in range(-dlt_rng, dlt_rng + 1):
+        for dj in range(-dlt_rng, dlt_rng + 1):
+            if di * di + dj * dj > thickness * thickness:
+                continue
+            nb_res = get_shifted_data(res_high_res, di, dj)
+            upd_mask = get_shifted_data(valid_high_res, di, dj) & (res == -1) & (nb_res != -1) & valid_high_res
+            res[upd_mask] = nb_res[upd_mask]
+
+    if (res != -1).any():
+        res[res != -1] -= res[res != -1].min()
+    res = res.cpu().numpy()
+    num_val = max(2, res.max().item() + 1)
+    valid_high_res = valid_high_res.cpu().numpy()
+
+    base_col = cv2.applyColorMap(np.arange(256, dtype=np.uint8)[None], colormap)[0].astype(np.float32)  # (256, 3)
+    idx = np.arange(num_val, dtype=np.float32) / (num_val - 1) * 255  # (itr,)
+    idx_lb = np.floor(idx).astype(np.int32)  # (itr,)
+    coef_lb = (idx_lb.astype(np.float32) + 1 - idx)[:, None]  # (itr, 1)
+    col = base_col[idx_lb] * coef_lb + base_col[np.clip(idx_lb + 1, a_min=None, a_max=255)] * (1 - coef_lb)  # (itr, 3)
+    col = np.round(col).astype(np.uint8)
+
+    img = np.zeros_like(rgb_high_res)
+    non_colored_mask = valid_high_res & (res == -1)
+    img[non_colored_mask] = rgb_high_res[non_colored_mask]
+    colored_mask = valid_high_res & (res != -1)
+    img[colored_mask] = col[res[colored_mask]]
+    return img, colored_mask, col
+
+
+def pil_ds(img: np.ndarray, H, W):
+    pil_img = Image.fromarray(img, mode='RGB')
+    pil_img = pil_img.resize((W, H), Image.Resampling.LANCZOS)
+    return np.array(pil_img)
+
+
+def render_contour_line_imgs(xyz: np.ndarray, valid: np.ndarray, rgb_low_res: np.ndarray, save_shape, out_root):
+    '''
+    :param xyz:
+    :param valid:
+    :param rgb_low_res:
+    :param save_shape: (H, W)
+    :param out_root:
+    :return:
+    '''
+    # hyperparams
+    texture_strength = 0.8
+    draw_dim_lb = 4 * np.linalg.norm([1920, 1080])
+
+    out_root = pathlib_file(out_root)
+
+    dim = np.linalg.norm(rgb_low_res.shape[:2])
+    us_sc = int(math.ceil(draw_dim_lb / dim))
+    us_shape = (us_sc * rgb_low_res.shape[0], us_sc * rgb_low_res.shape[1])
+    rgb_high_res = np.round(texture_strength * cv2.resize(rgb_low_res, (us_shape[1], us_shape[0]))).astype(np.uint8)
+    valid_high_res = torch_trans.functional.resize(torch.from_numpy(valid)[None], rgb_high_res.shape[:2], torch_trans.InterpolationMode.NEAREST_EXACT)[0].numpy()
+
+    summary = {}
+    for thickness in [5 * np.linalg.norm(rgb_high_res.shape[:2]) / (4 * np.linalg.norm([1920, 1080]))]:
+        for rel_num_gap in [0.015, 0.03, 0.06, 0.09, 0.12, 0.24, 0.42, 0.6]:
+            num_gap = int(dim * rel_num_gap)
+            for shift in [0.5]:
+                imgs, colored_masks, col_maps = {}, {}, {}
+                for i, name in enumerate(['x', 'y', 'z']):
+                    imgs[name], colored_masks[name], col_maps[name] = \
+                        gen_contour_line(rgb_high_res, xyz[..., i], valid, valid_high_res, name == 'z',
+                                         num_gap, shift, thickness)
+
+                    out_f = out_root / name / f'thickness__{thickness:.1f}___num_gap__{num_gap}___shift__{shift:.2f}.png'
+                    out_f.parent.mkdir(parents=True, exist_ok=True)
+                    cv2.imwrite(out_f.as_posix(), pil_ds(imgs[name][us_sc:-us_sc, us_sc:-us_sc, ::-1].copy(), save_shape[0], save_shape[1]))
+                    print(f'Saved to {out_f.resolve()}')
+                    assign_item_to_dict(summary, [name, thickness, num_gap, shift], str(out_f.resolve().relative_to(out_root.resolve())))
+
+                img_xy = rgb_high_res.copy()
+                img_xy[np.logical_and(colored_masks['x'], colored_masks['y'])] = np.round(.5 * (imgs['x'].astype(np.float32) + imgs['y'].astype(np.float32))).astype(np.uint8)[np.logical_and(colored_masks['x'], colored_masks['y'])]
+                img_xy[np.logical_and(colored_masks['x'], np.logical_not(colored_masks['y']))] = imgs['x'][np.logical_and(colored_masks['x'], np.logical_not(colored_masks['y']))]
+                img_xy[np.logical_and(np.logical_not(colored_masks['x']), colored_masks['y'])] = imgs['y'][np.logical_and(np.logical_not(colored_masks['x']), colored_masks['y'])]
+                img_xy[~valid_high_res] = 0
+                # img_xy = caption_img_xy(img_xy, col_maps)
+                out_f = out_root / 'xy' / f'thickness__{thickness:.1f}___num_gap__{num_gap}___shift__{shift:.2f}.png'
+                out_f.parent.mkdir(parents=True, exist_ok=True)
+                cv2.imwrite(out_f.as_posix(), pil_ds(img_xy[us_sc:-us_sc, us_sc:-us_sc, ::-1].copy(), save_shape[0], save_shape[1]))
+                print(f'Saved to {out_f.resolve()}')
+                assign_item_to_dict(summary, ['xy', thickness, num_gap, shift], str(out_f.resolve().relative_to(out_root.resolve())))
+    with (out_root / 'summary.json').open('w') as F:
+        json.dump(summary, F)
+
+
+def get_out_dir(work_dir, depth_f):
+    return work_dir / 'contour_line' / str((work_dir / depth_f).resolve().relative_to(work_dir.resolve()))[:-4].replace('/', '_')
+
+
+def main(args):
+    save_dim_ub = args.save_dim_ub
+
+    root = args.root
+    rgb_f = root / args.rgb_f
+    data_f = root / args.depth_f
+
+    raw_rgb = imread_rgb(rgb_f)
+
+    save_sc = int(math.floor(save_dim_ub / np.linalg.norm(raw_rgb.shape[:2])))
+    save_shape = (save_sc * raw_rgb.shape[0], save_sc * raw_rgb.shape[1])
+
+    depth, intr, valid = load_data(data_f)
+    xyz = depth_to_xyz(intr, depth)
+
+    render_contour_line_imgs(xyz, valid, raw_rgb, save_shape, get_out_dir(root, args.depth_f))
+
+
+if __name__ == "__main__":
+    parser = argparse.ArgumentParser()
+    parser.add_argument("root", type=Path)
+    parser.add_argument("--depth_f", type=str, help='Path to depth file, relative to root.')
+    parser.add_argument('--rgb_f', type=str, nargs='?', const=None, default='rgb.png', help='Path to rgb file, relative to root.')
+    parser.add_argument("--save_dim_ub", type=float, default=np.linalg.norm([1920, 1080]))
+    args = parser.parse_args()
+
+    main(args)

evalmde/visualization/render_textureless_relighting.py

@@ -0,0 +1,130 @@
+import os
+import argparse
+import json
+from pathlib import Path
+
+import bpy
+import mathutils
+import numpy as np
+
+from evalmde.utils.image import imread_rgb, resize, imwrite_rgb
+from evalmde.utils.proj import apply_SE3
+from evalmde.visualization import gen_rot_light__light_pos, ROT_LIGHT_NUM_LIGHT, ROT_LIGHT_NUM_LOOP
+from evalmde.visualization.cfg import (get_intermediate_mesh_f, get_vis_root,
+                                         get_crop_region, get_mesh_vertex_col, get_valid_triangle)
+from evalmde.utils.common import pathlib_file, current_time
+from evalmde.utils.depth_to_mesh import gen_mesh_and_pcd
+from evalmde.utils.depth import load_data
+from evalmde.utils.blender import (bpy_create_cam, bpy_add_ambient_light, bpy_set_tmp_dir, bpy_create_directional_light,
+                                     bpy_setup_rgbd_render, bpy_enable_gpu, bpy_render_rgb_and_filter_invalid)
+
+
+def render(mesh_f, output_root,
+           base_cam_pose, cam_intr_params, ds_ratio, num_sample,
+           light_i, light_src, overwrite, save_blend,
+           ambient, crop_region, cpu):
+    cam_pose = base_cam_pose.copy()
+    light_src_in_cam = light_src.copy()
+    light_src_in_world = apply_SE3(cam_pose, light_src_in_cam)
+    light_dst_in_world = apply_SE3(cam_pose, np.array([0, 0, 0.]))
+
+    cam_pose[..., 1:3] *= -1
+
+    output_root = pathlib_file(output_root)
+    output_root.mkdir(parents=True, exist_ok=True)
+    h, w, fx, fy, cx, cy = cam_intr_params
+
+    bpy.ops.wm.read_factory_settings(use_empty=True)
+    bpy_set_tmp_dir(output_root.parent / f'{output_root.name}__tmp')
+    if not cpu:
+        bpy_enable_gpu()
+
+    assert mesh_f.exists(), mesh_f
+    bpy.ops.import_scene.gltf(filepath=str(mesh_f))
+    for obj in bpy.context.scene.objects:
+        if obj.type == 'MESH':
+            obj.location = (0, 0, 0)
+            obj.scale = (1, 1, 1)
+            obj.rotation_mode = 'XYZ'
+            obj.rotation_euler = mathutils.Euler((-np.pi / 2, 0, 0), 'XYZ')
+
+    # Set render engine and resolution
+    bpy.context.scene.render.engine = 'CYCLES'
+    bpy.context.scene.render.resolution_percentage = 100
+    bpy_create_directional_light(light_src_in_world, light_dst_in_world)
+    bpy_add_ambient_light(ambient)
+
+    depth_node, rgb_node = bpy_setup_rgbd_render()
+    if (not overwrite) and (output_root / f'image_{light_i:06}.png').exists() and \
+            (output_root / f'metadata_{light_i:06}.json').exists():
+        try:
+            with (output_root / f'metadata_{light_i:06}.json').open('r') as F:
+                metadata = json.load(F)
+            if metadata['num_sample'] == num_sample:
+                return
+        except Exception as E:
+            print(f'{light_i=}, {E=}')
+
+    cam_object = bpy_create_cam(f"cam_{light_i:06}", cam_pose,
+                                int(fx), int(fy), int(cx), int(cy), int(w), int(h))
+    bpy_render_rgb_and_filter_invalid(cam_object, int(h), int(w), num_sample, depth_node,
+                                      rgb_node, str(output_root), f'{light_i:06}', [0, 0, 0], save_depth=False)
+
+    if (output_root / f'image_{light_i:06}.png').exists():
+        img = imread_rgb(output_root / f'image_{light_i:06}.png')
+        if crop_region is not None and len(crop_region) > 0:
+            lb_i, ub_i, lb_j, ub_j = crop_region
+            img = img[lb_i:ub_i, lb_j:ub_j]
+        img = resize(img, H=ds_ratio * img.shape[0])
+        imwrite_rgb(output_root / f'image_{light_i:06}.png', img)
+        with (output_root / f'metadata_{light_i:06}.json').open('w') as F:
+            json.dump({'num_sample': num_sample, 'time': current_time()}, F)
+
+    if save_blend and light_i == 0:
+        out_f = output_root / f'{mesh_f.stem}.blend'
+        bpy.ops.wm.save_as_mainfile(filepath=str(out_f))
+        print(f'Saved to {out_f.resolve()}')
+
+
+if __name__ == "__main__":
+    parser = argparse.ArgumentParser()
+    parser.add_argument('root', type=Path)
+    parser.add_argument('--num_sample', type=int, default=256)
+    parser.add_argument('--depth_f', type=str, nargs='?', const=None, default='gt_depth.npz', help='Path to depth file, relative to root.')
+    parser.add_argument('--valid_triangle_f', type=str, nargs='?', const=None, default='valid_triangle.npz', help='Path to valid triangle file, relative to root.')
+    parser.add_argument('--overwrite', action='store_true')
+    parser.add_argument('--save_blend', action='store_true')
+    parser.add_argument('--filter_quad', action='store_true', help='Filter out neighboring square if any of triangle is invalid')
+    parser.add_argument('--ds_ratio', type=float, default=1)
+    parser.add_argument('--ambient', type=float, default=0.2)
+    parser.add_argument('--light_l', type=int)
+    parser.add_argument('--light_r', type=int)
+    parser.add_argument('--crop_region', nargs='*', type=int, default=[], help='Specify 4 integers: lb_i, ub_i, lb_j, ub_j, and only render mesh of [lb_i, ub_i)x[lb_j, ub_j)')
+    parser.add_argument('--mesh_dir', type=Path, nargs='?', const=None, default=None)
+    parser.add_argument('--cpu', action='store_true')
+    args = parser.parse_args()
+
+    root = args.root
+    mesh_f = get_intermediate_mesh_f(args)
+    vis_root = get_vis_root(args)
+
+    crop_region = get_crop_region(args)
+    depth, intr, valid = load_data(root / args.depth_f)
+    rgb = get_mesh_vertex_col(args, depth.shape)
+    valid_triangle = get_valid_triangle(args, depth.shape)
+
+    mesh, pcd = gen_mesh_and_pcd(intr, depth, valid, rgb=rgb, valid_triangle=valid_triangle, crop_region=crop_region)
+    del pcd
+
+    light_pos = gen_rot_light__light_pos(ROT_LIGHT_NUM_LIGHT, ROT_LIGHT_NUM_LOOP)
+
+    mesh_f.parent.mkdir(parents=True, exist_ok=True)
+    # mesh.show()
+    mesh.export(mesh_f)
+    # print(f'Mesh saved to {mesh_f.resolve()}')
+    for light_i in range(args.light_l, args.light_r):
+        render(mesh_f, vis_root / 'textureless_relighting',
+               np.eye(4), list(depth.shape) + intr.tolist(), args.ds_ratio, args.num_sample,
+               light_i, light_pos[light_i], args.overwrite, args.save_blend,
+               args.ambient, crop_region, args.cpu)
+    os.remove(mesh_f)

induce_valid_triangle_from_gt_depth.py

@@ -0,0 +1,29 @@
+from pathlib import Path
+gt_depth_f = Path('sample_data_2/gt_depth.npz')
+valid_triangle_f = Path('sample_data_2/valid_triangle.npz')
+
+
+THRESH = 1.1
+
+import numpy as np
+def induce_valid_triangle_from_gt_depth(gt_depth: np.ndarray, valid: np.ndarray):
+    '''
+    :param gt_depth: shape (H, W)
+    :param valid: shape (H, W)
+    :return: valid_triangle, shape (H - 1, W - 1, 2)
+    '''
+    min_d_0 = np.min(np.stack([gt_depth[:-1, :-1], gt_depth[1:, :-1], gt_depth[:-1, 1:]], axis=0), axis=0)
+    max_d_0 = np.max(np.stack([gt_depth[:-1, :-1], gt_depth[1:, :-1], gt_depth[:-1, 1:]], axis=0), axis=0)
+    valid_0 = valid[:-1, :-1] & valid[:-1, 1:] & valid[1:, :-1] & (max_d_0 <= THRESH * min_d_0)
+
+    min_d_1 = np.min(np.stack([gt_depth[1:, 1:], gt_depth[1:, :-1], gt_depth[:-1, 1:]], axis=0), axis=0)
+    max_d_1 = np.max(np.stack([gt_depth[1:, 1:], gt_depth[1:, :-1], gt_depth[:-1, 1:]], axis=0), axis=0)
+    valid_1 = valid[1:, 1:] & valid[:-1, 1:] & valid[1:, :-1] & (max_d_1 <= THRESH * min_d_1)
+    return np.stack([valid_0, valid_1], axis=-1)
+
+
+from evalmde.utils.depth import load_data
+gt_depth, gt_intr, gt_valid = load_data(gt_depth_f)
+valid_triangle = induce_valid_triangle_from_gt_depth(gt_depth, gt_valid)
+np.savez(valid_triangle_f, valid_triangle=valid_triangle)
+print(f'Saved to {valid_triangle_f.resolve()}')

infinigen5_12612.log

@@ -0,0 +1,256 @@
+============================================
+infinigen5 started at Thu May 14 06:28:59 PM AEST 2026
+Data: /home/ywan0794/EvalMDE/data/infinigen/test_scenes_release_cleaned_final   Output: /home/ywan0794/EvalMDE/output/infinigen5
+============================================
+Thu May 14 18:28:59 2026       
++-----------------------------------------------------------------------------------------+
+| NVIDIA-SMI 550.163.01             Driver Version: 550.163.01     CUDA Version: 12.4     |
+|-----------------------------------------+------------------------+----------------------+
+| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
+| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
+|                                         |                        |               MIG M. |
+|=========================================+========================+======================|
+|   0  NVIDIA H100 NVL                Off |   00000000:61:00.0 Off |                    0 |
+| N/A   51C    P0             98W /  400W |      14MiB /  95830MiB |      0%      Default |
+|                                         |                        |             Disabled |
++-----------------------------------------+------------------------+----------------------+
+                                                                                         
++-----------------------------------------------------------------------------------------+
+| Processes:                                                                              |
+|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
+|        ID   ID                                                               Usage      |
+|=========================================================================================|
+|    0   N/A  N/A      4274      G   /usr/lib/xorg/Xorg                              4MiB |
++-----------------------------------------------------------------------------------------+
+
+============================================
+[depth_pro inference] Thu May 14 06:28:59 PM AEST 2026  env=depth-pro
+============================================
+Found 5 scenes
+  [1/5] indoor__slow_solve-long_focal-2025-05-16-09-19-19___209591a0: shape=(720, 1280)
+Saved 5 predictions to /home/ywan0794/EvalMDE/output/infinigen5/depth_pro
+[INF-OK] depth_pro
+
+============================================
+[marigold inference] Thu May 14 06:29:39 PM AEST 2026  env=marigold
+============================================
+The config attributes {'prediction_type': 'depth'} were passed to MarigoldDepthPipeline, but are not expected and will be ignored. Please verify your model_index.json configuration file.
+Keyword arguments {'prediction_type': 'depth'} are not expected by MarigoldDepthPipeline and will be ignored.
+
+
+
+Found 5 scenes
+  [1/5] indoor__slow_solve-long_focal-2025-05-16-09-19-19___209591a0: shape=(720, 1280)
+Saved 5 predictions to /home/ywan0794/EvalMDE/output/infinigen5/marigold
+[INF-OK] marigold
+
+============================================
+[lotus inference] Thu May 14 06:29:57 PM AEST 2026  env=lotus
+============================================
+
+Found 5 scenes
+  [1/5] indoor__slow_solve-long_focal-2025-05-16-09-19-19___209591a0: shape=(720, 1280)
+Saved 5 predictions to /home/ywan0794/EvalMDE/output/infinigen5/lotus
+[INF-OK] lotus
+
+============================================
+[depthmaster inference] Thu May 14 06:30:10 PM AEST 2026  env=depthmaster
+============================================
+The config attributes {'default_denoising_steps': 10, 'scheduler': ['diffusers', 'DDIMScheduler']} were passed to DepthMasterPipeline, but are not expected and will be ignored. Please verify your model_index.json configuration file.
+Keyword arguments {'default_denoising_steps': 10, 'scheduler': ['diffusers', 'DDIMScheduler']} are not expected by DepthMasterPipeline and will be ignored.
+
+
+An error occurred while trying to fetch /home/ywan0794/EvalMDE/DepthMaster/ckpt/eval/unet: Error no file named diffusion_pytorch_model.safetensors found in directory /home/ywan0794/EvalMDE/DepthMaster/ckpt/eval/unet.
+Defaulting to unsafe serialization. Pass `allow_pickle=False` to raise an error instead.
+Some weights of the model checkpoint at /home/ywan0794/EvalMDE/DepthMaster/ckpt/eval/unet were not used when initializing UNet2DConditionModel: 
+ ['fftblock.conv_s1.weight, fftblock.norm.weight, fftblock.conv_f4.bias, fftblock.conv_f3.bias, fftblock.conv_f1.weight, fftblock.conv_f1.bias, fftblock.conv_f2.weight, fftblock.norm.bias, fftblock.fuse.weight, fftblock.conv_f4.weight, fftblock.fuse.bias, fftblock.conv_s2.bias, fftblock.conv_f3.weight, fftblock.conv_s2.weight, fftblock.conv_f2.bias, fftblock.conv_s1.bias']
+
+Expected types for unet: (<class 'depthmaster.modules.unet_2d_condition_s2.UNet2DConditionModel'>,), got <class 'diffusers.models.unets.unet_2d_condition.UNet2DConditionModel'>.
+An error occurred while trying to fetch /home/ywan0794/EvalMDE/DepthMaster/ckpt/eval/unet: Error no file named diffusion_pytorch_model.safetensors found in directory /home/ywan0794/EvalMDE/DepthMaster/ckpt/eval/unet.
+Defaulting to unsafe serialization. Pass `allow_pickle=False` to raise an error instead.
+Found 5 scenes
+  [1/5] indoor__slow_solve-long_focal-2025-05-16-09-19-19___209591a0: shape=(720, 1280)
+Saved 5 predictions to /home/ywan0794/EvalMDE/output/infinigen5/depthmaster
+[INF-OK] depthmaster
+
+============================================
+[ppd inference] Thu May 14 06:30:28 PM AEST 2026  env=ppd
+============================================
+xFormers not available
+xFormers not available
+Found 5 scenes
+  [1/5] indoor__slow_solve-long_focal-2025-05-16-09-19-19___209591a0: shape=(720, 1280)
+Saved 5 predictions to /home/ywan0794/EvalMDE/output/infinigen5/ppd
+[INF-OK] ppd
+
+============================================
+[da3_mono inference] Thu May 14 06:31:08 PM AEST 2026  env=da3
+============================================
+[WARN ] Dependency `gsplat` is required for rendering 3DGS. Install via: pip install git+https://github.com/nerfstudio-project/gsplat.git@0b4dddf04cb687367602c01196913cde6a743d70
+Found 5 scenes
+  [1/5] indoor__slow_solve-long_focal-2025-05-16-09-19-19___209591a0: shape=(720, 1280)
+Saved 5 predictions to /home/ywan0794/EvalMDE/output/infinigen5/da3_mono
+[INF-OK] da3_mono
+
+============================================
+[fe2e inference] Thu May 14 06:31:30 PM AEST 2026  env=fe2e
+============================================
+[INFO] prompt_type=empty, 跳过Qwen模型加载
+create LoRA network from weights
+train all blocks only
+create LoRA for DIT all blocks: 304 modules.
+enable LoRA for U-Net
+weights are merged
+Found 5 scenes
+  [1/5] indoor__slow_solve-long_focal-2025-05-16-09-19-19___209591a0: shape=(720, 1280)
+Saved 5 predictions to /home/ywan0794/EvalMDE/output/infinigen5/fe2e
+[INF-OK] fe2e
+
+============================================
+Stage 2: metric aggregation (evalmde env)
+============================================
+--- metric: depth_pro ---
+Found 5 scenes for depth_pro
+/home/ywan0794/miniconda3/envs/evalmde/lib/python3.10/site-packages/torch/functional.py:554: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at /pytorch/aten/src/ATen/native/TensorShape.cpp:4314.)
+  return _VF.meshgrid(tensors, **kwargs)  # type: ignore[attr-defined]
+  indoor__slow_solve-long_focal-2025-05-16-09-19-19___209591a0: sawa_h raw=1.295 aln=1.334  |  relnorm raw=0.206 aln=0.240  |  boundF1_err raw=0.874 aln=0.735
+  indoor__slow_solve-long_focal-2025-09-06-19-53-32___142efe82: sawa_h raw=1.010 aln=1.015  |  relnorm raw=0.272 aln=0.264  |  boundF1_err raw=0.693 aln=0.736
+  indoor__slow_solve-long_focal-2025-09-06-19-53-32___27c2f1fd: sawa_h raw=0.706 aln=0.698  |  relnorm raw=0.208 aln=0.199  |  boundF1_err raw=0.625 aln=0.666
+  nature__arctic-long_focal-2025-09-06-14-10-00___4bc42ce8: sawa_h raw=0.695 aln=0.831  |  relnorm raw=0.189 aln=0.203  |  boundF1_err raw=0.833 aln=0.803
+  nature__desert-long_focal-2025-09-07-11-43-28___cbe875d: sawa_h raw=0.443 aln=0.544  |  relnorm raw=0.075 aln=0.083  |  boundF1_err raw=0.892 aln=0.779
+
+Mean RAW    : {'wkdr_no_align': 0.06662386655807495, 'delta0125_disparity_affine_err': 0.3108652591705322, 'delta0125_depth_affine_err': 0.5642925873398781, 'boundary_f1_err': 0.7832580580591701, 'rel_normal': 0.1899009395071665, 'sawa_h': 0.8298352197168052}
+Mean ALIGNED: {'wkdr_no_align': 0.06664336919784546, 'delta0125_disparity_affine_err': 0.5713996738195419, 'delta0125_depth_affine_err': 0.5642925873398781, 'boundary_f1_err': 0.7437403012403084, 'rel_normal': 0.1979266966601841, 'sawa_h': 0.8844690165018712}
+Saved → /home/ywan0794/EvalMDE/output/infinigen5/depth_pro_metrics.json
+[METRIC-OK] depth_pro
+--- metric: marigold ---
+Found 5 scenes for marigold
+/home/ywan0794/miniconda3/envs/evalmde/lib/python3.10/site-packages/torch/functional.py:554: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at /pytorch/aten/src/ATen/native/TensorShape.cpp:4314.)
+  return _VF.meshgrid(tensors, **kwargs)  # type: ignore[attr-defined]
+  indoor__slow_solve-long_focal-2025-05-16-09-19-19___209591a0: sawa_h raw=2.214 aln=1.792  |  relnorm raw=0.378 aln=0.233  |  boundF1_err raw=0.979 aln=0.973
+  indoor__slow_solve-long_focal-2025-09-06-19-53-32___142efe82: sawa_h raw=1.646 aln=1.294  |  relnorm raw=0.532 aln=0.393  |  boundF1_err raw=0.903 aln=0.858
+  indoor__slow_solve-long_focal-2025-09-06-19-53-32___27c2f1fd: sawa_h raw=1.332 aln=0.943  |  relnorm raw=0.401 aln=0.265  |  boundF1_err raw=0.845 aln=0.920
+  nature__arctic-long_focal-2025-09-06-14-10-00___4bc42ce8: sawa_h raw=1.548 aln=1.536  |  relnorm raw=0.417 aln=0.415  |  boundF1_err raw=0.984 aln=0.963
+  nature__desert-long_focal-2025-09-07-11-43-28___cbe875d: sawa_h raw=1.120 aln=1.097  |  relnorm raw=0.248 aln=0.254  |  boundF1_err raw=0.933 aln=0.927
+
+Mean RAW    : {'wkdr_no_align': 0.12133046388626098, 'delta0125_disparity_affine_err': 0.9506231024861336, 'delta0125_depth_affine_err': 0.5405407793819904, 'boundary_f1_err': 0.9286170091147612, 'rel_normal': 0.39523183786670485, 'sawa_h': 1.5718469267105362}
+Mean ALIGNED: {'wkdr_no_align': 0.12138602733612061, 'delta0125_disparity_affine_err': 0.5170192375779152, 'delta0125_depth_affine_err': 0.5405403502285481, 'boundary_f1_err': 0.9283054545742395, 'rel_normal': 0.31193256845236983, 'sawa_h': 1.332338139755596}
+Saved → /home/ywan0794/EvalMDE/output/infinigen5/marigold_metrics.json
+[METRIC-OK] marigold
+--- metric: lotus ---
+Found 5 scenes for lotus
+/home/ywan0794/miniconda3/envs/evalmde/lib/python3.10/site-packages/torch/functional.py:554: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at /pytorch/aten/src/ATen/native/TensorShape.cpp:4314.)
+  return _VF.meshgrid(tensors, **kwargs)  # type: ignore[attr-defined]
+  indoor__slow_solve-long_focal-2025-05-16-09-19-19___209591a0: sawa_h raw=1.433 aln=1.027  |  relnorm raw=0.348 aln=0.209  |  boundF1_err raw=0.955 aln=0.905
+  indoor__slow_solve-long_focal-2025-09-06-19-53-32___142efe82: sawa_h raw=1.705 aln=1.539  |  relnorm raw=0.478 aln=0.384  |  boundF1_err raw=0.922 aln=0.859
+  indoor__slow_solve-long_focal-2025-09-06-19-53-32___27c2f1fd: sawa_h raw=1.029 aln=0.674  |  relnorm raw=0.296 aln=0.196  |  boundF1_err raw=0.833 aln=0.715
+  nature__arctic-long_focal-2025-09-06-14-10-00___4bc42ce8: sawa_h raw=1.131 aln=1.119  |  relnorm raw=0.307 aln=0.304  |  boundF1_err raw=0.969 aln=0.945
+  nature__desert-long_focal-2025-09-07-11-43-28___cbe875d: sawa_h raw=0.872 aln=0.860  |  relnorm raw=0.160 aln=0.159  |  boundF1_err raw=0.969 aln=0.936
+
+Mean RAW    : {'wkdr_no_align': 0.06971110105514526, 'delta0125_disparity_affine_err': 0.9483198569156229, 'delta0125_depth_affine_err': 0.6215518534183502, 'boundary_f1_err': 0.9296553861535948, 'rel_normal': 0.31790587652021685, 'sawa_h': 1.2340270893102154}
+Mean ALIGNED: {'wkdr_no_align': 0.06982688903808594, 'delta0125_disparity_affine_err': 0.6784043271094561, 'delta0125_depth_affine_err': 0.6215518534183502, 'boundary_f1_err': 0.8718338372591947, 'rel_normal': 0.250615008987667, 'sawa_h': 1.0437563272908121}
+Saved → /home/ywan0794/EvalMDE/output/infinigen5/lotus_metrics.json
+[METRIC-OK] lotus
+--- metric: depthmaster ---
+Found 5 scenes for depthmaster
+/home/ywan0794/miniconda3/envs/evalmde/lib/python3.10/site-packages/torch/functional.py:554: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at /pytorch/aten/src/ATen/native/TensorShape.cpp:4314.)
+  return _VF.meshgrid(tensors, **kwargs)  # type: ignore[attr-defined]
+  indoor__slow_solve-long_focal-2025-05-16-09-19-19___209591a0: sawa_h raw=4.224 aln=1.305  |  relnorm raw=0.530 aln=0.136  |  boundF1_err raw=0.997 aln=0.850
+  indoor__slow_solve-long_focal-2025-09-06-19-53-32___142efe82: sawa_h raw=4.541 aln=1.198  |  relnorm raw=0.401 aln=0.341  |  boundF1_err raw=0.999 aln=0.743
+  indoor__slow_solve-long_focal-2025-09-06-19-53-32___27c2f1fd: sawa_h raw=4.862 aln=1.305  |  relnorm raw=0.666 aln=0.306  |  boundF1_err raw=1.000 aln=0.885
+  nature__arctic-long_focal-2025-09-06-14-10-00___4bc42ce8: sawa_h raw=4.517 aln=1.027  |  relnorm raw=0.302 aln=0.273  |  boundF1_err raw=1.000 aln=0.996
+  nature__desert-long_focal-2025-09-07-11-43-28___cbe875d: sawa_h raw=4.075 aln=0.810  |  relnorm raw=0.097 aln=0.148  |  boundF1_err raw=0.999 aln=0.975
+
+Mean RAW    : {'wkdr_no_align': 0.9019776806235313, 'delta0125_disparity_affine_err': 0.9493729234673083, 'delta0125_depth_affine_err': 0.6568526294082403, 'boundary_f1_err': 0.9990700138797344, 'rel_normal': 0.39917262599449505, 'sawa_h': 4.443883083999355}
+Mean ALIGNED: {'wkdr_no_align': 0.0985716462135315, 'delta0125_disparity_affine_err': 0.651621462404728, 'delta0125_depth_affine_err': 0.6576177909970283, 'boundary_f1_err': 0.8899633566556169, 'rel_normal': 0.24088768568456684, 'sawa_h': 1.1289693313813944}
+Saved → /home/ywan0794/EvalMDE/output/infinigen5/depthmaster_metrics.json
+[METRIC-OK] depthmaster
+--- metric: ppd ---
+Found 5 scenes for ppd
+/home/ywan0794/miniconda3/envs/evalmde/lib/python3.10/site-packages/torch/functional.py:554: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at /pytorch/aten/src/ATen/native/TensorShape.cpp:4314.)
+  return _VF.meshgrid(tensors, **kwargs)  # type: ignore[attr-defined]
+  indoor__slow_solve-long_focal-2025-05-16-09-19-19___209591a0: sawa_h raw=3.384 aln=2.552  |  relnorm raw=1.066 aln=0.680  |  boundF1_err raw=0.959 aln=0.878
+  indoor__slow_solve-long_focal-2025-09-06-19-53-32___142efe82: sawa_h raw=2.301 aln=1.944  |  relnorm raw=0.897 aln=0.760  |  boundF1_err raw=0.866 aln=0.760
+  indoor__slow_solve-long_focal-2025-09-06-19-53-32___27c2f1fd: sawa_h raw=2.149 aln=1.496  |  relnorm raw=0.867 aln=0.624  |  boundF1_err raw=0.865 aln=0.626
+  nature__arctic-long_focal-2025-09-06-14-10-00___4bc42ce8: sawa_h raw=1.505 aln=1.543  |  relnorm raw=0.532 aln=0.553  |  boundF1_err raw=0.996 aln=0.996
+  nature__desert-long_focal-2025-09-07-11-43-28___cbe875d: sawa_h raw=1.538 aln=1.560  |  relnorm raw=0.485 aln=0.497  |  boundF1_err raw=0.968 aln=0.974
+
+Mean RAW    : {'wkdr_no_align': 0.08756059408187866, 'delta0125_disparity_affine_err': 0.950529617164284, 'delta0125_depth_affine_err': 0.6146524578332901, 'boundary_f1_err': 0.9310581919470687, 'rel_normal': 0.7692545256509766, 'sawa_h': 2.1754034422190696}
+Mean ALIGNED: {'wkdr_no_align': 0.08780105113983154, 'delta0125_disparity_affine_err': 0.639212078601122, 'delta0125_depth_affine_err': 0.6143273778259755, 'boundary_f1_err': 0.8466643323124072, 'rel_normal': 0.6228359304030748, 'sawa_h': 1.8193098560312937}
+Saved → /home/ywan0794/EvalMDE/output/infinigen5/ppd_metrics.json
+[METRIC-OK] ppd
+--- metric: da3_mono ---
+Found 5 scenes for da3_mono
+/home/ywan0794/miniconda3/envs/evalmde/lib/python3.10/site-packages/torch/functional.py:554: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at /pytorch/aten/src/ATen/native/TensorShape.cpp:4314.)
+  return _VF.meshgrid(tensors, **kwargs)  # type: ignore[attr-defined]
+  indoor__slow_solve-long_focal-2025-05-16-09-19-19___209591a0: sawa_h raw=0.632 aln=0.587  |  relnorm raw=0.131 aln=0.115  |  boundF1_err raw=0.962 aln=0.982
+  indoor__slow_solve-long_focal-2025-09-06-19-53-32___142efe82: sawa_h raw=0.884 aln=0.793  |  relnorm raw=0.247 aln=0.223  |  boundF1_err raw=0.836 aln=0.873
+  indoor__slow_solve-long_focal-2025-09-06-19-53-32___27c2f1fd: sawa_h raw=0.691 aln=0.643  |  relnorm raw=0.192 aln=0.165  |  boundF1_err raw=0.870 aln=0.934
+  nature__arctic-long_focal-2025-09-06-14-10-00___4bc42ce8: sawa_h raw=0.877 aln=0.898  |  relnorm raw=0.194 aln=0.200  |  boundF1_err raw=0.958 aln=0.950
+  nature__desert-long_focal-2025-09-07-11-43-28___cbe875d: sawa_h raw=0.605 aln=0.607  |  relnorm raw=0.099 aln=0.099  |  boundF1_err raw=0.936 aln=0.935
+
+Mean RAW    : {'wkdr_no_align': 0.033317041397094724, 'delta0125_disparity_affine_err': 0.5275852054357528, 'delta0125_depth_affine_err': 0.40731881856918334, 'boundary_f1_err': 0.9124339414097011, 'rel_normal': 0.17258852279331505, 'sawa_h': 0.7379542487644946}
+Mean ALIGNED: {'wkdr_no_align': 0.03334666490554809, 'delta0125_disparity_affine_err': 0.4545227389782667, 'delta0125_depth_affine_err': 0.40731837004423144, 'boundary_f1_err': 0.9345920700164015, 'rel_normal': 0.16045701111115004, 'sawa_h': 0.7058076191806922}
+Saved → /home/ywan0794/EvalMDE/output/infinigen5/da3_mono_metrics.json
+[METRIC-OK] da3_mono
+--- metric: fe2e ---
+Found 5 scenes for fe2e
+/home/ywan0794/miniconda3/envs/evalmde/lib/python3.10/site-packages/torch/functional.py:554: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at /pytorch/aten/src/ATen/native/TensorShape.cpp:4314.)
+  return _VF.meshgrid(tensors, **kwargs)  # type: ignore[attr-defined]
+  indoor__slow_solve-long_focal-2025-05-16-09-19-19___209591a0: sawa_h raw=1.291 aln=0.960  |  relnorm raw=0.239 aln=0.139  |  boundF1_err raw=0.910 aln=0.883
+  indoor__slow_solve-long_focal-2025-09-06-19-53-32___142efe82: sawa_h raw=1.080 aln=0.869  |  relnorm raw=0.341 aln=0.286  |  boundF1_err raw=0.826 aln=0.793
+  indoor__slow_solve-long_focal-2025-09-06-19-53-32___27c2f1fd: sawa_h raw=0.994 aln=0.625  |  relnorm raw=0.285 aln=0.175  |  boundF1_err raw=0.852 aln=0.804
+  nature__arctic-long_focal-2025-09-06-14-10-00___4bc42ce8: sawa_h raw=0.902 aln=1.025  |  relnorm raw=0.224 aln=0.288  |  boundF1_err raw=0.986 aln=0.967
+  nature__desert-long_focal-2025-09-07-11-43-28___cbe875d: sawa_h raw=0.731 aln=0.837  |  relnorm raw=0.119 aln=0.171  |  boundF1_err raw=0.950 aln=0.970
+
+Mean RAW    : {'wkdr_no_align': 0.047560691833496094, 'delta0125_disparity_affine_err': 0.952283101901412, 'delta0125_depth_affine_err': 0.5054923050105572, 'boundary_f1_err': 0.9046317621819318, 'rel_normal': 0.24158862658068156, 'sawa_h': 0.9996706945875289}
+Mean ALIGNED: {'wkdr_no_align': 0.04835277795791626, 'delta0125_disparity_affine_err': 0.5249058477580547, 'delta0125_depth_affine_err': 0.5062363661825657, 'boundary_f1_err': 0.8833090678944918, 'rel_normal': 0.21159110840747117, 'sawa_h': 0.8631816196940623}
+Saved → /home/ywan0794/EvalMDE/output/infinigen5/fe2e_metrics.json
+[METRIC-OK] fe2e
+
+============================================
+infinigen5 finished at Thu May 14 06:32:37 PM AEST 2026
+=== Summary ===
+[INF-OK] depth_pro
+[INF-OK] marigold
+[INF-OK] lotus
+[INF-OK] depthmaster
+[INF-OK] ppd
+[INF-OK] da3_mono
+[INF-OK] fe2e
+[METRIC-OK] depth_pro
+[METRIC-OK] marigold
+[METRIC-OK] lotus
+[METRIC-OK] depthmaster
+[METRIC-OK] ppd
+[METRIC-OK] da3_mono
+[METRIC-OK] fe2e
+=== Per-model means ===
+Traceback (most recent call last):
+  File "<string>", line 1, in <module>
+KeyError: 'mean'
+depth_pro: 
+Traceback (most recent call last):
+  File "<string>", line 1, in <module>
+KeyError: 'mean'
+marigold: 
+Traceback (most recent call last):
+  File "<string>", line 1, in <module>
+KeyError: 'mean'
+lotus: 
+Traceback (most recent call last):
+  File "<string>", line 1, in <module>
+KeyError: 'mean'
+depthmaster: 
+Traceback (most recent call last):
+  File "<string>", line 1, in <module>
+KeyError: 'mean'
+ppd: 
+Traceback (most recent call last):
+  File "<string>", line 1, in <module>
+KeyError: 'mean'
+da3_mono: 
+Traceback (most recent call last):
+  File "<string>", line 1, in <module>
+KeyError: 'mean'
+fe2e: 

setup.py

@@ -0,0 +1,39 @@
+import os.path as osp
+from setuptools import setup, find_packages
+
+ROOT = osp.dirname(osp.abspath(__file__))
+
+setup(
+    name='evalmde',
+    packages=find_packages(),
+    install_requires=[
+        "numpy==2.0.0",
+        "opencv-python==4.12.0.88",
+        "open3d==0.19.0",
+        "pyglet==1.5.28",
+        "imageio==2.33.1",
+        "hydra-core==1.3.0",
+        "pyrender==0.1.45",
+        "evo==1.26.0",
+        "loguru==0.7.2",
+        "shortuuid==1.0.13",
+        "DateTime==5.5",
+        "plyfile==1.1",
+        "HTML4Vision==0.4.3",
+        "timm==1.0.9",
+        "imgaug==0.4.0",
+        "iopath==0.1.10",
+        "imagecorruptions==1.1.2",
+        "gitpython==3.1.44",
+        "pomegranate==1.1.1",
+        "matplotlib==3.9.0",
+        "wandb==0.22.2",
+        "cvxpy==1.6.5",
+        "mathutils==3.3.0",
+        "OpenEXR==3.3.3",
+        "Imath==0.0.2",
+        "pywavelets==1.8.0",
+        "h5py==3.14.0",
+    ],
+)
+

smoke_all_12114.log

@@ -0,0 +1,218 @@
+============================================
+smoke-all started at Thu May 14 10:45:07 AM AEST 2026
+Data: /home/ywan0794/EvalMDE/data/smoke   Output: /home/ywan0794/EvalMDE/output/smoke_all
+============================================
+Thu May 14 10:45:07 2026       
++-----------------------------------------------------------------------------------------+
+| NVIDIA-SMI 550.163.01             Driver Version: 550.163.01     CUDA Version: 12.4     |
+|-----------------------------------------+------------------------+----------------------+
+| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
+| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
+|                                         |                        |               MIG M. |
+|=========================================+========================+======================|
+|   0  NVIDIA H100 NVL                Off |   00000000:61:00.0 Off |                    0 |
+| N/A   38C    P0             61W /  400W |      14MiB /  95830MiB |      0%      Default |
+|                                         |                        |             Disabled |
++-----------------------------------------+------------------------+----------------------+
+                                                                                         
++-----------------------------------------------------------------------------------------+
+| Processes:                                                                              |
+|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
+|        ID   ID                                                               Usage      |
+|=========================================================================================|
+|    0   N/A  N/A      4274      G   /usr/lib/xorg/Xorg                              4MiB |
++-----------------------------------------------------------------------------------------+
+
+============================================
+[depth_pro inference] Thu May 14 10:45:07 AM AEST 2026  env=depth-pro
+============================================
+Found 2 scenes
+  [1/2] sample_data: shape=(720, 1280)
+Saved 2 predictions to /home/ywan0794/EvalMDE/output/smoke_all/depth_pro
+[INF-OK] depth_pro
+
+============================================
+[marigold inference] Thu May 14 10:45:25 AM AEST 2026  env=marigold
+============================================
+The config attributes {'prediction_type': 'depth'} were passed to MarigoldDepthPipeline, but are not expected and will be ignored. Please verify your model_index.json configuration file.
+Keyword arguments {'prediction_type': 'depth'} are not expected by MarigoldDepthPipeline and will be ignored.
+
+
+
+Found 2 scenes
+  [1/2] sample_data: shape=(720, 1280)
+Saved 2 predictions to /home/ywan0794/EvalMDE/output/smoke_all/marigold
+[INF-OK] marigold
+
+============================================
+[lotus inference] Thu May 14 10:45:49 AM AEST 2026  env=lotus
+============================================
+
+Found 2 scenes
+  [1/2] sample_data: shape=(720, 1280)
+Saved 2 predictions to /home/ywan0794/EvalMDE/output/smoke_all/lotus
+[INF-OK] lotus
+
+============================================
+[depthmaster inference] Thu May 14 10:46:09 AM AEST 2026  env=depthmaster
+============================================
+The config attributes {'default_denoising_steps': 10, 'scheduler': ['diffusers', 'DDIMScheduler']} were passed to DepthMasterPipeline, but are not expected and will be ignored. Please verify your model_index.json configuration file.
+Keyword arguments {'default_denoising_steps': 10, 'scheduler': ['diffusers', 'DDIMScheduler']} are not expected by DepthMasterPipeline and will be ignored.
+
+
+
+Defaulting to unsafe serialization. Pass `allow_pickle=False` to raise an error instead.
+Some weights of the model checkpoint at /home/ywan0794/EvalMDE/DepthMaster/ckpt/eval/unet were not used when initializing UNet2DConditionModel: 
+ ['fftblock.conv_s1.weight, fftblock.conv_f4.weight, fftblock.conv_f3.weight, fftblock.fuse.weight, fftblock.conv_s2.bias, fftblock.conv_s2.weight, fftblock.conv_s1.bias, fftblock.conv_f2.bias, fftblock.fuse.bias, fftblock.conv_f1.weight, fftblock.norm.bias, fftblock.conv_f1.bias, fftblock.norm.weight, fftblock.conv_f3.bias, fftblock.conv_f2.weight, fftblock.conv_f4.bias']
+
+Expected types for unet: (<class 'depthmaster.modules.unet_2d_condition_s2.UNet2DConditionModel'>,), got <class 'diffusers.models.unets.unet_2d_condition.UNet2DConditionModel'>.
+An error occurred while trying to fetch /home/ywan0794/EvalMDE/DepthMaster/ckpt/eval/unet: Error no file named diffusion_pytorch_model.safetensors found in directory /home/ywan0794/EvalMDE/DepthMaster/ckpt/eval/unet.
+Defaulting to unsafe serialization. Pass `allow_pickle=False` to raise an error instead.
+Found 2 scenes
+  [1/2] sample_data: shape=(720, 1280)
+Saved 2 predictions to /home/ywan0794/EvalMDE/output/smoke_all/depthmaster
+[INF-OK] depthmaster
+
+============================================
+[ppd inference] Thu May 14 10:46:51 AM AEST 2026  env=ppd
+============================================
+xFormers not available
+xFormers not available
+Found 2 scenes
+  [1/2] sample_data: shape=(720, 1280)
+Saved 2 predictions to /home/ywan0794/EvalMDE/output/smoke_all/ppd
+[INF-OK] ppd
+
+============================================
+[da3_mono inference] Thu May 14 10:47:30 AM AEST 2026  env=da3
+============================================
+[WARN ] Dependency `gsplat` is required for rendering 3DGS. Install via: pip install git+https://github.com/nerfstudio-project/gsplat.git@0b4dddf04cb687367602c01196913cde6a743d70
+Found 2 scenes
+Traceback (most recent call last):
+  File "/home/ywan0794/EvalMDE/scripts/run_inference.py", line 94, in <module>
+    main()
+  File "/home/ywan0794/miniconda3/envs/da3/lib/python3.10/site-packages/click/core.py", line 1485, in __call__
+    return self.main(*args, **kwargs)
+  File "/home/ywan0794/miniconda3/envs/da3/lib/python3.10/site-packages/click/core.py", line 1406, in main
+    rv = self.invoke(ctx)
+  File "/home/ywan0794/miniconda3/envs/da3/lib/python3.10/site-packages/click/core.py", line 1269, in invoke
+    return ctx.invoke(self.callback, **ctx.params)
+  File "/home/ywan0794/miniconda3/envs/da3/lib/python3.10/site-packages/click/core.py", line 824, in invoke
+    return callback(*args, **kwargs)
+  File "/home/ywan0794/miniconda3/envs/da3/lib/python3.10/site-packages/click/decorators.py", line 34, in new_func
+    return f(get_current_context(), *args, **kwargs)
+  File "/home/ywan0794/EvalMDE/scripts/run_inference.py", line 59, in main
+    pred = baseline.infer_for_evaluation(img, K_norm)
+  File "/home/ywan0794/MoGe/moge/test/baseline.py", line 43, in infer_for_evaluation
+    return self.infer(image, intrinsics)
+  File "/home/ywan0794/miniconda3/envs/da3/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
+    return func(*args, **kwargs)
+  File "/home/ywan0794/EvalMDE/baselines/da3_mono.py", line 69, in infer
+    assert intrinsics is None, "DA3-Mono does not consume intrinsics."
+AssertionError: DA3-Mono does not consume intrinsics.
+[INF-FAIL rc=1] da3_mono
+
+============================================
+[fe2e inference] Thu May 14 10:48:02 AM AEST 2026  env=fe2e
+============================================
+[INFO] prompt_type=empty, 跳过Qwen模型加载
+create LoRA network from weights
+train all blocks only
+create LoRA for DIT all blocks: 304 modules.
+enable LoRA for U-Net
+weights are merged
+Found 2 scenes
+  [1/2] sample_data: shape=(720, 1280)
+Saved 2 predictions to /home/ywan0794/EvalMDE/output/smoke_all/fe2e
+[INF-OK] fe2e
+
+============================================
+Stage 2: metric aggregation (evalmde env)
+============================================
+--- metric: depth_pro ---
+Found 2 scenes for depth_pro
+/home/ywan0794/miniconda3/envs/evalmde/lib/python3.10/site-packages/torch/functional.py:554: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at /pytorch/aten/src/ATen/native/TensorShape.cpp:4314.)
+  return _VF.meshgrid(tensors, **kwargs)  # type: ignore[attr-defined]
+  sample_data: sawa_h=0.5913, rel_normal=0.1919
+  sample_data_2: sawa_h=1.2677, rel_normal=0.3900
+
+Mean: {'sawa_h': 0.9295024154567082, 'rel_normal': 0.2909630531817561}
+Saved → /home/ywan0794/EvalMDE/output/smoke_all/depth_pro_metrics.json
+[METRIC-OK] depth_pro
+--- metric: marigold ---
+Found 2 scenes for marigold
+/home/ywan0794/miniconda3/envs/evalmde/lib/python3.10/site-packages/torch/functional.py:554: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at /pytorch/aten/src/ATen/native/TensorShape.cpp:4314.)
+  return _VF.meshgrid(tensors, **kwargs)  # type: ignore[attr-defined]
+  sample_data: sawa_h=1.1787, rel_normal=0.3820
+  sample_data_2: sawa_h=2.1863, rel_normal=0.7493
+
+Mean: {'sawa_h': 1.682470514493703, 'rel_normal': 0.5656452301519006}
+Saved → /home/ywan0794/EvalMDE/output/smoke_all/marigold_metrics.json
+[METRIC-OK] marigold
+--- metric: lotus ---
+Found 2 scenes for lotus
+/home/ywan0794/miniconda3/envs/evalmde/lib/python3.10/site-packages/torch/functional.py:554: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at /pytorch/aten/src/ATen/native/TensorShape.cpp:4314.)
+  return _VF.meshgrid(tensors, **kwargs)  # type: ignore[attr-defined]
+  sample_data: sawa_h=1.0975, rel_normal=0.3560
+  sample_data_2: sawa_h=1.9437, rel_normal=0.5927
+
+Mean: {'sawa_h': 1.520615413262182, 'rel_normal': 0.47437434867840383}
+Saved → /home/ywan0794/EvalMDE/output/smoke_all/lotus_metrics.json
+[METRIC-OK] lotus
+--- metric: depthmaster ---
+Found 2 scenes for depthmaster
+/home/ywan0794/miniconda3/envs/evalmde/lib/python3.10/site-packages/torch/functional.py:554: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at /pytorch/aten/src/ATen/native/TensorShape.cpp:4314.)
+  return _VF.meshgrid(tensors, **kwargs)  # type: ignore[attr-defined]
+  sample_data: sawa_h=4.4748, rel_normal=0.3334
+  sample_data_2: sawa_h=4.7746, rel_normal=0.5856
+
+Mean: {'sawa_h': 4.624733318991572, 'rel_normal': 0.4595408906976621}
+Saved → /home/ywan0794/EvalMDE/output/smoke_all/depthmaster_metrics.json
+[METRIC-OK] depthmaster
+--- metric: ppd ---
+Found 2 scenes for ppd
+/home/ywan0794/miniconda3/envs/evalmde/lib/python3.10/site-packages/torch/functional.py:554: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at /pytorch/aten/src/ATen/native/TensorShape.cpp:4314.)
+  return _VF.meshgrid(tensors, **kwargs)  # type: ignore[attr-defined]
+  sample_data: sawa_h=2.1980, rel_normal=0.8372
+  sample_data_2: sawa_h=2.5355, rel_normal=0.9053
+
+Mean: {'sawa_h': 2.3667450420848906, 'rel_normal': 0.8712330618410348}
+Saved → /home/ywan0794/EvalMDE/output/smoke_all/ppd_metrics.json
+[METRIC-OK] ppd
+--- metric: da3_mono ---
+[METRIC-SKIP no inference output] da3_mono
+--- metric: fe2e ---
+Found 2 scenes for fe2e
+/home/ywan0794/miniconda3/envs/evalmde/lib/python3.10/site-packages/torch/functional.py:554: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at /pytorch/aten/src/ATen/native/TensorShape.cpp:4314.)
+  return _VF.meshgrid(tensors, **kwargs)  # type: ignore[attr-defined]
+  sample_data: sawa_h=1.1134, rel_normal=0.3317
+  sample_data_2: sawa_h=1.9205, rel_normal=0.6153
+
+Mean: {'sawa_h': 1.5169872681088794, 'rel_normal': 0.47354231407887987}
+Saved → /home/ywan0794/EvalMDE/output/smoke_all/fe2e_metrics.json
+[METRIC-OK] fe2e
+
+============================================
+smoke-all finished at Thu May 14 10:48:46 AM AEST 2026
+=== Summary ===
+[INF-OK] depth_pro
+[INF-OK] marigold
+[INF-OK] lotus
+[INF-OK] depthmaster
+[INF-OK] ppd
+[INF-FAIL rc=1] da3_mono
+[INF-OK] fe2e
+[METRIC-OK] depth_pro
+[METRIC-OK] marigold
+[METRIC-OK] lotus
+[METRIC-OK] depthmaster
+[METRIC-OK] ppd
+[METRIC-SKIP no inference output] da3_mono
+[METRIC-OK] fe2e
+=== Per-model means ===
+depth_pro: {'sawa_h': 0.9295024154567082, 'rel_normal': 0.2909630531817561}
+marigold: {'sawa_h': 1.682470514493703, 'rel_normal': 0.5656452301519006}
+lotus: {'sawa_h': 1.520615413262182, 'rel_normal': 0.47437434867840383}
+depthmaster: {'sawa_h': 4.624733318991572, 'rel_normal': 0.4595408906976621}
+ppd: {'sawa_h': 2.3667450420848906, 'rel_normal': 0.8712330618410348}
+fe2e: {'sawa_h': 1.5169872681088794, 'rel_normal': 0.47354231407887987}

smoke_all_12115.log

@@ -0,0 +1,207 @@
+============================================
+smoke-all started at Thu May 14 10:56:25 AM AEST 2026
+Data: /home/ywan0794/EvalMDE/data/smoke   Output: /home/ywan0794/EvalMDE/output/smoke_all
+============================================
+Thu May 14 10:56:25 2026       
++-----------------------------------------------------------------------------------------+
+| NVIDIA-SMI 550.163.01             Driver Version: 550.163.01     CUDA Version: 12.4     |
+|-----------------------------------------+------------------------+----------------------+
+| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
+| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
+|                                         |                        |               MIG M. |
+|=========================================+========================+======================|
+|   0  NVIDIA H100 NVL                Off |   00000000:61:00.0 Off |                    0 |
+| N/A   38C    P0             61W /  400W |      14MiB /  95830MiB |      0%      Default |
+|                                         |                        |             Disabled |
++-----------------------------------------+------------------------+----------------------+
+                                                                                         
++-----------------------------------------------------------------------------------------+
+| Processes:                                                                              |
+|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
+|        ID   ID                                                               Usage      |
+|=========================================================================================|
+|    0   N/A  N/A      4274      G   /usr/lib/xorg/Xorg                              4MiB |
++-----------------------------------------------------------------------------------------+
+
+============================================
+[depth_pro inference] Thu May 14 10:56:25 AM AEST 2026  env=depth-pro
+============================================
+Found 2 scenes
+  [1/2] sample_data: shape=(720, 1280)
+Saved 2 predictions to /home/ywan0794/EvalMDE/output/smoke_all/depth_pro
+[INF-OK] depth_pro
+
+============================================
+[marigold inference] Thu May 14 10:56:44 AM AEST 2026  env=marigold
+============================================
+The config attributes {'prediction_type': 'depth'} were passed to MarigoldDepthPipeline, but are not expected and will be ignored. Please verify your model_index.json configuration file.
+Keyword arguments {'prediction_type': 'depth'} are not expected by MarigoldDepthPipeline and will be ignored.
+
+
+
+Found 2 scenes
+  [1/2] sample_data: shape=(720, 1280)
+Saved 2 predictions to /home/ywan0794/EvalMDE/output/smoke_all/marigold
+[INF-OK] marigold
+
+============================================
+[lotus inference] Thu May 14 10:57:01 AM AEST 2026  env=lotus
+============================================
+
+Found 2 scenes
+  [1/2] sample_data: shape=(720, 1280)
+Saved 2 predictions to /home/ywan0794/EvalMDE/output/smoke_all/lotus
+[INF-OK] lotus
+
+============================================
+[depthmaster inference] Thu May 14 10:57:14 AM AEST 2026  env=depthmaster
+============================================
+The config attributes {'default_denoising_steps': 10, 'scheduler': ['diffusers', 'DDIMScheduler']} were passed to DepthMasterPipeline, but are not expected and will be ignored. Please verify your model_index.json configuration file.
+Keyword arguments {'default_denoising_steps': 10, 'scheduler': ['diffusers', 'DDIMScheduler']} are not expected by DepthMasterPipeline and will be ignored.
+
+Defaulting to unsafe serialization. Pass `allow_pickle=False` to raise an error instead.
+Some weights of the model checkpoint at /home/ywan0794/EvalMDE/DepthMaster/ckpt/eval/unet were not used when initializing UNet2DConditionModel: 
+ ['fftblock.norm.weight, fftblock.norm.bias, fftblock.conv_f4.bias, fftblock.fuse.bias, fftblock.conv_s1.bias, fftblock.conv_f4.weight, fftblock.conv_f2.weight, fftblock.conv_f3.bias, fftblock.conv_f1.weight, fftblock.fuse.weight, fftblock.conv_s1.weight, fftblock.conv_s2.weight, fftblock.conv_f2.bias, fftblock.conv_f3.weight, fftblock.conv_f1.bias, fftblock.conv_s2.bias']
+
+
+
+Expected types for unet: (<class 'depthmaster.modules.unet_2d_condition_s2.UNet2DConditionModel'>,), got <class 'diffusers.models.unets.unet_2d_condition.UNet2DConditionModel'>.
+An error occurred while trying to fetch /home/ywan0794/EvalMDE/DepthMaster/ckpt/eval/unet: Error no file named diffusion_pytorch_model.safetensors found in directory /home/ywan0794/EvalMDE/DepthMaster/ckpt/eval/unet.
+Defaulting to unsafe serialization. Pass `allow_pickle=False` to raise an error instead.
+Found 2 scenes
+  [1/2] sample_data: shape=(720, 1280)
+Saved 2 predictions to /home/ywan0794/EvalMDE/output/smoke_all/depthmaster
+[INF-OK] depthmaster
+
+============================================
+[ppd inference] Thu May 14 10:57:32 AM AEST 2026  env=ppd
+============================================
+xFormers not available
+xFormers not available
+Found 2 scenes
+  [1/2] sample_data: shape=(720, 1280)
+Saved 2 predictions to /home/ywan0794/EvalMDE/output/smoke_all/ppd
+[INF-OK] ppd
+
+============================================
+[da3_mono inference] Thu May 14 10:57:50 AM AEST 2026  env=da3
+============================================
+[WARN ] Dependency `gsplat` is required for rendering 3DGS. Install via: pip install git+https://github.com/nerfstudio-project/gsplat.git@0b4dddf04cb687367602c01196913cde6a743d70
+Found 2 scenes
+  [1/2] sample_data: shape=(720, 1280)
+Saved 2 predictions to /home/ywan0794/EvalMDE/output/smoke_all/da3_mono
+[INF-OK] da3_mono
+
+============================================
+[fe2e inference] Thu May 14 10:58:06 AM AEST 2026  env=fe2e
+============================================
+[INFO] prompt_type=empty, 跳过Qwen模型加载
+create LoRA network from weights
+train all blocks only
+create LoRA for DIT all blocks: 304 modules.
+enable LoRA for U-Net
+weights are merged
+Found 2 scenes
+  [1/2] sample_data: shape=(720, 1280)
+Saved 2 predictions to /home/ywan0794/EvalMDE/output/smoke_all/fe2e
+[INF-OK] fe2e
+
+============================================
+Stage 2: metric aggregation (evalmde env)
+============================================
+--- metric: depth_pro ---
+Found 2 scenes for depth_pro
+/home/ywan0794/miniconda3/envs/evalmde/lib/python3.10/site-packages/torch/functional.py:554: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at /pytorch/aten/src/ATen/native/TensorShape.cpp:4314.)
+  return _VF.meshgrid(tensors, **kwargs)  # type: ignore[attr-defined]
+  sample_data: sawa_h=0.5911, rel_normal=0.1919
+  sample_data_2: sawa_h=1.2681, rel_normal=0.3900
+
+Mean: {'sawa_h': 0.9295960737251597, 'rel_normal': 0.2909630531817561}
+Saved → /home/ywan0794/EvalMDE/output/smoke_all/depth_pro_metrics.json
+[METRIC-OK] depth_pro
+--- metric: marigold ---
+Found 2 scenes for marigold
+/home/ywan0794/miniconda3/envs/evalmde/lib/python3.10/site-packages/torch/functional.py:554: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at /pytorch/aten/src/ATen/native/TensorShape.cpp:4314.)
+  return _VF.meshgrid(tensors, **kwargs)  # type: ignore[attr-defined]
+  sample_data: sawa_h=1.1825, rel_normal=0.3738
+  sample_data_2: sawa_h=2.3192, rel_normal=0.7576
+
+Mean: {'sawa_h': 1.7508343150610857, 'rel_normal': 0.5656810754863019}
+Saved → /home/ywan0794/EvalMDE/output/smoke_all/marigold_metrics.json
+[METRIC-OK] marigold
+--- metric: lotus ---
+Found 2 scenes for lotus
+/home/ywan0794/miniconda3/envs/evalmde/lib/python3.10/site-packages/torch/functional.py:554: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at /pytorch/aten/src/ATen/native/TensorShape.cpp:4314.)
+  return _VF.meshgrid(tensors, **kwargs)  # type: ignore[attr-defined]
+  sample_data: sawa_h=1.0973, rel_normal=0.3560
+  sample_data_2: sawa_h=1.9433, rel_normal=0.5927
+
+Mean: {'sawa_h': 1.5202771508195192, 'rel_normal': 0.4743600947637731}
+Saved → /home/ywan0794/EvalMDE/output/smoke_all/lotus_metrics.json
+[METRIC-OK] lotus
+--- metric: depthmaster ---
+Found 2 scenes for depthmaster
+/home/ywan0794/miniconda3/envs/evalmde/lib/python3.10/site-packages/torch/functional.py:554: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at /pytorch/aten/src/ATen/native/TensorShape.cpp:4314.)
+  return _VF.meshgrid(tensors, **kwargs)  # type: ignore[attr-defined]
+  sample_data: sawa_h=4.4750, rel_normal=0.3334
+  sample_data_2: sawa_h=4.7744, rel_normal=0.5856
+
+Mean: {'sawa_h': 4.624715064603448, 'rel_normal': 0.4595408906976621}
+Saved → /home/ywan0794/EvalMDE/output/smoke_all/depthmaster_metrics.json
+[METRIC-OK] depthmaster
+--- metric: ppd ---
+Found 2 scenes for ppd
+/home/ywan0794/miniconda3/envs/evalmde/lib/python3.10/site-packages/torch/functional.py:554: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at /pytorch/aten/src/ATen/native/TensorShape.cpp:4314.)
+  return _VF.meshgrid(tensors, **kwargs)  # type: ignore[attr-defined]
+  sample_data: sawa_h=2.1984, rel_normal=0.8372
+  sample_data_2: sawa_h=2.5360, rel_normal=0.9053
+
+Mean: {'sawa_h': 2.3671746082894383, 'rel_normal': 0.8712330618410348}
+Saved → /home/ywan0794/EvalMDE/output/smoke_all/ppd_metrics.json
+[METRIC-OK] ppd
+--- metric: da3_mono ---
+Found 2 scenes for da3_mono
+/home/ywan0794/miniconda3/envs/evalmde/lib/python3.10/site-packages/torch/functional.py:554: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at /pytorch/aten/src/ATen/native/TensorShape.cpp:4314.)
+  return _VF.meshgrid(tensors, **kwargs)  # type: ignore[attr-defined]
+  sample_data: sawa_h=0.8455, rel_normal=0.2068
+  sample_data_2: sawa_h=1.4472, rel_normal=0.4535
+
+Mean: {'sawa_h': 1.1463719382024506, 'rel_normal': 0.3301750209283423}
+Saved → /home/ywan0794/EvalMDE/output/smoke_all/da3_mono_metrics.json
+[METRIC-OK] da3_mono
+--- metric: fe2e ---
+Found 2 scenes for fe2e
+/home/ywan0794/miniconda3/envs/evalmde/lib/python3.10/site-packages/torch/functional.py:554: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at /pytorch/aten/src/ATen/native/TensorShape.cpp:4314.)
+  return _VF.meshgrid(tensors, **kwargs)  # type: ignore[attr-defined]
+  sample_data: sawa_h=1.1133, rel_normal=0.3317
+  sample_data_2: sawa_h=1.9202, rel_normal=0.6153
+
+Mean: {'sawa_h': 1.5167008543796885, 'rel_normal': 0.47354231407887987}
+Saved → /home/ywan0794/EvalMDE/output/smoke_all/fe2e_metrics.json
+[METRIC-OK] fe2e
+
+============================================
+smoke-all finished at Thu May 14 10:58:51 AM AEST 2026
+=== Summary ===
+[INF-OK] depth_pro
+[INF-OK] marigold
+[INF-OK] lotus
+[INF-OK] depthmaster
+[INF-OK] ppd
+[INF-OK] da3_mono
+[INF-OK] fe2e
+[METRIC-OK] depth_pro
+[METRIC-OK] marigold
+[METRIC-OK] lotus
+[METRIC-OK] depthmaster
+[METRIC-OK] ppd
+[METRIC-OK] da3_mono
+[METRIC-OK] fe2e
+=== Per-model means ===
+depth_pro: {'sawa_h': 0.9295960737251597, 'rel_normal': 0.2909630531817561}
+marigold: {'sawa_h': 1.7508343150610857, 'rel_normal': 0.5656810754863019}
+lotus: {'sawa_h': 1.5202771508195192, 'rel_normal': 0.4743600947637731}
+depthmaster: {'sawa_h': 4.624715064603448, 'rel_normal': 0.4595408906976621}
+ppd: {'sawa_h': 2.3671746082894383, 'rel_normal': 0.8712330618410348}
+da3_mono: {'sawa_h': 1.1463719382024506, 'rel_normal': 0.3301750209283423}
+fe2e: {'sawa_h': 1.5167008543796885, 'rel_normal': 0.47354231407887987}

smoke_all_12351.log

@@ -0,0 +1,235 @@
+============================================
+smoke-all started at Thu May 14 11:35:01 AM AEST 2026
+Data: /home/ywan0794/EvalMDE/data/smoke   Output: /home/ywan0794/EvalMDE/output/smoke_all
+============================================
+Thu May 14 11:35:01 2026       
++-----------------------------------------------------------------------------------------+
+| NVIDIA-SMI 550.163.01             Driver Version: 550.163.01     CUDA Version: 12.4     |
+|-----------------------------------------+------------------------+----------------------+
+| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
+| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
+|                                         |                        |               MIG M. |
+|=========================================+========================+======================|
+|   0  NVIDIA H100 NVL                Off |   00000000:E1:00.0 Off |                    0 |
+| N/A   37C    P0             93W /  400W |      14MiB /  95830MiB |      2%      Default |
+|                                         |                        |             Disabled |
++-----------------------------------------+------------------------+----------------------+
+                                                                                         
++-----------------------------------------------------------------------------------------+
+| Processes:                                                                              |
+|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
+|        ID   ID                                                               Usage      |
+|=========================================================================================|
+|    0   N/A  N/A      4274      G   /usr/lib/xorg/Xorg                              4MiB |
++-----------------------------------------------------------------------------------------+
+
+============================================
+[depth_pro inference] Thu May 14 11:35:01 AM AEST 2026  env=depth-pro
+============================================
+Found 2 scenes
+  [1/2] sample_data: shape=(720, 1280)
+Saved 2 predictions to /home/ywan0794/EvalMDE/output/smoke_all/depth_pro
+[INF-OK] depth_pro
+
+============================================
+[marigold inference] Thu May 14 11:35:20 AM AEST 2026  env=marigold
+============================================
+The config attributes {'prediction_type': 'depth'} were passed to MarigoldDepthPipeline, but are not expected and will be ignored. Please verify your model_index.json configuration file.
+Keyword arguments {'prediction_type': 'depth'} are not expected by MarigoldDepthPipeline and will be ignored.
+
+
+
+Found 2 scenes
+  [1/2] sample_data: shape=(720, 1280)
+Saved 2 predictions to /home/ywan0794/EvalMDE/output/smoke_all/marigold
+[INF-OK] marigold
+
+============================================
+[lotus inference] Thu May 14 11:35:37 AM AEST 2026  env=lotus
+============================================
+
+Found 2 scenes
+  [1/2] sample_data: shape=(720, 1280)
+Saved 2 predictions to /home/ywan0794/EvalMDE/output/smoke_all/lotus
+[INF-OK] lotus
+
+============================================
+[depthmaster inference] Thu May 14 11:35:52 AM AEST 2026  env=depthmaster
+============================================
+The config attributes {'default_denoising_steps': 10, 'scheduler': ['diffusers', 'DDIMScheduler']} were passed to DepthMasterPipeline, but are not expected and will be ignored. Please verify your model_index.json configuration file.
+Keyword arguments {'default_denoising_steps': 10, 'scheduler': ['diffusers', 'DDIMScheduler']} are not expected by DepthMasterPipeline and will be ignored.
+
+
+
+Defaulting to unsafe serialization. Pass `allow_pickle=False` to raise an error instead.
+Some weights of the model checkpoint at /home/ywan0794/EvalMDE/DepthMaster/ckpt/eval/unet were not used when initializing UNet2DConditionModel: 
+ ['fftblock.fuse.bias, fftblock.conv_f1.weight, fftblock.conv_f2.weight, fftblock.conv_f4.weight, fftblock.conv_s2.bias, fftblock.fuse.weight, fftblock.conv_f2.bias, fftblock.conv_f3.bias, fftblock.conv_s1.weight, fftblock.conv_f3.weight, fftblock.conv_f4.bias, fftblock.norm.weight, fftblock.conv_f1.bias, fftblock.conv_s2.weight, fftblock.conv_s1.bias, fftblock.norm.bias']
+
+Expected types for unet: (<class 'depthmaster.modules.unet_2d_condition_s2.UNet2DConditionModel'>,), got <class 'diffusers.models.unets.unet_2d_condition.UNet2DConditionModel'>.
+An error occurred while trying to fetch /home/ywan0794/EvalMDE/DepthMaster/ckpt/eval/unet: Error no file named diffusion_pytorch_model.safetensors found in directory /home/ywan0794/EvalMDE/DepthMaster/ckpt/eval/unet.
+Defaulting to unsafe serialization. Pass `allow_pickle=False` to raise an error instead.
+Found 2 scenes
+  [1/2] sample_data: shape=(720, 1280)
+Saved 2 predictions to /home/ywan0794/EvalMDE/output/smoke_all/depthmaster
+[INF-OK] depthmaster
+
+============================================
+[ppd inference] Thu May 14 11:36:09 AM AEST 2026  env=ppd
+============================================
+xFormers not available
+xFormers not available
+Found 2 scenes
+  [1/2] sample_data: shape=(720, 1280)
+Saved 2 predictions to /home/ywan0794/EvalMDE/output/smoke_all/ppd
+[INF-OK] ppd
+
+============================================
+[da3_mono inference] Thu May 14 11:36:29 AM AEST 2026  env=da3
+============================================
+[WARN ] Dependency `gsplat` is required for rendering 3DGS. Install via: pip install git+https://github.com/nerfstudio-project/gsplat.git@0b4dddf04cb687367602c01196913cde6a743d70
+Found 2 scenes
+  [1/2] sample_data: shape=(720, 1280)
+Saved 2 predictions to /home/ywan0794/EvalMDE/output/smoke_all/da3_mono
+[INF-OK] da3_mono
+
+============================================
+[fe2e inference] Thu May 14 11:36:45 AM AEST 2026  env=fe2e
+============================================
+[INFO] prompt_type=empty, 跳过Qwen模型加载
+create LoRA network from weights
+train all blocks only
+create LoRA for DIT all blocks: 304 modules.
+enable LoRA for U-Net
+weights are merged
+Found 2 scenes
+  [1/2] sample_data: shape=(720, 1280)
+Saved 2 predictions to /home/ywan0794/EvalMDE/output/smoke_all/fe2e
+[INF-OK] fe2e
+
+============================================
+Stage 2: metric aggregation (evalmde env)
+============================================
+--- metric: depth_pro ---
+Found 2 scenes for depth_pro
+/home/ywan0794/miniconda3/envs/evalmde/lib/python3.10/site-packages/torch/functional.py:554: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at /pytorch/aten/src/ATen/native/TensorShape.cpp:4314.)
+  return _VF.meshgrid(tensors, **kwargs)  # type: ignore[attr-defined]
+  sample_data: sawa_h raw=0.591 aln=0.586  |  relnorm raw=0.192 aln=0.184  |  boundF1_err raw=0.736 aln=0.796
+  sample_data_2: sawa_h raw=1.268 aln=1.572  |  relnorm raw=0.390 aln=0.510  |  boundF1_err raw=0.513 aln=0.730
+
+Mean RAW    : {'wkdr_no_align': 0.040659576654434204, 'delta0125_disparity_affine_err': 0.4819862172007561, 'delta0125_depth_affine_err': 0.5122604453936219, 'boundary_f1_err': 0.6248274250948582, 'rel_normal': 0.2909630531817561, 'sawa_h': 0.9297213865303355}
+Mean ALIGNED: {'wkdr_no_align': 0.04262185096740723, 'delta0125_disparity_affine_err': 0.5153419096022844, 'delta0125_depth_affine_err': 0.5080458391457796, 'boundary_f1_err': 0.76309088091753, 'rel_normal': 0.3469443633141419, 'sawa_h': 1.0791019991638469}
+Saved → /home/ywan0794/EvalMDE/output/smoke_all/depth_pro_metrics.json
+[METRIC-OK] depth_pro
+--- metric: marigold ---
+Found 2 scenes for marigold
+/home/ywan0794/miniconda3/envs/evalmde/lib/python3.10/site-packages/torch/functional.py:554: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at /pytorch/aten/src/ATen/native/TensorShape.cpp:4314.)
+  return _VF.meshgrid(tensors, **kwargs)  # type: ignore[attr-defined]
+  sample_data: sawa_h raw=1.157 aln=0.930  |  relnorm raw=0.364 aln=0.301  |  boundF1_err raw=0.823 aln=0.929
+  sample_data_2: sawa_h raw=2.127 aln=2.129  |  relnorm raw=0.700 aln=0.703  |  boundF1_err raw=0.927 aln=0.923
+
+Mean RAW    : {'wkdr_no_align': 0.06999608874320984, 'delta0125_disparity_affine_err': 0.9599380735307932, 'delta0125_depth_affine_err': 0.6116695962846279, 'boundary_f1_err': 0.8753077063320032, 'rel_normal': 0.5323382569438899, 'sawa_h': 1.6421890328486521}
+Mean ALIGNED: {'wkdr_no_align': 0.07159394025802612, 'delta0125_disparity_affine_err': 0.5699970349669456, 'delta0125_depth_affine_err': 0.6097928117960691, 'boundary_f1_err': 0.9258331507737796, 'rel_normal': 0.5021106104687787, 'sawa_h': 1.5292764908179928}
+Saved → /home/ywan0794/EvalMDE/output/smoke_all/marigold_metrics.json
+[METRIC-OK] marigold
+--- metric: lotus ---
+Found 2 scenes for lotus
+/home/ywan0794/miniconda3/envs/evalmde/lib/python3.10/site-packages/torch/functional.py:554: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at /pytorch/aten/src/ATen/native/TensorShape.cpp:4314.)
+  return _VF.meshgrid(tensors, **kwargs)  # type: ignore[attr-defined]
+  sample_data: sawa_h raw=1.295 aln=1.095  |  relnorm raw=0.385 aln=0.339  |  boundF1_err raw=0.917 aln=0.865
+  sample_data_2: sawa_h raw=2.195 aln=2.157  |  relnorm raw=0.710 aln=0.690  |  boundF1_err raw=0.948 aln=0.937
+
+Mean RAW    : {'wkdr_no_align': 0.0865098237991333, 'delta0125_disparity_affine_err': 0.9658380672335625, 'delta0125_depth_affine_err': 0.6983374059200287, 'boundary_f1_err': 0.9324993093468176, 'rel_normal': 0.5473170225478743, 'sawa_h': 1.7448899686403179}
+Mean ALIGNED: {'wkdr_no_align': 0.08659347891807556, 'delta0125_disparity_affine_err': 0.6961807310581207, 'delta0125_depth_affine_err': 0.6983374059200287, 'boundary_f1_err': 0.900953527425987, 'rel_normal': 0.5143106302233235, 'sawa_h': 1.6263154318190827}
+Saved → /home/ywan0794/EvalMDE/output/smoke_all/lotus_metrics.json
+[METRIC-OK] lotus
+--- metric: depthmaster ---
+Found 2 scenes for depthmaster
+/home/ywan0794/miniconda3/envs/evalmde/lib/python3.10/site-packages/torch/functional.py:554: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at /pytorch/aten/src/ATen/native/TensorShape.cpp:4314.)
+  return _VF.meshgrid(tensors, **kwargs)  # type: ignore[attr-defined]
+  sample_data: sawa_h raw=4.475 aln=1.124  |  relnorm raw=0.333 aln=0.293  |  boundF1_err raw=0.998 aln=0.925
+  sample_data_2: sawa_h raw=4.774 aln=2.044  |  relnorm raw=0.586 aln=0.654  |  boundF1_err raw=0.991 aln=0.933
+
+Mean RAW    : {'wkdr_no_align': 0.9196415990591049, 'delta0125_disparity_affine_err': 0.9356035124510527, 'delta0125_depth_affine_err': 0.9198634652420878, 'boundary_f1_err': 0.9947631304516369, 'rel_normal': 0.4595408906976621, 'sawa_h': 4.624761057503135}
+Mean ALIGNED: {'wkdr_no_align': 0.08632892370223999, 'delta0125_disparity_affine_err': 0.8615778312087059, 'delta0125_depth_affine_err': 0.9183715572580695, 'boundary_f1_err': 0.9292771149464188, 'rel_normal': 0.4737114471098748, 'sawa_h': 1.5842239270857643}
+Saved → /home/ywan0794/EvalMDE/output/smoke_all/depthmaster_metrics.json
+[METRIC-OK] depthmaster
+--- metric: ppd ---
+Found 2 scenes for ppd
+/home/ywan0794/miniconda3/envs/evalmde/lib/python3.10/site-packages/torch/functional.py:554: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at /pytorch/aten/src/ATen/native/TensorShape.cpp:4314.)
+  return _VF.meshgrid(tensors, **kwargs)  # type: ignore[attr-defined]
+  sample_data: sawa_h raw=2.198 aln=1.901  |  relnorm raw=0.837 aln=0.732  |  boundF1_err raw=0.857 aln=0.813
+  sample_data_2: sawa_h raw=2.535 aln=2.444  |  relnorm raw=0.905 aln=0.852  |  boundF1_err raw=0.922 aln=0.860
+
+Mean RAW    : {'wkdr_no_align': 0.0871766209602356, 'delta0125_disparity_affine_err': 0.9600839260965586, 'delta0125_depth_affine_err': 0.7343441895209253, 'boundary_f1_err': 0.8895310134282803, 'rel_normal': 0.8712330618410348, 'sawa_h': 2.366451557754713}
+Mean ALIGNED: {'wkdr_no_align': 0.09249627590179443, 'delta0125_disparity_affine_err': 0.6880950853228569, 'delta0125_depth_affine_err': 0.7330712396651506, 'boundary_f1_err': 0.8366181924177966, 'rel_normal': 0.7918025637799675, 'sawa_h': 2.172219847013012}
+Saved → /home/ywan0794/EvalMDE/output/smoke_all/ppd_metrics.json
+[METRIC-OK] ppd
+--- metric: da3_mono ---
+Found 2 scenes for da3_mono
+/home/ywan0794/miniconda3/envs/evalmde/lib/python3.10/site-packages/torch/functional.py:554: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at /pytorch/aten/src/ATen/native/TensorShape.cpp:4314.)
+  return _VF.meshgrid(tensors, **kwargs)  # type: ignore[attr-defined]
+  sample_data: sawa_h raw=0.845 aln=0.707  |  relnorm raw=0.207 aln=0.177  |  boundF1_err raw=0.931 aln=0.979
+  sample_data_2: sawa_h raw=1.447 aln=1.688  |  relnorm raw=0.454 aln=0.561  |  boundF1_err raw=0.821 aln=0.832
+
+Mean RAW    : {'wkdr_no_align': 0.047021448612213135, 'delta0125_disparity_affine_err': 0.8515407452359796, 'delta0125_depth_affine_err': 0.5784242674708366, 'boundary_f1_err': 0.8758562543377832, 'rel_normal': 0.3301750209283423, 'sawa_h': 1.1464006557203033}
+Mean ALIGNED: {'wkdr_no_align': 0.051027655601501465, 'delta0125_disparity_affine_err': 0.6030512889847159, 'delta0125_depth_affine_err': 0.5845414977520704, 'boundary_f1_err': 0.905371028814778, 'rel_normal': 0.36910983958422094, 'sawa_h': 1.1977928844965942}
+Saved → /home/ywan0794/EvalMDE/output/smoke_all/da3_mono_metrics.json
+[METRIC-OK] da3_mono
+--- metric: fe2e ---
+Found 2 scenes for fe2e
+/home/ywan0794/miniconda3/envs/evalmde/lib/python3.10/site-packages/torch/functional.py:554: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at /pytorch/aten/src/ATen/native/TensorShape.cpp:4314.)
+  return _VF.meshgrid(tensors, **kwargs)  # type: ignore[attr-defined]
+  sample_data: sawa_h raw=1.113 aln=0.967  |  relnorm raw=0.332 aln=0.282  |  boundF1_err raw=0.879 aln=0.837
+  sample_data_2: sawa_h raw=1.921 aln=2.007  |  relnorm raw=0.615 aln=0.642  |  boundF1_err raw=0.792 aln=0.843
+
+Mean RAW    : {'wkdr_no_align': 0.06889474391937256, 'delta0125_disparity_affine_err': 0.9601075295358896, 'delta0125_depth_affine_err': 0.6917768018320203, 'boundary_f1_err': 0.8355541301758247, 'rel_normal': 0.47354231407887987, 'sawa_h': 1.516985853988682}
+Mean ALIGNED: {'wkdr_no_align': 0.07494029402732849, 'delta0125_disparity_affine_err': 0.7892066687345505, 'delta0125_depth_affine_err': 0.6908030491322279, 'boundary_f1_err': 0.8398841726062642, 'rel_normal': 0.46222504542092735, 'sawa_h': 1.4871907267011424}
+Saved → /home/ywan0794/EvalMDE/output/smoke_all/fe2e_metrics.json
+[METRIC-OK] fe2e
+
+============================================
+smoke-all finished at Thu May 14 11:37:36 AM AEST 2026
+=== Summary ===
+[INF-OK] depth_pro
+[INF-OK] marigold
+[INF-OK] lotus
+[INF-OK] depthmaster
+[INF-OK] ppd
+[INF-OK] da3_mono
+[INF-OK] fe2e
+[METRIC-OK] depth_pro
+[METRIC-OK] marigold
+[METRIC-OK] lotus
+[METRIC-OK] depthmaster
+[METRIC-OK] ppd
+[METRIC-OK] da3_mono
+[METRIC-OK] fe2e
+=== Per-model means ===
+Traceback (most recent call last):
+  File "<string>", line 1, in <module>
+KeyError: 'mean'
+depth_pro: 
+Traceback (most recent call last):
+  File "<string>", line 1, in <module>
+KeyError: 'mean'
+marigold: 
+Traceback (most recent call last):
+  File "<string>", line 1, in <module>
+KeyError: 'mean'
+lotus: 
+Traceback (most recent call last):
+  File "<string>", line 1, in <module>
+KeyError: 'mean'
+depthmaster: 
+Traceback (most recent call last):
+  File "<string>", line 1, in <module>
+KeyError: 'mean'
+ppd: 
+Traceback (most recent call last):
+  File "<string>", line 1, in <module>
+KeyError: 'mean'
+da3_mono: 
+Traceback (most recent call last):
+  File "<string>", line 1, in <module>
+KeyError: 'mean'
+fe2e: 

smoke_evalmde_12112.log

@@ -0,0 +1,2 @@
+=== Smoke step 1: depth_pro inference ===
+/var/spool/slurmd/job12112/slurm_script: line 26: PYTHONPATH: unbound variable

smoke_evalmde_12113.log

@@ -0,0 +1,34 @@
+=== Smoke step 1: depth_pro inference ===
+Found 2 scenes
+  [1/2] sample_data: shape=(720, 1280)
+Saved 2 predictions to /home/ywan0794/EvalMDE/output/smoke/depth_pro
+=== Smoke step 2: compute_metrics in evalmde env ===
+Found 2 scenes for depth_pro
+/home/ywan0794/miniconda3/envs/evalmde/lib/python3.10/site-packages/torch/functional.py:554: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at /pytorch/aten/src/ATen/native/TensorShape.cpp:4314.)
+  return _VF.meshgrid(tensors, **kwargs)  # type: ignore[attr-defined]
+  sample_data: sawa_h=0.5910, rel_normal=0.1919
+  sample_data_2: sawa_h=1.2678, rel_normal=0.3900
+
+Mean: {'sawa_h': 0.9293891770624477, 'rel_normal': 0.2909630531817561}
+Saved → /home/ywan0794/EvalMDE/output/smoke/depth_pro_metrics.json
+=== Smoke summary ===
+{
+  "model": "depth_pro",
+  "n_scenes": 2,
+  "per_scene": [
+    {
+      "scene": "sample_data",
+      "sawa_h": 0.5910246923517875,
+      "rel_normal": 0.19190798903378145
+    },
+    {
+      "scene": "sample_data_2",
+      "sawa_h": 1.267753661773108,
+      "rel_normal": 0.3900181173297307
+    }
+  ],
+  "mean": {
+    "sawa_h": 0.9293891770624477,
+    "rel_normal": 0.2909630531817561
+  }
+}

smoke_lotus_v1_12348.log

@@ -0,0 +1,20 @@
+=== Stage 1: lotus v1-0 inference ===
+
+Found 2 scenes
+  [1/2] sample_data: shape=(720, 1280)
+Saved 2 predictions to /home/ywan0794/EvalMDE/output/smoke_lotus_v1/lotus
+
+=== Stage 2: metric (evalmde env, dual-track) ===
+Found 2 scenes for lotus
+/home/ywan0794/miniconda3/envs/evalmde/lib/python3.10/site-packages/torch/functional.py:554: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at /pytorch/aten/src/ATen/native/TensorShape.cpp:4314.)
+  return _VF.meshgrid(tensors, **kwargs)  # type: ignore[attr-defined]
+  sample_data: sawa_h raw=1.296 aln=1.098  |  relnorm raw=0.385 aln=0.339  |  boundF1_err raw=0.917 aln=0.866
+  sample_data_2: sawa_h raw=2.194 aln=2.158  |  relnorm raw=0.710 aln=0.690  |  boundF1_err raw=0.948 aln=0.937
+
+Mean RAW    : {'wkdr_no_align': 0.08665573596954346, 'delta0125_disparity_affine_err': 0.9616885241121054, 'delta0125_depth_affine_err': 0.6993474271148443, 'boundary_f1_err': 0.9321780361920082, 'rel_normal': 0.5475669290628759, 'sawa_h': 1.7451062945205418}
+Mean ALIGNED: {'wkdr_no_align': 0.08664768934249878, 'delta0125_disparity_affine_err': 0.6976255280897021, 'delta0125_depth_affine_err': 0.6993474271148443, 'boundary_f1_err': 0.9015399104142445, 'rel_normal': 0.5147056572154382, 'sawa_h': 1.6276670925082142}
+Saved → /home/ywan0794/EvalMDE/output/smoke_lotus_v1/lotus_metrics.json
+
+=== Summary ===
+RAW    mean: {'wkdr_no_align': 0.08665573596954346, 'delta0125_disparity_affine_err': 0.9616885241121054, 'delta0125_depth_affine_err': 0.6993474271148443, 'boundary_f1_err': 0.9321780361920082, 'rel_normal': 0.5475669290628759, 'sawa_h': 1.7451062945205418}
+ALIGNED mean: {'wkdr_no_align': 0.08664768934249878, 'delta0125_disparity_affine_err': 0.6976255280897021, 'delta0125_depth_affine_err': 0.6993474271148443, 'boundary_f1_err': 0.9015399104142445, 'rel_normal': 0.5147056572154382, 'sawa_h': 1.6276670925082142}