ilessio-aiflowlab/project_gs_ghost

GS-GHOST — ANIMA Module

Part of the ANIMA Perception Suite by Robot Flow Labs.

Paper

GHOST: Fast Category-agnostic Hand-Object Interaction Reconstruction from RGB Videos using Gaussian Splatting (arXiv:2603.18912)

Ahmed Tawfik Aboukhadra, Marcel Rogge, Nadia Robertini, Abdalla Arafa, Jameel Malik, Ahmed Elhayek, Didier Stricker

Architecture

GHOST reconstructs hand-object interactions from monocular RGB video using 3D Gaussian Splatting:

1. Preprocessing: SAM2 masks + SfM (VGGSfM/HLoc) + geometric prior retrieval
2. HO Alignment: Grasp detection + joint scale/translation optimization
3. Object GS: 30K-iteration object-only Gaussian optimization with L_rgb + L_bkg,h + L_geo
4. Combined GS: 30K-iteration joint hand-object optimization with mesh-bound hand Gaussians

Current Status

Synthetic validation checkpoint — trained on procedural sphere data to validate the GPU pipeline. Real ARCTIC/HO3D training requires dataset provisioning.

Exported Formats

Format	File	Use Case
PyTorch (.pth)	pytorch/gs_ghost_v1.pth	Training, fine-tuning
SafeTensors	pytorch/gs_ghost_v1.safetensors	Fast loading, safe
ONNX	N/A	GS rasterizer is CUDA-only
TensorRT	N/A	Requires ONNX first

Usage

import torch
ckpt = torch.load("pytorch/gs_ghost_v1.pth", map_location="cpu")
# Keys: xyz, scaling, rotation, opacity, features_dc, features_rest, n_gaussians, step

Training

• Hardware: NVIDIA L4 (23GB VRAM)
• Framework: PyTorch 2.11 + cu128
• Rasterizer: GRAPHDECO semantic Gaussian rasterizer (shared CUDA extension)
• Speed: 135 it/s on L4 at 480x640
• Config: See configs/training.toml

Benchmarks (Paper Targets)

Metric	Paper	Our Target
ARCTIC CDh	18.40 cm²	≤ 20 cm²
ARCTIC PSNR	25.93	≥ 25.0
HO3D LPIPS	0.03	≤ 0.04

License

Apache 2.0 — Robot Flow Labs / AIFLOW LABS LIMITED