README.md

---
tags:
- ml-intern
---
# Point-SAM: Promptable 3D Segmentation

A clean, self-contained Python inference package for **Point-SAM** (ICLR 2025), extending SAM's promptable segmentation to 3D point clouds.

> **Paper**: [Point-SAM: Promptable 3D Segmentation Model for Point Clouds](https://arxiv.org/abs/2406.17741)  
> **Original Code**: [github.com/zyc00/Point-SAM](https://github.com/zyc00/Point-SAM)  
> **Pretrained Weights**: [`yuchen0187/Point-SAM`](https://huggingface.co/yuchen0187/Point-SAM)

---

## Quick Start

```bash
pip install torch timm safetensors huggingface_hub numpy
```

```python
from point_sam import PointSAM, load_pointcloud

# 1. Load a point cloud (PLY or PCD)
coords, rgb, original = load_pointcloud("scene.ply")
# coords: [N, 3] normalized to [-1, 1]
# rgb:    [N, 3] in [0, 255]

# 2. Load the pretrained model (downloads weights from HF Hub)
model = PointSAM.from_pretrained(checkpoint_path="model.safetensors", device="cuda")

# 3. Cache the cloud for fast repeated queries
model.set_pointcloud(coords, rgb)

# 4. Segment with a prompt point (in normalized [-1, 1] space)
masks, iou_scores = model.predict(
    coords=None,           # use cached cloud
    rgb=None,
    prompt_point=[0.5, 0.1, -0.2],
    prompt_label=1,        # 1 = foreground, 0 = background
    multimask_output=True,
)

# 5. Pick the best mask by IoU score
best_mask = masks[iou_scores.argmax()]   # [N] boolean
```

Command-line example:

```bash
python examples/segment_ply.py scene.ply 0.5 0.1 -0.2 --checkpoint model.safetensors
```

---

## How It Works Internally

Point-SAM is a direct 3D adaptation of [SAM](https://github.com/facebookresearch/segment-anything). It has the same three-part architecture, but replaces the 2D image backbone with a **point cloud encoder**.

### 1. Point-Cloud Encoder

The encoder turns an unstructured point cloud into a compact set of **patch embeddings** — the 3D equivalent of image patches.

**Voronoi Tokenizer** (the key speed trick)
- Sample `G` center points from the cloud via **Farthest Point Sampling** (FPS). This spreads centers evenly across the shape.
- Group each point with its **K nearest neighbors** around one of those centers.
- Run a small **PointNet-style MLP** on each group:
  - Input: relative XYZ positions + RGB colors
  - Max-pool over the K neighbors → one vector per group
- Result: `G` patch embeddings, each summarizing a local neighborhood.

**Vision Transformer (ViT) backbone**
- The patch embeddings are fed into a standard ViT — `eva02_large_patch14_448` for the *large* variant, or `eva_giant_patch14_560` for *giant*.
- The ViT adds learned positional embeddings based on the 3D center coordinates and runs self-attention to build a global scene representation.
- Output: `[B, num_patches, D]` embedding tensor (default `D = 256`).

### 2. Prompt Encoder

- **Point prompts**: A user clicks (or specifies) a 3D coordinate. The coordinate is mapped through a random Fourier positional encoding (same Gaussian-frequency trick SAM uses) and then a learned embedding is added depending on whether the label is **positive** (foreground) or **negative** (background).
- **Mask prompts** (optional): If you already have a rough mask from a previous iteration, it is grouped into patches (same KNN grouping as the encoder) and encoded into dense embeddings. On the first call this is `None`, so a learned "no mask" embedding is used instead.

### 3. Mask Decoder

The decoder is a **two-way transformer** — identical in spirit to SAM's decoder:

1. **Cross-attention layers** alternate between:
   - *Prompt tokens → point cloud patches* (the prompts "look at" the scene)
   - *Point cloud patches → prompt tokens* (the scene "looks back" at the prompts)
2. After 2 layers, a **final attention** from prompts to patches refines the token representation.
3. **Upsampling**: The decoder works at patch resolution. To get back to per-point logits, features are interpolated to every original point using **inverse-distance weighted KNN** (3 nearest patch centers).
4. **Hypernetwork MLPs**: Each candidate mask has its own tiny MLP that produces a dynamic weight vector. This vector is dot-producted with the upsampled per-point features to produce the final mask logits.
5. **IoU head**: A small MLP on the IoU token predicts the quality of each mask candidate. At inference time you simply pick the one with the highest predicted IoU.

The decoder always outputs **4 candidates** (1 default + 3 multimask). The first candidate is a "safe" single mask; the other three are alternatives at different granularities.

### 4. Iterative Prompt Refinement (training only)

During training, Point-SAM simulates a user iteratively adding prompts:
- Iteration 0: no prompt → random positive point from the target object.
- Iteration 1: previous mask is fed back as a mask prompt; a new point prompt is sampled from the **error region** (false positives / false negatives).
- ... repeated for 5 iterations (large model) or 10 (giant).

At **inference time** you only do a single forward pass with whatever prompt you provide — the model was trained to produce a good mask even from one point.

---

## Supported File Formats

| Format | Notes |
|--------|-------|
| **PLY** | ASCII `.ply` with `x y z r g b` columns |
| **PCD** | ASCII `.pcd` with `x y z r g b` columns (Point Cloud Library format) |

Both loaders normalize coordinates to a **unit sphere in [-1, 1]** and scale colors to **[0, 255]**. This normalization is **required** — the positional encoding will raise a `ValueError` if coordinates fall outside [-1, 1].

---

## Handling Large Point Clouds

If your cloud has > 100k points, increase the patch resolution to avoid OOM:

```python
model.adjust_patch_params(num_groups=2048, group_size=256)
```

The default is `num_groups=1024, group_size=256` for the large model.

---

## What Changed From the Original Repo?

| Original | This Package |
|----------|-------------|
| Requires `hydra` + `omegaconf` for config | Pure Python, no YAML configs needed |
| Requires compiling `torkit3d` (CUDA ops) | Pure-PyTorch FPS, KNN, and index operations |
| Requires compiling `apex` for FusedLayerNorm | Standard `nn.LayerNorm` by default; apex optional |
| Scattered evaluation scripts | One clean `PointSAM` class with `predict()` |
| Heavy training codebase | Only inference + minimal model code |

---

## Citation

```bibtex
@inproceedings{
  zhou2025pointsam,
  title={Point-{SAM}: Promptable 3D Segmentation Model for Point Clouds},
  author={Yuchen Zhou and Jiayuan Gu and Tung Yen Chiang and Fanbo Xiang and Hao Su},
  booktitle={The Thirteenth International Conference on Learning Representations},
  year={2025},
  url={https://openreview.net/forum?id=yXCTDhZDh6}
}
```

## License

MIT (same as the original repository).

<!-- ml-intern-provenance -->
## Generated by ML Intern

This model repository was generated by [ML Intern](https://github.com/huggingface/ml-intern), an agent for machine learning research and development on the Hugging Face Hub.

- Try ML Intern: https://smolagents-ml-intern.hf.space
- Source code: https://github.com/huggingface/ml-intern

## Usage

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "bdck/point-sam-inference"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id)
```

For non-causal architectures, replace `AutoModelForCausalLM` with the appropriate `AutoModel` class.