DeepChoice

DeepChoice is a lightweight multi-view fusion framework for image-guided 3D semantic segmentation.
For each 3D point, the preprocessing pipeline gathers the visible images, computes geometric and radiometric visibility criteria, attaches per-view 2D semantic scores, and the model learns one weight per view before fusing the evidence into a final 3D prediction.

This repository contains:

• the DeepChoice training, evaluation, and inference code
• preprocessing pipelines for GridNet-HD and Cultural Heritage
• two DeepChoice backbones:
  • DeepChoiceTransformer
  • DeepChoiceMLP
• a 2D image baseline used to export per-image logits / softmax scores
• public config files and released checkpoints

Citation

If you use this repository, please cite:

@article{Carreaud2026DeepChoice,
  title={DeepChoice: Learning View Weighting for Image-Guided 3D Semantic Segmentation},
  author={Antoine Carreaud and Digre Frinde and Shanci Li and Jan Skaloud and Adrien Gressin},
  year={2026},
  url={https://cspaper.org/openprint/20260331.0001v1},
  journal={OpenPrint:20260331.0001v1}
}

Repository Layout

.
├── main.py
├── configs/
├── dataset/
├── dataset_generation/
├── experiments/
├── image_baseline/
├── model/
├── scripts/
├── train/
├── utils/
└── weights/

Main directories:

• dataset_generation/: preprocessing and batch generation
• dataset/: PyTorch datasets for training and evaluation
• model/: DeepChoice backbones
• train/: training, testing, and inference logic
• image_baseline/: 2D segmentation baseline used to export image scores
• experiments/: reusable experiment runners
• scripts/: small utilities for figure generation or format conversion
• weights/: released checkpoints

Public Config Files

DeepChoice configs:

• configs/config_deepchoice_base.yaml
• configs/config_gridnet_hd_transformer.yaml
• configs/config_gridnet_hd_mlp.yaml
• configs/config_cultural_heritage_transformer.yaml
• configs/config_cultural_heritage_mlp.yaml

Dataset generation configs:

• configs/config_gridnet_hd_dataset_generation.yaml
• configs/config_cultural_heritage_dataset_generation.yaml

Released Checkpoints

DeepChoice:

• weights/deepchoice/gridnet_hd_transformer_best_model.pt
• weights/deepchoice/gridnet_hd_mlp_best_model.pt
• weights/deepchoice/cultural_heritage_transformer_best_model.pt
• weights/deepchoice/cultural_heritage_mlp_best_model.pt

Image baseline:

• weights/image_baseline/gridnet_hd_best_model.pth
• weights/image_baseline/cultural_heritage_best_model.pth

Main Results

The tables below summarize the main quantitative results reported in the paper.

GridNet-HD test split

Method	mIoU (%)	Params
Hard vote	66.78	60 M
Mean-probability fusion	69.37	60 M
DeepChoice-MLP	70.52	60 M + 0.09 M
DeepChoice-Transformer	70.63	60 M + 0.02 M
AnyView oracle	84.33	-

On the full GridNet-HD benchmark, DeepChoice-Transformer improves over hard voting by +3.85 mIoU points and over mean-probability fusion by +1.26 mIoU points.

Cultural Heritage (3_SS)

Method	mIoU (%)	mF1 (%)
Hard vote	65.66	73.87
Mean-probability fusion	66.73	74.81
DeepChoice-MLP	66.35	74.54
DeepChoice-Transformer	67.21	75.49
AnyView oracle	86.92	92.83

On Images&PointClouds Cultural Heritage, DeepChoice-Transformer improves over hard voting by +1.55 mIoU points and over mean-probability fusion by +0.48 mIoU points.

Environment

Reference environment used during development:

Component	Version
OS	Ubuntu 24.04
Python	3.12
PyTorch	2.7
CUDA	12.x
JAX	0.6
laspy	2.x

Installation:

git clone https://huggingface.co/heig-vd-geo/DeepChoice
cd DeepChoice

conda create -n deepchoice python=3.12
conda activate deepchoice

pip install --upgrade pip
pip install -r requirements.txt

The repository includes a .gitattributes file configured for Git LFS so released .pt and .pth checkpoints can be pushed cleanly.

GridNet-HD Preprocessing

The GridNet-HD preprocessing pipeline expects, for each tile:

• a LAS point cloud
• oriented images
• camera poses and intrinsics / extrinsics
• per-image logits or softmax scores exported by the 2D baseline

Configuration:

• configs/config_gridnet_hd_dataset_generation.yaml

The GridNet-HD preprocessing config expects the official split file distributed with the dataset at ./data/gridnet_hd/public/split.json.

Compute normals:

python scripts/compute_normals.py --config configs/config_gridnet_hd_dataset_generation.yaml

Generate DeepChoice batches:

python scripts/run_batch_generation.py --config configs/config_gridnet_hd_dataset_generation.yaml

For each visible (point, image) pair, the pipeline computes:

• incidence angle
• distance
• local contrast
• blur response
• signal-to-noise ratio
• saturation

Generated batches are written under:

• artifacts/gridnet_hd/batches/train
• artifacts/gridnet_hd/batches/val
• artifacts/gridnet_hd/batches/test

Each .pt batch contains:

• visibility: [N, max_views, num_features]
• logits: [N, max_views, num_classes]
• mask: [N, max_views]
• target: [N]
• coords_int
• coords_scale
• coords_offset
• coords_tile_offset

Cultural Heritage Preprocessing

Configuration:

• configs/config_cultural_heritage_dataset_generation.yaml

Compute normals:

python scripts/compute_normals_cultural_heritage.py --config configs/config_cultural_heritage_dataset_generation.yaml

Generate batches:

python scripts/run_batch_generation_cultural_heritage.py --config configs/config_cultural_heritage_dataset_generation.yaml

This setup is useful for internal experiments and custom split protocols. Exact reproduction of a paper-specific Cultural Heritage protocol may require additional split management.

DeepChoice Training

Main DeepChoice entry point:

• main.py

Supported modes:

• train
• test
• infer

Base config:

• configs/config_deepchoice_base.yaml

Example single-process training:

python main.py --config configs/config_deepchoice_base.yaml --mode train

Example distributed training:

torchrun --standalone --nproc_per_node=4 main.py --config configs/config_deepchoice_base.yaml --mode train

Useful config sections:

• data: batch roots and split-specific overrides
• dataset: selected visibility features and normalization ranges
• model: backbone choice and architecture hyperparameters
• training: optimizer, scheduler, dataloading, and DDP settings
• test: evaluation and inference split settings

Important training options:

• training.file_batch_size
• training.eval_file_batch_size
• training.train_limit_files
• training.randomize_train_limit_each_epoch
• training.val_split

Evaluation

Example evaluation with baselines:

python main.py \
  --config configs/config_deepchoice_base.yaml \
  --mode test \
  --weights_path /path/to/best_model.pt \
  --split test

The test pipeline reports:

• model mIoU and mF1
• hard_vote
• mean_prob_vote
• anyview

Inference and LAS Export

Standard inference:

python main.py \
  --config configs/config_deepchoice_base.yaml \
  --mode infer \
  --weights_path /path/to/best_model.pt \
  --split test \
  --inference_output /path/to/output_dir

Inference with an additional comparison model:

python main.py \
  --config /path/to/transformer_config.yaml \
  --mode infer \
  --weights_path /path/to/transformer_best_model.pt \
  --comparison_config /path/to/mlp_config.yaml \
  --comparison_weights_path /path/to/mlp_best_model.pt \
  --comparison_field_name best_mlp \
  --split test \
  --inference_output /path/to/output_dir

Generated outputs:

• pred_batch_*.pt
• one LAS file per tile

Each LAS may contain:

• classification: primary model prediction
• ground_truth
• best_transformer
• best_mlp
• mean_prob_vote
• hard_vote

Image Baseline

The repository also contains the 2D image segmentation baseline used to export per-image semantic scores:

• image_baseline/main.py

Provided configs:

• image_baseline/configs/gridnet_hd.yaml
• image_baseline/configs/cultural_heritage.yaml

Train:

python image_baseline/main.py --config image_baseline/configs/gridnet_hd.yaml --mode train

Validate:

python image_baseline/main.py \
  --config image_baseline/configs/gridnet_hd.yaml \
  --mode val \
  --weights_path /path/to/best_model.pth

Export logits / softmax scores:

python image_baseline/main.py \
  --config image_baseline/configs/gridnet_hd.yaml \
  --mode export_logits \
  --weights_path /path/to/best_model.pth

The exported arrays are stored as .npy files following the dataset-relative image structure.

Experiment Runners

Reusable experiment runners:

• experiments/run_weekend_experiments.py
• experiments/run_validation_experiments.py

Paper-related plans included in the repository:

• experiments/weekend_vislogits_nobalanced_plan.yaml
• experiments/weekend_vislogits_maxviews_plan.yaml
• experiments/test_vislogits_best_plan.yaml
• experiments/weekend_vislogits_cultural_heritage_plan.yaml
• experiments/test_vislogits_cultural_heritage_best_plan.yaml

Typical launch:

python experiments/run_weekend_experiments.py \
  --base-config configs/config_deepchoice_base.yaml \
  --plan /path/to/weekend_plan.yaml \
  --output-root /path/to/output_root \
  --continue-on-error

GridNet-HD input ablation:

python experiments/run_weekend_experiments.py \
  --base-config configs/config_deepchoice_base.yaml \
  --plan experiments/weekend_vislogits_nobalanced_plan.yaml \
  --output-root /path/to/output_root \
  --continue-on-error

GridNet-HD max-views ablation:

python experiments/run_weekend_experiments.py \
  --base-config configs/config_deepchoice_base.yaml \
  --plan experiments/weekend_vislogits_maxviews_plan.yaml \
  --output-root /path/to/output_root \
  --continue-on-error

GridNet-HD final test evaluation:

python experiments/run_validation_experiments.py \
  --base-config configs/config_deepchoice_base.yaml \
  --plan experiments/test_vislogits_best_plan.yaml \
  --output-root /path/to/output_root

Cultural Heritage experiment:

python experiments/run_weekend_experiments.py \
  --base-config configs/config_deepchoice_base.yaml \
  --plan experiments/weekend_vislogits_cultural_heritage_plan.yaml \
  --output-root /path/to/output_root \
  --continue-on-error

Cultural Heritage evaluation of the released best models:

python experiments/run_validation_experiments.py \
  --base-config configs/config_deepchoice_base.yaml \
  --plan experiments/test_vislogits_cultural_heritage_best_plan.yaml \
  --output-root /path/to/output_root

Utility Scripts

Currently kept in scripts/:

• scripts/compute_normals.py
• scripts/compute_normals_cultural_heritage.py
• scripts/run_batch_generation.py
• scripts/run_batch_generation_cultural_heritage.py
• scripts/make_visibility_paper_figure.py
• scripts/convert_cultural_heritage_txt_to_las.py