Initial release

README.md

@@ -0,0 +1,121 @@
+---
+license: cc-by-nc-4.0
+tags:
+  - relighting
+  - inverse-rendering
+  - pbr
+  - image-to-image
+  - pytorch
+pipeline_tag: image-to-image
+library_name: pytorch
+---
+
+# PIXLRelight
+
+PIXLRelight is a feed-forward model for physically controllable single-image relighting. Given a source photograph and a target lighting condition — specified either as an RGB image or as a path-traced Blender Cycles render — it produces a relit version of the source under the new illumination in under a tenth of a second.
+
+This repository hosts the trained model checkpoint and matching configuration. Full code, training details, and additional results are on the [project page](https://mlfarinha.github.io/pixl-relight/).
+
+## Abstract
+
+We present PIXLRelight, a feed-forward approach for physically controllable single-image relighting. Existing methods either provide limited lighting control (e.g. through text or environment maps), accumulate errors when chaining inverse and forward rendering, or require costly per-image optimization. Our key idea is to bridge physically based rendering (PBR) and learned image synthesis through a shared intrinsic conditioning that can be obtained from either real photographs or PBR renders. At training time, paired multi-illumination photographs are decomposed into albedo, diffuse shading, and non-diffuse residuals, which condition the model. At inference time, the same conditioning is computed from a path-traced render of a coarse 3D reconstruction of the input under user-specified PBR lights. A transformer-based neural renderer then applies the target illumination to the source photograph, preserving fine image detail through a per-pixel affine modulation. PIXLRelight enables arbitrary PBR-style lighting control, achieves state-of-the-art relighting quality, and runs in under a tenth of a second per image.
+
+## Files
+
+- `model.safetensors` — model weights
+- `config.yaml` — OmegaConf config defining the model architecture (`net`, `head`, `rgbx` submodules)
+
+## Quick start
+
+### Command line
+
+After cloning the [GitHub repository](https://github.com/mlfarinha/pixl-relight) and installing its requirements, place each sample in its own directory:
+
+    samples/
+    ├── room00/
+    │   ├── source.jpg
+    │   └── target.jpg                  # target lighting as an RGB image
+    ├── room01/
+    │   ├── source.jpg
+    │   └── target.jpg                  # target lighting as an RGB image
+    ├── room02/
+    │   ├── source.jpg
+    │   ├── diffuse_color.exr           # ...or as Blender Cycles passes
+    │   ├── diffuse_direct.exr
+    │   ├── diffuse_indirect.exr
+    │   ├── glossy_color.exr
+    │   ├── glossy_direct.exr
+    │   ├── glossy_indirect.exr
+    │   ├── transmission_color.exr
+    │   ├── transmission_direct.exr
+    │   ├── transmission_indirect.exr
+    │   ├── volume_direct.exr
+    │   ├── volume_indirect.exr
+    │   ├── environment.exr
+    │   └── emission.exr
+    ├── room03/
+    │   └── ...
+    └── ...
+
+Then run:
+
+    python infer.py --input_dir samples/ --output_dir outputs/
+
+The model and config are downloaded from this repository on first run and cached locally.
+
+### Python
+
+```python
+from huggingface_hub import hf_hub_download
+from omegaconf import OmegaConf
+from safetensors.torch import load_file
+
+from src.utils.cfg import create_object
+
+REPO_ID = "mlfarinha/pixlrelight"
+
+config_path = hf_hub_download(repo_id=REPO_ID, filename="config.yaml")
+ckpt_path = hf_hub_download(repo_id=REPO_ID, filename="model.safetensors")
+
+cfg = OmegaConf.load(config_path)
+model = create_object(cfg.model)
+model.load_state_dict(load_file(ckpt_path), strict=False)
+model.cuda().eval()
+
+# Target lighting as an RGB image (decomposed on the fly).
+relit = model.inference(
+    source_images="source.jpg",
+    target_images="target.jpg",
+    output_type="pil",
+)
+relit[0].save("relit.png")
+
+# Or as a directory of Blender Cycles render passes.
+relit = model.inference(
+    source_images="source.jpg",
+    cycles="path/to/cycles_passes/",
+    output_type="pil",
+)
+relit[0].save("relit.png")
+```
+
+## Limitations
+
+PIXLRelight is trained on indoor scenes and relights at the source's native resolution by applying a per-pixel affine modulation. The model does not synthesise content not present in the source — strong cast shadows or specular highlights that change the visible surface geometry may not transfer faithfully.
+
+## Citation
+
+If you use PIXLRelight in your work, please cite:
+
+```bibtex
+@article{farinha2026pixlrelight,
+  title     = {PIXLRelight: Controllable Relighting via Intrinsic Conditioning},
+  author    = {Farinha, Miguel and Clark, Ronald},
+  journal = {arXiv preprint arXiv:2605.18735},
+  year      = {2026}
+}
+```
+
+## License
+
+This model is released under the [Creative Commons Attribution-NonCommercial 4.0 International (CC BY-NC 4.0)](https://creativecommons.org/licenses/by-nc/4.0/) license. The model weights and configuration may be used for research and personal purposes with attribution, but not for commercial purposes.

config.yaml

@@ -0,0 +1,68 @@
+model:
+  __object__:
+    path: src.models.pixl_relight
+    name: PIXLRelight
+    args: as_params
+
+  net:
+    __object__:
+      path: src.models.net
+      name: Net
+      args: as_params
+
+    img_size: 512
+    patch_size: 16
+    pos_embed_rope_base: 100.0
+    pos_embed_rope_min_period: null
+    pos_embed_rope_max_period: null
+    pos_embed_rope_normalize_coords: separate
+    pos_embed_rope_shift_coords: null
+    pos_embed_rope_jitter_coords: null
+    pos_embed_rope_rescale_coords: null
+    pos_embed_rope_dtype: bf16
+    embed_dim: 1024
+    depth: 24
+    num_heads: 16
+    ffn_ratio: 4.0
+    qkv_bias: True
+    drop_path_rate: 0.0
+    layerscale_init: 1.0
+    norm_layer: layernormbf16
+    n_storage_tokens: 8
+    mask_k_bias: False
+    qk_norm: True
+    rgb_patch_embed: vit_large
+    intrinsics_patch_embed: convnext_base
+    intermediate_layer_idx: [4, 11, 17, 23]
+    gradient_checkpointing: False
+
+  head:
+    __object__:
+      path: src.models.head
+      name: DPTHead
+      args: as_params
+    
+    dim_in: 1024
+    patch_size: 16
+    output_dim: 6
+    activation: sigmoid
+    conf_activation: null
+    features: 256
+    out_channels: [256, 512, 1024, 1024]
+    pos_embed: True
+    head_name: rgb
+    output_mode: modulation
+    modulation_clamp: True
+
+  rgbx:
+    __object__:
+      path: src.rgbx.inference
+      name: Marigold
+      args: as_params
+
+    checkpoint_path: prs-eth/marigold-iid-lighting-v1-1
+    denoise_steps: 1
+    processing_res: 0
+    half_precision: True
+    output_processing_res: True
+    resample_method: bilinear

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:69c2bd11c2f272754f7080bc33e4b049fd334ea5d596df0c475f51c93699710e
+size 2563499816