Image Generators are Generalist Vision Learners
Paper • 2604.20329 • Published • 20
Image-to-Image
Diffusers
English
rppg
remote-photoplethysmography
cardiac-signal
lora
flux2
vision-banana
import torch
from diffusers import DiffusionPipeline
from diffusers.utils import load_image
# switch to "mps" for apple devices
pipe = DiffusionPipeline.from_pretrained("black-forest-labs/FLUX.2-klein-base-4B", dtype=torch.bfloat16, device_map="cuda")
pipe.load_lora_weights("phanerozoic/pulse-plantain")
prompt = "Turn this cat into a dog"
input_image = load_image("https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/cat.png")
image = pipe(image=input_image, prompt=prompt).images[0]pulse-plantain
A LoRA adapter on FLUX.2 Klein 4B for remote photoplethysmography. Given a short clip of a face, the model emits a per-pixel pulse-color field decodable to instantaneous heart rate (BPM). Companion to Image Generators are Generalist Vision Learners (Gabeur et al., 2026; arXiv:2604.20329).
Hypothesis
Remote photoplethysmography (rPPG) recovers cardiac signals from sub-perceptual color variation in skin pixels. The literature spans more than fifteen years and is dominated by hand-engineered pipelines: skin-region detection, RGB-to-chrominance projections, bandpass filtering, blind source separation (Verkruysse 2008; Poh et al. 2010), and recent end-to-end CNNs (DeepPhys, MTTS-CAN). All pipelines impose explicit signal-processing inductive biases.
The test here is whether a general image generator carries enough latent representation of skin micro-color dynamics that a parameter-efficient adapter can produce the per-pixel pulse field directly, without an explicit bandpass + ICA pipeline. The output is a 2D color visualization invertible back to a metric BPM trace via a fixed bijection (analogous to deep-plantain's metric-depth encoding).
Method
Input: a short RGB face clip (5–10 seconds at standard frame rate). Output: a per-pixel pulse-color visualization frame in which color encodes the instantaneous BVP signal phase and amplitude over the local skin region. Decoded by inverting the bijection to recover a per-region BVP trace, from which BPM is estimated by peak-counting or FFT.
Training data is drawn from public rPPG datasets with synchronized ground-truth BVP (UBFC-rPPG, PURE, COHFACE, MMPD). Each training example pairs (a) a windowed face clip with (b) the corresponding ground-truth BVP trace rendered as the bijection-encoded RGB target. Evaluation reports BPM mean absolute error against held-out subjects.
Status
Placeholder. Weights and training data forthcoming.
License
Apache 2.0 — matches base FLUX.2 Klein 4B.
References
- Gabeur, V., Long, S., Peng, S., et al. Image Generators are Generalist Vision Learners. arXiv:2604.20329 (2026).
- Verkruysse, W., Svaasand, L. O., Nelson, J. S. Remote plethysmographic imaging using ambient light. Optics Express (2008).
- Poh, M.-Z., McDuff, D. J., Picard, R. W. Non-contact, automated cardiac pulse measurements using video imaging and blind source separation. Optics Express (2010).
- Liu, X., Fromm, J., Patel, S., McDuff, D. MTTS-CAN: Multi-Task Temporal Shift Attention Networks for On-Device Contactless Vitals Measurement. NeurIPS (2020).
- Black Forest Labs. FLUX.2 Klein. https://bfl.ai/models/flux-2-klein (2025).
- Downloads last month
- -
Model tree for phanerozoic/pulse-plantain
Base model
black-forest-labs/FLUX.2-klein-base-4B