docs: complete inference snippet + embed rubric prompt templates

README.md

@@ -9,26 +9,225 @@ tags:
   - anonymous-release
 ---
 
-# Anonymous Release — Judge LoRA Adapter
+# physground-judger9B — Anonymous Judge LoRA Adapter
 
-A LoRA adapter trained as a judge model that scores generated videos against
-physical-law sub-rubrics derived from text prompts. Released anonymously
-alongside the companion dataset
+LoRA adapter trained as a judge model that scores generated videos against
+prompt-alignment, temporal, persistence, and 13 physical-law sub-rubrics.
+Released anonymously alongside the companion dataset
 [`anonymouscla/physground`](https://huggingface.co/datasets/anonymouscla/physground).
 
-The base model identifier required to load this adapter is recorded in
-`adapter_config.json` (`base_model_name_or_path`).
+The base model identifier required to attach this adapter is recorded in
+`adapter_config.json` (`base_model_name_or_path`); the inference script
+reads it automatically.
 
 ## Files
 
+| File | Purpose |
+| --- | --- |
+| `adapter_config.json` | PEFT/LoRA config (records base model id) |
+| `adapter_model.safetensors` | LoRA weights (~167 MB) |
+| `additional_config.json` | ms-swift extras (lora_dtype / lr ratios) |
+| `training_args.json` | sanitized training hyperparameters |
+| `subq+human.yaml` | prompt template used at training and inference time |
+| `infer.py` | standalone end-to-end inference script |
+
+## Setup
+
+```bash
+pip install "transformers>=4.49" peft accelerate pyyaml \
+            "qwen-vl-utils[decord]" huggingface_hub
+```
+
+Loading the base model in bf16 needs roughly 24 GB of GPU memory.
+
+## Quickstart — Hugging Face Hub
+
+`infer.py` accepts either a local folder or a HF Hub repo id via
+`--adapter-dir`; the default value already points at this repo, so the
+following commands work without cloning anything.
+
+```bash
+# General axes (1–5 each): SA / PTV / persistence
+python infer.py \
+  --video /path/to/video.mp4 \
+  --caption "A ball rolls down a ramp and knocks over a block." \
+  --metric SA
+
+# Physical-law axes (1–5 each): one of the 13 laws below
+python infer.py \
+  --video /path/to/video.mp4 \
+  --caption "A ball rolls down a ramp and knocks over a block." \
+  --law gravity
 ```
-adapter_config.json            # PEFT/LoRA config
-adapter_model.safetensors      # LoRA weights (~167 MB)
-additional_config.json         # ms-swift extras (lora_dtype / lr ratios)
-training_args.json             # sanitized training hyperparameters
+
+`infer.py` will:
+
+1. Resolve `--adapter-dir` to a local directory (`huggingface_hub.snapshot_download`
+   if it is a Hub id).
+2. Read `adapter_config.json` to find the base model and load it via
+   `transformers` (`Qwen/Qwen3.5-9B`).
+3. Attach the LoRA adapter via PEFT.
+4. Render the scoring prompt from `subq+human.yaml`, plus the relevant
+   sub-questions / per-law criterion (constants embedded in `infer.py`).
+5. Run greedy decoding with `--max-new-tokens 64` (matches training).
+6. Parse the JSON object and print the integer score.
+
+Output is a single JSON line:
+
+```json
+{"key": "gravity", "score": 4, "raw": "{\"gravity\": 4}"}
 ```
 
-## Training
+`--metric` choices: `SA`, `PTV`, `persistence`.
+`--law` choices: `gravity`, `inertia`, `momentum`, `impenetrability`,
+`collision`, `material`, `buoyancy`, `displacement`, `flow_dynamics`,
+`boundary_interaction`, `fluid_continuity`, `reflection`, `shadow`.
+
+Add `--print-prompt` to inspect the exact rendered system + user prompt
+before generation.
+
+## Programmatic use
+
+```python
+from pathlib import Path
+import torch
+
+from infer import (
+    build_messages,
+    build_prompt,
+    decode_generated,
+    load_model,
+    load_yaml,
+    parse_score,
+    prepare_inputs,
+)
+
+processor, model, adapter_dir = load_model(
+    "anonymouscla/physground-judger9B",
+    dtype=torch.bfloat16,
+    device_map="auto",
+)
+cfg = load_yaml(adapter_dir / "subq+human.yaml")
+
+system, user, key = build_prompt(
+    cfg,
+    caption="A ball rolls down a ramp and knocks over a block.",
+    law="gravity",
+)
+messages = build_messages(system, user, Path("video.mp4"))
+inputs = prepare_inputs(
+    processor,
+    messages,
+    next(model.parameters()).device,
+    fps=2.0,
+    max_pixels=360 * 640,
+)
+
+with torch.inference_mode():
+    out = model.generate(**inputs, max_new_tokens=64, do_sample=False)
+
+raw = decode_generated(processor, inputs, out)
+print({"key": key, "score": parse_score(raw, key), "raw": raw})
+```
+
+## Prompt templates
+
+System prompt (all axes): `You are a strict video evaluation model.`
+
+The prompt always asks the judge to consider observable sub-questions in
+its mind and then output **only** a JSON object with one 1–5 score.
+
+### General axes — `subq+human.yaml`
+
+**SA — Prompt alignment**
+```
+Evaluate Prompt Alignment (SA).
+
+Caption:
+"{prompt}"
+
+The video was generated using a text+image-to-video (ti2v) model,
+conditioned on the first frame and the text prompt above.
+
+Sub-questions to consider in your mind before scoring:
+1. Are the main objects in the caption present in the video?
+2. Are the key actions or interactions from the caption visible?
+3. Are important scene attributes and relationships preserved?
+4. Does the video avoid major contradictions to the caption?
+
+Score 1-5: 5=fully aligned, 4=mostly aligned with minor deviations,
+           3=partially aligned with notable gaps, 2=mostly misaligned,
+           1=not aligned
+
+Then output ONLY a JSON object with exactly one key: SA.
+Example: {"SA": 3}
+```
+
+**PTV — Temporal coherence** uses the same shape with sub-questions:
+
+1. Do causes appear before their effects?
+2. Do physical events unfold in a plausible temporal order?
+3. Are motion transitions continuous rather than abrupt jumps or loops?
+4. Does the sequence avoid impossible reversals or repeated resets?
+
+**persistence — Object persistence** uses:
+
+1. Do objects maintain consistent existence throughout the video?
+2. Do objects keep a stable shape, size, color, and texture?
+3. Do objects avoid disappearing, appearing, or transforming unexpectedly?
+4. Do objects preserve identity through motion and brief occlusion?
+
+(See `subq+human.yaml` for the verbatim PTV / persistence rubric anchors.)
+
+### Physical-law axes — `physical_template`
+
+```
+Evaluate physical realism for one physical law: {law}.
+
+Criterion:
+{criteria}
+
+Caption, for context only:
+"{prompt}"
+
+Sub-questions to consider in your mind before scoring:
+{questions_block}
+
+Judge the video itself. Do not penalize prompt mismatch unless it affects
+whether this physical law can be evaluated.
+
+Score 1-5: 5=clearly correct, 4=mostly correct with minor issues,
+           3=partially correct or ambiguous, 2=mostly incorrect,
+           1=severely incorrect
+
+Then output ONLY a JSON object with exactly one key: {law}.
+Example: {"{law}": 3}
+```
+
+`{criteria}` and `{questions_block}` for each of the 13 laws are listed
+below. They are also embedded in `infer.py` (`PHYSICAL_CRITERIA`,
+`PHYSICAL_SUB_QUESTIONS`) so the script is a self-contained reference.
+
+| Law | Criterion | Sub-questions |
+| --- | --- | --- |
+| `gravity` | Do unsupported objects fall downward? Do thrown objects follow a curved trajectory? Does poured liquid fall with gravity? | (1) Do unsupported objects or liquids move downward over time? (2) Do thrown or falling objects follow a plausible gravity-driven path? (3) Does the video avoid objects floating or rising without support? |
+| `inertia` | Do stationary objects remain still unless acted upon? Do moving objects maintain their motion unless stopped by friction, collision, or an obstacle? | (1) Do stationary objects remain still unless a visible force acts on them? (2) Do moving objects continue plausibly until friction, collision, or an obstacle changes their motion? (3) Does the video avoid unexplained starts, stops, or direction changes? |
+| `momentum` | After collision, push, or pull, is the direction of motion reasonable? Ignore speed magnitude. | (1) After contact, push, pull, or collision, are motion directions plausible? (2) Does the reacting object move in a direction consistent with the interaction? (3) Does the video avoid impossible reversals or unrelated motion changes? |
+| `impenetrability` | Do objects maintain impenetrability — no passing through each other? | (1) Do solid objects avoid passing through one another? (2) Do contacts and overlaps remain physically plausible? (3) Does the video avoid obvious clipping or penetration artifacts? |
+| `collision` | After impact, is there reasonable bounce/shatter/deformation? Does response match impact force? | (1) Does impact cause a plausible bounce, break, deformation, or transfer of motion? (2) Is the response direction consistent with the collision? (3) Does the response avoid being much too weak, too strong, or unrelated to the impact? |
+| `material` | Does each material respond according to its properties? (glass shatters, rubber bounces, metal is rigid, cloth deforms softly, etc.) | (1) Do objects respond consistently with their apparent material? (2) Are rigid, soft, brittle, elastic, or fluid-like objects animated appropriately? (3) Does the video avoid material behavior that contradicts the scene? |
+| `buoyancy` | Do dense objects sink? Do wood/plastic float? | (1) Do objects sink or float in a way consistent with apparent density? (2) Does the floating or sinking behavior stay stable over time? (3) Does the video avoid unsupported hovering or impossible underwater motion? |
+| `displacement` | When you add more liquid or put an object into it, does the liquid level rise in a realistic way? Does it overflow when full? | (1) Does liquid level rise when volume is added or an object enters it? (2) Does overflow happen only when the container is plausibly full? (3) Does the liquid volume remain visually plausible? |
+| `flow_dynamics` | Does the liquid's overall motion behave realistically over time — flowing along surfaces, spreading, draining naturally? | (1) Does liquid flow along surfaces, spread, or drain naturally? (2) Does the flow direction follow gravity and boundaries? (3) Does the video avoid abrupt stops, reversals, or unsupported uphill flow? |
+| `boundary_interaction` | When the liquid hits a boundary such as a rock face, container wall, or floor, does it respond realistically? Do local splash, rebound, or split patterns on impact look physically plausible? | (1) Does liquid react plausibly when hitting a wall, floor, container, or obstacle? (2) Are splash, rebound, or split patterns locally plausible? (3) Does the liquid remain consistent after interacting with boundaries? |
+| `fluid_continuity` | Does the liquid avoid disappearing or appearing out of nowhere? Small splashes that briefly break apart are okay. | (1) Does liquid avoid disappearing or appearing without cause? (2) Does the amount of liquid remain broadly consistent? (3) Are splashes and separations temporary and physically plausible? |
+| `reflection` | Does the reflection roughly match objects and colors in the scene, and avoid completely unrelated content? | (1) Does the reflection match nearby objects, colors, and motion? (2) Does the reflected content stay spatially consistent with the scene? (3) Does the video avoid unrelated or impossible reflection content? |
+| `shadow` | Are shadow directions consistent with light source? Do shadows move with objects? | (1) Are shadows consistent with the apparent light source direction? (2) Do shadows move with the objects that cast them? (3) Does the video avoid missing, detached, or contradictory shadows? |
+
+Pass `--criteria "..."` to override a per-law criterion at inference time
+without editing the YAML or script.
+
+## Training summary
 
 | Item | Value |
 | --- | --- |
@@ -41,32 +240,13 @@ training_args.json             # sanitized training hyperparameters
 | Best eval loss | 0.1063 (step 294) |
 | Framework | ms-swift 4.1.2, PEFT 0.19.1, DeepSpeed ZeRO-2 |
 
-The training mixture combines automatically derived sub-question judgements
-with human-rated samples (the `subq+human` split). See the companion
-anonymous dataset for prompts, physical-law tags, and example videos.
-
-## Usage
-
-```python
-import json
-from peft import PeftModel
-from transformers import AutoModelForCausalLM, AutoTokenizer
-
-adapter_dir = "."  # this directory
-base_id = json.load(open(f"{adapter_dir}/adapter_config.json"))["base_model_name_or_path"]
-
-tokenizer = AutoTokenizer.from_pretrained(base_id)
-base = AutoModelForCausalLM.from_pretrained(base_id, torch_dtype="bfloat16", device_map="auto")
-model = PeftModel.from_pretrained(base, adapter_dir)
-model.eval()
-```
-
-The adapter expects the base model's default chat template, with a prompt
-that asks the judge to answer one or more sub-rubric questions about a
-candidate video frame/caption. Greedy decoding (`temperature = 0`) with
-`max_new_tokens = 64` matches the training-time generation config.
+The training mixture combines automatically derived sub-question
+judgements with human-rated samples (the `subq+human` split). See the
+companion anonymous dataset for prompts, physical-law tags, and example
+videos.
 
 ## License
 
-The base model is released by its respective authors; this LoRA adapter is
-shared for anonymous review purposes. No identifying metadata is included.
+The base model is released by its respective authors; this LoRA adapter
+is shared for anonymous review purposes. No identifying metadata is
+included.