lsnu/VLAarchTestsBench

VLAarchTests Bench

This repository is a bundled benchmark workspace for future VLAarchTests model iterations, with emphasis on RLBench2 take tray out of oven.

Included

• VLAarchTests benchmark code and generated public benchmark manifest
• Patched AnyBimanual RLBench runtime used to execute the public oven benchmark
• Patched PointFlowMatch + diffusion_policy source snapshot used to execute take shoes out of box
• Official katefgroup/3d_flowmatch_actor PerAct2 checkpoint and public test data
• Official PointFlowMatch 1717447341-indigo-quokka checkpoint for take_shoes_out_of_box
• Public AnyBimanual LF baseline weights and comparison logs
• Verified benchmark reports:
  • oven subset run: 9/10
  • oven full official run: 95/100 = 0.95
  • shoes GPU search: non-zero success verified before later simulator crash
  • hybrid public benchmark smoke outputs
• DexGarmentLab benchmark-related validation scripts and validation logs

Key Result

The strongest public out-of-box checkpoint validated here is:

• models/3dfa_peract2/3dfa_peract2.pth

Official oven result artifacts:

• reports/3dfa_peract2_take_tray_out_of_oven_subset10/eval_after_official_ttm.json
• reports/3dfa_peract2_take_tray_out_of_oven_full100/eval.json

Shoes result artifacts:

• reports/pointflowmatch_take_shoes_out_of_box_ep10_k50_gpu/summary.json
• reports/pointflowmatch_take_shoes_out_of_box_ep10_k50_gpu/run.log

Important Code Paths

• code/VLAarchtests4/code/VLAarchtests2_code/VLAarchtests/code/reveal_vla_bimanual/eval/public_benchmark_package.py
• code/VLAarchtests4/code/VLAarchtests2_code/VLAarchtests/code/reveal_vla_bimanual/eval/run_rlbench_hybrid_smoke.py
• third_party/AnyBimanual/third_party/RLBench/rlbench/bimanual_tasks/bimanual_take_tray_out_of_oven.py
• third_party/AnyBimanual/third_party/RLBench/rlbench/task_ttms/bimanual_take_tray_out_of_oven.ttm
• third_party/PointFlowMatch/pfp/envs/rlbench_env.py
• third_party/PointFlowMatch/pfp/policy/fm_policy.py
• scripts/run_pointflowmatch_take_shoes_out_of_box.sh

External Dependencies Not Mirrored Here

• CoppeliaSim v4.1.0 binary runtime
• Local Python environments under /workspace/envs
• Full IsaacSim installation
• Full DexGarmentLab simulator assets beyond the benchmark-related scripts and logs

See docs/ENVIRONMENT_NOTES.md for the runtime notes used in this workspace.