Dev/guidelines.md

Overview

This document describes how to integrate a Vision-Language-Action-Critic (VLAC) into the SimpleVLA-RL training stack. The goal is to replace the simulator-provided terminal success signal during training with a VLAC-predicted value, while preserving the simulator signal for evaluation.

Repository structure (high level)

• verl Core RL training framework for SimpleVLA-RL. Uses Ray for distributed orchestration and vLLM for fast inference/parallelization. Originally built for LLM RL, adapted here for VLA training. Ships with a pre-trained OpenVLA-OFT model and the LIBERO simulation environment as Python packages.

• examples Entry scripts and helpers for training and evaluation. run_openvla_oft_rl.sh is the main training entrypoint. eval_openvla_oft.sh runs evaluation (with trainer.val_only=True).

• evo_vlac External module imported from the VLAC project. Provides models to predict task progress/success for robot manipulation. evo_vlac/examples shows how to run the critic on images/videos and obtain pairwise critic, per-frame values, and done/success estimates.

Objective

• Training Do not use the environment done during training. Termination and terminal rewards are determined by VLAC only. After each step, call VLAC to detect done.

  • If VLAC determined done == True: terminate the episode immediately and set the terminal reward to 1.0 (no need to call VLAC value again).
  • Else if max_step is reached: terminate the rollout and call the VLAC model to compute the value list and use the last value the reward.
  • Else: continue the episode.

• Evaluation Keep using the environment done signal to compute success rate. VLAC is not used to determine evaluation success.

• Rollout termination logic

  • Training: "done" signal from VLAC
  • Testing: "done" signal from the environment

• Reward logic

  • Training: if "done" detected by VLAC, reward=1.0; else, reward comes from VLAC value.

Integration approach (service-oriented)

• Why service verl is Ray/vLLM-managed; VLAC is managed via Hugging Face and ms-swift. To keep systems decoupled and switchable, expose VLAC as a lightweight service that verl calls at needed times.

• Deployment flexibility The VLAC service can run on the same node (sharing GPU memory fraction) or on a different node.

• Transport options During training, frames are kept in memory as a Python list (TODO: you need to double-confirm this in the code in implementation); they are not written to disk until the episode finishes. Therefore, prefer sending the frames for communication in the service.

Additional Notes

• Try to use simple implementation. Avoding using complicated environment, e.g., docker.
• GPU resource sharing: Likely to share GPUs with SimpleVLA-RL on H100 80GB cards. During rollout generation, usage is ~20–30 GB; during actor gradient updates, peaks at ~60–70 GB.
• Development environment: Use this desktop only for editing and light, non-intrusive smoke tests. I will push to a server for real training/evaluation.
• Collaboration: Before starting, I will confirm the open questions above with you and then proceed with the plan.