how to fix RuntimeError: shape mismatch
hello, I am trying to inference with the model and got the error:
RuntimeError: shape mismatch: value tensor of shape [50, 2048] cannot be broadcast to indexing result of shape [8, 2048]
I cannot fix it, though I've tried all shapes of observation.state, can you show me your inference code? thanks a lot!
You can check this file. I've run the evaluation on Libero benchmark: https://github.com/EO-Robotics/EO1/blob/main/experiments/2_libero/eval_libero.py
...
img = np.ascontiguousarray(obs["agentview_image"][::-1, ::-1])
wrist_img = np.ascontiguousarray(obs["robot0_eye_in_hand_image"][::-1, ::-1])
# Save preprocessed image for replay video
replay_images.append(img)
state = np.concatenate(
(
obs["robot0_eef_pos"],
_quat2axisangle(obs["robot0_eef_quat"]),
obs["robot0_gripper_qpos"],
)
)
if not action_plan:
img = Image.fromarray(img)
wrist_img = Image.fromarray(wrist_img)
batch = {
"observation.images.image": [img],
"observation.images.wrist_image": [wrist_img],
"observation.state": [state],
"task": [str(task_description)],
# "repo_id": [f"{args.task_suite_name}_no_noops_1.0.0_lerobot"],
}
ov_out = processor.select_action(
model,
batch,
)
action_chunk = ov_out.action[0].numpy()
if args.post_process_action:
action_chunk[..., -1] = 2 * (1 - action_chunk[..., -1]) - 1
assert len(action_chunk) >= args.replan_steps, (
f"We want to replan every {args.replan_steps} steps, but policy only predicts {len(action_chunk)} steps."
)
action_plan.extend(action_chunk[: args.replan_steps])
action = action_plan.popleft()
# Execute action in environment
# print("Run to 174")
obs, reward, done, info = env.step(action.tolist())
...
Thanks for answering!
so this model is a LoRA-tuned version of qwen2.5VL, and it is tuned with /embodied-onevision/EO1/experiments/2_libero/train.sh and uses Libero's data? can you share your result on Libero?
thanks a lot!
Here is my results on LIBERO benchmark. I paper they show that EO-1 is trained 50 epochs with 8 GPUs and batch size 64 for each so that golabal batchsize is 512. My results is EO-1 finetuned on 4 H100 GPUs about 1.5-2.5 days (<20 epochs) due to limitation and constraint of hardware.