OpenVLA-7B · LIBERO · serverless GPU · real results

Simulate, evaluate, and benchmark robot policies.

One platform to run Vision-Language-Action policies in simulation on serverless GPUs, measure what actually decides deployability success, robustness, control rate, memory and cost, and benchmark models head-to-head. Every run is reproducible from its config hash. There is no universally best model, only the best one for your task, robot and constraints.

Run an evaluation →See the platform

RUN · OpenVLA-7Bsuccess

agentview · 256²

t=000

“open the middle drawer of the cabinet”

unedited · 1× speed · frozen init state

100%

success

331

ms/step

$0.031

cost

libero_spatial · task 0

Optimise for

Max GPU · 16 GB

3 of 3 models eligible

Recommended

OpenVLA-7B

highest success, ties broken by robustness · beat OpenVLA-7B 4-bit

1OpenVLA-7B

pick

success

100%

gpu

16 GB

$/ep

$0.021

latency

335 ms

robustness

−33 pts

2OpenVLA-7B 4-bit

success

100%

gpu

8 GB

$/ep

$0.024

latency

445 ms

robustness

−67 pts

3OpenVLA-7B 8-bit

success

100%

gpu

10 GB

$/ep

$0.050

latency

1197 ms

robustness

not run

Ranking is a plain sort on the axis you picked, never a hidden weighted score. Every figure is measured, and “not run” stays blank rather than being estimated.

LIBERORoboCasaIsaac LabMuJoCorobosuiteOpenVLAFranka PandaPandaOmronModalA10GL40SBDDLLeRobotLIBERORoboCasaIsaac LabMuJoCorobosuiteOpenVLAFranka PandaPandaOmronModalA10GL40SBDDLLeRobot

What the harness is for

A success rate without its conditions is not a result.

Published VLA numbers are hard to act on because the things that determine whether a policy is deployable (control rate, memory, degradation under noise) are usually not reported alongside them, and rarely re-derivable.

Control rate is a deployment constraint

OpenVLA-7B measures 335 ms/step on an A10, roughly 3 Hz closed-loop. Workable on quasi-static LIBERO scenes, far below what reactive manipulation needs. That number belongs beside the success rate, not in an appendix.

Quantization changes more than memory

4-bit held 100% clean and fell further under σ=0.1 action noise than full precision. n=3, so this is a hypothesis worth more episodes rather than a finding, and the harness exists to make that follow-up cheap.

A failed run should name its suspects

π0 scores 0% here. The cause is a checkpoint/library mismatch (state_dict missing language_model.embed_tokens.weight), not a statement about the policy. Separating those two is most of the work.

How it works

Provision → Run → Verify → Decide → Improve

Five stages, one system. Most tooling stops after “Run”.

Provision

Define an environment (policy, cameras, task) and attach a GPU instance. Cold start to first rollout in minutes.

Run

Execute across simulators and embodiments on serverless GPUs. Frozen initial states, so every model sees an identical scene.

Verify

Score every episode with evidence: goal predicates in sim, plus rollout video and partial progress for anything a human must trust.

Decide

Success, cost, latency, GPU and robustness on one frontier, with a recommendation for your constraints rather than a global #1.

Improve

Ranked failure hypotheses name the likely cause: gripper polarity, image orientation, embodiment mismatch, plus the slice to retrain on.

One command

One command, one config hash, one reproducible result.

Pre-flight checks action-space and embodiment compatibility before a GPU starts. Episodes are scored against LIBERO goal predicates and stored with their video. The perturbation and its magnitude are part of the config hash, so a perturbed run is a distinct, re-derivable result rather than an overwrite.

Action chunk · prediction horizon

VLAs emit an action chunk per forward pass. ROBIT’s adapter interface is chunk-native, so a 1-step model and a 50-step model run through the identical loop.

robit run

Robustness

Identical clean scores, different failure behaviour.

Each model runs clean, then under a seeded perturbation applied at the policy’s input/output boundary. Both quantizations hold 100% on a clean bench; they separate under σ=0.1 action noise. n=3 per condition: directional, not conclusive.

−0

full precision

−0

4-bit quantized

Cost per successful task diverges once conditions degrade: $0.043 vs $0.141 here. With n=3 that ratio rests on a single episode, so treat it as a reason to run more, not as a measured 3×.

CLEAN · nominal conditions100% success

OpenVLA-7Blibero_spatial · task 0

RoboCasa · PandaOmron · kitchen

harness check · null policy · no success claimed

One harness, many simulators

Simulators sit behind one protocol.

SimEnv and ModelAdapter are narrow Protocols, so the eval loop is simulator-agnostic and each backend gets its own container image and GPU tier. LIBERO is verified with a real policy; RoboCasa currently proves only that the environment loads and steps.

LIBERO

MuJoCo · Franka Panda · 2 of 40 tasks run

policy verified

RoboCasa

robosuite · PandaOmron · null policy only

env loads

Isaac Lab

PhysX · blocked: no /dev/dri on host

not running

Platform

What the harness records per run.

Decide

Trade-off frontier

Pareto-optimal models circled, dominated models greyed out. Nothing beats a frontier model on every axis at once.

Verify

Evidence, not a score

Goal predicates in sim, plus rollout video and partial progress kept per episode, so a number a human must trust always has something behind it.

episode_000

success

episode_001

success

episode_002

success

Decide

Constraint-aware recommendation

Set a priority and hard limits (max GPU, min success, max latency) and get the strongest eligible model, plus the runner-up it beat.

Improve

Failure diagnosis

Turns a bare “0%” into ranked, actionable causes drawn from documented VLA-on-sim failure modes, each with a concrete fix.

Run

Pre-flight validation

Embodiment, adapter and action-space checks before a GPU spins up. A 7-DoF policy cannot silently drive a 12-DoF mobile base.

Provision

Ask in plain English

Describe the constraint and ROBIT configures the comparison, launches it, and lands you on the answer.

cheapest VLA that opens a drawer under 8 GB

Scope and limitations

What has actually been run.

Stated plainly so results are not read as broader than they are. This is early infrastructure with a small experimental record, not a benchmark suite.

Established

OpenVLA-7B runs end to end on LIBERO under a shared harness: 33 episodes across libero_spatial/0 and libero_goal/0.
Control rate of 335 ms/step (~3 Hz) and 16 GB resident on an A10, measured from GPU wall clock.
Frozen LIBERO init states, so a rerun of the same config hash replays the same scene.
4-bit and 8-bit quantization measured on the same task: 445 ms and 1197 ms per step respectively.
Perturbations are seeded and their magnitude is part of the config hash.

Not established

No hardware. Every result here is simulation; nothing has run on a physical robot.
n=3 per condition, single seed set, two tasks. Not enough to separate models; a binomial CI on 2/3 spans roughly 9-99%.
RoboCasa has only executed a null policy. Isaac Lab has never completed an episode (no /dev/dri render nodes on the host).
pi0 scores 0%, traced to a checkpoint/library mismatch rather than policy quality. Unresolved.
Success rates here (100%) come from n=3 on one task and should not be compared with published LIBERO numbers.

Run it against your own checkpoint.

Point the harness at a policy and a LIBERO task. You get per-episode video, control rate, memory and cost from GPU wall clock, and a config hash that reproduces the run.

Run an evaluation →Explore the platform