Assessment and Failure Recovery in Remote Vision-Language-Action Deployment - From Pipeline Measurement to Proprioceptive Retry

Abstract

Vision-language-action (VLA) models integrate visual perception, language understanding, and motor control within a single network, enabling robots to perform manipulation tasks from natural-language instructions without task-specific programming. Because these models are computationally demanding, they are often deployed on a remote GPU server: the robot client sends observations over a network, receives action chunks asynchronously, and continues executing motion while the next chunk is being inferred. However, this asynchronous deployment introduces failure modes that are not visible when the policy is evaluated in isolation, including stale action chunks, waiting periods at chunk boundaries, and physical states that have diverged from the observations used to generate the action chunks. In this thesis, we study these issues on both real robots and simulation benchmarks under a unified workflow. We collected an SO-101 dataset, fine-tuned π0.5 and SmolVLA on it, ran both models on a remote GPU cluster through the LeRobot asynchronous-inference stack, and executed the returned action chunks on the robotic arm. The same workflow is also used as a measurement tool: it decomposes end-to-end latency across the pipeline and shows how image cropping and Real-Time Chunking choices shift the main bottleneck between the network and the GPU. To address execution failures in this workflow, we propose Failure-Aware Recovery (FAR), a lightweight runtime monitor that wraps an unchanged VLA policy without adding extra sensors. FAR jointly uses gripper intent from the action queue and gripper feedback to detect empty-grasp and slip events. The feedback comes from motor load and position on the real robot, and from finger separation and MuJoCo contact information in simulation. Once a failure is detected, FAR drains stale queued actions, executes a bounded recovery motion, refreshes the server context, and requests a new action chunk from the recovered state. FAR relies only on existing gripper feedback and requires no additional sensors, no policy retraining, and no additional learned module for action evaluation, state prediction, or vision-language reasoning. With FAR, the success rate improves across most LIBERO suites and on the real robot, while these gains are generally accompanied by shorter episodes. Under scripted human interference that forces empty grasps, task success rises from 32% to 100% for π0.5 and from 40% to 82% for SmolVLA, with negligible additional latency from the recovery layer.