A benchmark of spatial-constraint puzzles for robot manipulation

Two claws, hooked through each other.

Pulling only jams them. They come apart when turned in the right order.

Here two robot arms do it in simulation, replaying the motion they planned.

RoboARC

Spatial reasoning, with robot hands.

94 physical puzzles in 14 families: pieces hooked, threaded, knotted, caged or folded, where only the right sequence of moves solves them.

See the puzzles How it is graded Results Paper soon Code soon

Scroll

RoboARC is a benchmark of physical puzzles for robots: interlocked claws, cast-metal brainteasers, a caged hedgehog, a lock, a buzz wire, fibers, knots, threaded rope, chains, and a blanket to flatten or fold.

Each task is a frozen simulation instance that always starts from the same state, on MuJoCo for the rigid, rope and chain puzzles and a deterministic GPU cloth solver for the blankets. An agent gets three hours and drives a simulated robot, two ALOHA arms or one Franka Panda arm, by writing code. An attempt counts only if its recorded motion solves the puzzle again when it is replayed in a fresh simulator.

14: puzzle families
94: frozen tasks, very easy to hard
2: evaluation modes, 188 tasks in all
3 h: per attempt

Reference solutions of the families, one after another, sped up.

Puzzles

Each card plays a reference solution: a trajectory that solves the task, replayed in a fresh simulator and sped up. Step through a family's tasks on its card, from very easy to hard.

In 3D

The same reference solutions, redrawn in 3D from the replays the videos show: every piece, the robot, the rope and the cloth where the simulator had them, frame by frame. Turn the view, zoom in, scrub through the run, or hide the robot to see the pieces.

sim 0:00 loading blocked · the pieces touch

Two modes

Every task is posed twice. In both, the agent's recorded motion is replayed in a fresh simulator, and only a replay that solves the puzzle counts.

privileged

Code with the simulator open

Object poses, bounds and the scene's state can be read at any time.
Resets and replays are unlimited within the three hours.
The agent submits one trajectory, replayed open-loop in a fresh process and graded.

from rcb_topogym import Session
s = Session("cast_keyhole", instance="/app/instance.json")
s.reset()                      # back to the frozen start, any time
s.object_pose("keyhole_silver")   # one object's pose
s.record_start()
s.step(action)                 # joints + grippers, one 1/40 s step
actions = s.record_stop()
s.replay(actions)              # self-check, as the grader will
s.save_trajectory(actions, "/app/output/trajectory.npz")

standard

The robot as a remote service

Camera images (RGB, depth on request) and the robot's own state, nothing else.
One episode, no object poses and no resets: like real hardware.
The service records the episode, which is then replayed and graded.

Overhead camera view of a cast keyhole puzzle — overhead

Results

No results for RoboARC 0.1.0 yet. The first evaluation of coding agents, each model inside its harness, will be posted here, with VLA policies ranked on their own.