Robot harness for Dream-1 (Physical AI Hacks)
We built a harness that runs Dream Labs' Dream-1 policy through the Hill Climb relay on two MakerMods Metal arms, at the MakerMods Physical AI Hackathon on 7 and 8 November 2026. We placed 5th of 8 teams. Dream-1 is a vision-language-action model. It takes camera frames and a text prompt and returns joint actions. It has no idea what a match is, so the harness supplies that. The harness picks the prompt for each stage, asks a camera judge whether the stage is done, and retries or stops when it is not.
The relay has three parts in a fixed order. The robot unplugs USB cables from a power strip, stacks foam blocks into a tower taller than 4 inches, then puts golf balls in a bin. As we read the rubric, touching a later part's objects early is a disqualification, and an illegal ball in the bin loses the match. The black 8-balls are illegal, and on our table the white golf ball was too.
Around the harness we built a stress test that plays whole matches against scripted faults, a MuJoCo copy of the table that runs the real Dream-1 model, and two teleoperation recorders. We recorded 170 episodes and fine-tuned Dream-1 on them. The fine-tune did worse than base Dream-1 on the table, and we traced that to the training data.
How the harness works
A plan is a list of stages. Each stage has a prompt for Dream, a yes or no question for the camera judge, a timeout, a limit on tries and recovery prompts. Between stages the harness changes the prompt on the running Dream session, because a fresh start takes 2.9 s at the median before the arms move.
A false "done" can send the robot to an object it must not touch yet. So a stage that opens the next part of the relay needs two agreeing verdicts. In the v4 plan the unplug, the two stacking stages and the tower check can each retry up to 99 times, until the match clock runs out.
Before a run, the harness checks that the station reports idle or error, that both arms are alive and that the camera frames are fresh. An arm fault, a Dream error, the time budget or Ctrl+C sends /stop. At the end of a run the bot plans send /reset to return the arms to the start pose.
With --live-guard on, a separate thread runs the local detector on both wrist cameras at 10 Hz. During a ball stage, three new frames in a row from one camera that show an illegal ball in the gripper send /stop. The camera judge takes seconds per call, which is too slow for that job.
In the prompt-engine plan the stage prompts are templates. Claude Sonnet 5.5 picks the slot values from the scene, and the lock, recovery and put-down prompts are fixed text. The v4 plan sends fixed prompts, and the hybrid plans let the judge write them. Hand tests on 8 Nov gave us our prompt rules. Naming the pink and blue cubes produced a stack. A 574-character coached cable prompt pulled out no cable in 286 s, and a stacking prompt that named the right arm sent Dream to the balls near its base.
Dream refused 4 of about 1650 station calls on safety grounds. The reasons it gave included a hand holding the plug and unplugging from a live power strip. Our strip was unplugged from the wall, so we added a sentence saying so to the cable prompts.
Dream often closed on the wire or on nothing, and it almost never pulled. The bot plan has a grip assist for that. It reads the gripper opening from the station. If the opening shows no plug, it stops Dream and re-grips with a /move command, then pulls straight up in small steps. In hand tests the plug stayed in the gripper through the pull in 2 of 2 tries, after an earlier version managed 2 of 4.
Models and classifiers
The table lists each decision in the system and what makes it. Language models answer the slow questions about the scene. Local classifiers and rules answer the ones that need an answer in milliseconds.
| Job | Model or method | Notes |
|---|---|---|
| Move the arms | Dream-1 | The bot plans run base Dream-1 (dreamlabs/dream-1-beta). The v4 plans on main still default to our fine-tune |
| Judge whether a stage is done | Claude Haiku 5.5 | Sees a frame from the start of the stage, recent overhead frames and the newest wrist frames. Returns JSON with done, progress, status, recovery, holding and guard fields |
| Frequent looks in the bot plans | Gemini 3.1 Flash-Lite through OpenRouter | The OpenRouter key hit its limit on 8 Nov and the control panel went back to Haiku |
| Confirming looks and prompt slots | Claude Sonnet 5.5 | Second look before a stage that opens the next part of the relay |
| Illegal ball in a gripper | HSV colour thresholds plus shape rules | No learning. Milliseconds per frame, so it can run at 10 Hz in the live guard |
| USB-C or USB-A cable end | MobileNetV3-small on 96 by 96 crops | Trained on 891 hand-labelled crops. Draws boxes for the judge |
| Plug in the gripper | Gripper-width thresholds | Set by hand from 8 Nov readings. Drives the grip assist |
| Start pose | Joint distance to recorded start poses, 0.35 rad tolerance | Before the first unplug, a far start homes the arms and restarts once |
| Joint faults | Hand-written rules and a small temporal convolutional network | On simulated faults the network caught 91% in time against 50% for the rules, with 0.2 false alarms per joint-hour against 197. Simulation only, cloud-sims branch |
Claude Opus also drove the station directly. Between 11:10 and 13:04 on 8 Nov a Claude Opus session sent prompts to Dream and scripted pulls by hand, in 15 to 30 s steps, outside the harness. Its half-open grip prompt followed by a scripted 10 cm pull was the best unplug method we found that day.
We tried adding Blender renders to the cable-end training set. Real crops alone scored higher on held-out station video, 94% against 92%, and on held-out photos, 73% against 56%, so the shipped model leaves the renders out. The shipped model trains on every labelled crop, so that exact fit has no held-out score. The cable-end network and the start-pose check are on the final-harness and cloud-sims branches. The detector is on main.
Judge benchmark
On 8 Nov we ran 25 judge settings on the same questions, with real model calls. Each setting answered 60 calls, which are six fixed cases asked ten times each. Four cases are questions about one station photo. The other two are MuJoCo renders, an empty gripper and a gripper holding an 8-ball. The held 8-ball column therefore counts ten looks at one simulated image. Every setting in the table answered all 20 no-ball looks correctly. The table shows nine of the 25 settings.
| Judge | Route | Reasoning | Stage-done correct (of 60) | Held 8-ball caught (of 10) | Median latency |
|---|---|---|---|---|---|
| GPT-6.1 Terra | OpenRouter | none | 59 | 9 | 2.8 s |
| GPT-6.1 Terra | OpenRouter | low | 60 | 10 | 3.7 s |
| Gemma 4 26B | OpenRouter | none | 58 | 7 | 2.4 s |
| Gemma 4 26B | OpenRouter | low | 60 | 10 | 12.0 s (90th percentile 30.8 s) |
| Claude Haiku 5.5 | OpenRouter | none | 60 | 3 | 2.7 s |
| Claude Haiku 5.5 | OpenRouter | low | 60 | 8 | 4.5 s |
| Claude Haiku 5.5 | Subscription CLI | low | 60 | 7 | 8.9 s |
| Claude Sonnet 5.5 | Subscription CLI | low | 60 | 10 | 7.1 s |
| Gemini 3.1 Flash-Lite | OpenRouter | none | 54 | 5 | 1.8 s |
GPT-6.1 Terra with low reasoning was the best trade in this test. It got every answer right at a 3.7 s median. Claude Sonnet 5.5 also got every answer right but took 7.1 s, and the CLI times include launching the process. Haiku without reasoning missed the held ball 7 times in 10. The live guard does not depend on any of these models. It uses the local detector.
Stress test
The stress test (experiments/stress_match.py) plays whole matches on a virtual clock, about 5 s for a 5-minute match. The real runner and the real monitor code run as written. The Dream station, the table, the policy and the judge call are scripted, and a referee scores rubric v4. Some faults copy the station log, such as Dream refusals, an empty action buffer and SVD failures. We injected the others, including gateway errors, frozen or corrupt camera frames, API outages and false "done" verdicts. It makes no Dream or Claude calls, so it tests the harness logic and says nothing about whether the robot can do the task.
The table shows the stress test before and after we fixed how the harness handles faults. One fix, for example, retries after a Dream refusal where the old code ended the match. Each cell lists the points for four seeds.
| Scenario (scripted) | Before the fixes | After |
|---|---|---|
| Happy path | 87 74 90 116 | 83 94 95 95 |
| Dream refuses the first start | 0 0 0 0, match ended | 83 94 95 95 |
| Dream refuses twice | 0 0 0 0, match ended | 78 74 95 90 |
| Action buffer empties mid-run | 10 10 15 15, match ended | 79 94 95 100 |
| SVD did not converge | 10 0 10 10, match ended | 83 84 95 95 |
| Policy keeps holding the unplugged plug | plugged it back in 4 of 4 | never |
| 12 s of corrupt frames | 20 15 20 20 | 83 94 95 95 |
| Arm dies | match ended, then sent /start | 15 10 15 15, no /start |
| Unplug brushes a block | disqualified 4 of 4 | disqualified 4 of 4 |
The last row is a policy or table-layout problem. The harness only sees the contact after it happens.
In a separate six-seed test with camera frames 2 s old, the harness lost 4 of 6 matches to an illegal ball without the local detector and 2 of 6 with a simulated one.
Full stress sweep
The full sweep runs 37 scenarios with 5 seeds each, which gives 185 matches per configuration. The judge call times are inputs we chose, 6.5 to 13 s for a slow judge and 4 to 6 s for a fast one. The sim also assumes each unplug try succeeds with probability 0.8, far above what we saw on the table. So these points compare harness configurations and do not predict a real match.
| Configuration | Judge call time (input) | Mean points | Lost to disqualification | Lost to illegal ball | Mean balls binned |
|---|---|---|---|---|---|
| main, v4 plan | 6.5 to 13 s | 60.5 | 4 | 0 | 0.00 |
| hybrid plan | 6.5 to 13 s | 61.2 | 4 | 1 | 0.09 |
| hybrid plan, fast judge | 4 to 6 s | 70.5 | 4 | 28 | 3.01 |
| final, v4 plan, live guard | 4 to 6 s | 66.6 | 6 | 0 | 2.37 |
| final, hybrid plan, live guard | 4 to 6 s | 69.8 | 4 | 0 | 2.90 |
A faster judge gets the robot to the golf balls, and the illegal ball is among them. The fast hybrid plan lost 28 of 185 matches to it and averaged 70.5 points. The final build with the live guard lost none and averaged 69.8. The two runs also differ in harness build, and the guard's detector in this test reads the simulated held object.
The 4 disqualifications every configuration shares come from the unplug brushing a block. The final v4 plan had 2 more, from false "done" verdicts in two scenarios. A minimum gap between confirming verdicts exists as an option on the cloud-sims branch. No plan turns it on, and we have not stress-tested it.
MuJoCo with the real Dream-1 model
sim_station.py serves the station's HTTP API on Dream's YAM simulator with our relay table, so the harness runs against it unchanged. The real Dream-1 model drives the simulated arms over the API. A referee reads object positions to score the rubric, and an oracle that reads the same positions stands in for the camera judge. The simulated robot is the YAM model, and we estimated the table layout from photos.
We kept 25 matches from the cluster, one of them a smoke test, and left out a batch with a referee bug.
| Task | Result |
|---|---|
| Unplug | A cable came out in 14 of 25 matches. A successful unplug stage took a median of 57 s, range 28 to 269 s |
| Put the cable down | Our notes count 0 of 54 tries. The policy keeps holding the plug |
| Stack | Nine matches reached a stacking stage. All 23 stacking tries in them failed |
| Golf balls | Never reached |
Three matches were disqualified, two for touching a block before a cable was out and one for touching a ball before the tower passed. A larger cloud run stopped at 12:05 UTC on 8 Nov when our Dream account ran out of credits. Only 10 of its matches were valid, too few for us to compare configurations.
The real table on 8 November
Our station logs do not record whether a run succeeded, so we scored outcomes by hand from the joint logs and the first and last overhead frames. One review covers 101 archived rollouts with 171 prompt steps, from 09:56 to 13:29 on 8 Nov. A second replays the last 23 runs, from 13:47 to 14:28. All runs used base Dream-1 unless marked. The counts are small, and we moved objects between runs.
| Task | What we saw |
|---|---|
| Unplug | The half-open grip prompt ended on the plug body in 15 of 26 steps. Of 12 scripted 10 cm pulls, 8 still held the plug after the lift. A short prompt pulled cable 1 out once by Dream alone, then none in the next 4 repeats. In the last 23 runs, 1 cable came out |
| Golf balls | With the right arm and a ball near it, 2 of 5 runs ended with the ball in the bin (orange and yellow). The green ball in the middle of the table failed 3 times. In the last 23 runs, 4 balls went into the bin, all with the right arm |
| Stack | A pink cube on a blue cube worked in several runs, and a 3-high tower stood for about 30 s. The next prompt took both stacks apart |
| Fine-tune | 0 new cable grips in about 10 minutes of unplug prompts |
Dream rarely pulled. It lifted a plug 4 times all day, and the other cables came out through the scripted pull. The second review's main finding is about sequencing. Nothing stopped Dream once a stack was finished, so the next prompt took it apart.
Data collection
We recorded 170 teleoperated episodes on 7 Nov, 48,413 frames and 26.9 minutes in all, in the kit's LeRobot v2.1 format that Dream's fine-tuning accepts. 97 came from a leader and follower pair of Metal arms and 73 from Quest controllers. By task, 69 are unplugs, 55 are put-downs and 19 are plug-ins. The other 27 are ball and block moves.
We wrote both recorders on top of the hackathon kit's data-collection tool. The leader/follower recorder (metal/leader_collect.py) uses one Metal arm as a gravity-compensated leader and the other as the follower. The Quest recorder (quest/quest_collect.py) adds heading calibration, a One Euro filter on each controller pose and a wrist lock.
Our first episodes showed the follower sitting 4 to 6 degrees below its command on the shoulder lift and 2.5 to 4.3 degrees on the elbow, because the follower sent no gravity torque. We added MakerMods' own Metal gravity model to the follower's commands, and the mean shoulder lift error came to +0.2 degrees on the station's arms.
In our Quest recordings, a small hand turn near joint 5's limit could swing joints 4 and 6 by 130 to 190 degrees while the tool turned only 8 to 40. On the quest-continuity branch we added a wrist-flip guard. In an offline replay of 40 fast wrist events from the episodes we designed it on, 7 swung a wrist joint more than 90 degrees with the kit's solver and 2 did with the guard. A second guard treats a controller pose that jumps 8 cm or 45 degrees in one reading as a tracking glitch and holds the arm until the clutch re-engages. We tested both guards offline. The team was recording on 8 Nov, so we put the station dry run on hold.
Why the fine-tune failed
We fine-tuned Dream-1 on the 170 episodes. The job ran 1000 steps at learning rate 1e-5 and batch size 32, and ended at loss 0.0034. On the table it did worse than base Dream-1. The right wrist rolled about 90 degrees on the approach, and the gripper closed 11 to 16 cm above the table, where the demos grasp at 4 to 6 cm. Base Dream-1 reached 1 to 6 cm on most attempts.
Our diagnosis blames the training data. The Quest episodes were recorded without gravity feedforward, so the arm sagged below its command. On slow frames, the Quest commands put the tool 3.7 to 5.4 cm above its measured height, depending on the recording set. The leader episodes agree within 0.7 cm. Every live run has gravity feedforward, so the two sets gave the policy two answers for the same state. We suspect this caused the high grasps. We did not run the policy on held-out frames to confirm it, and a corrected fine-tune could not be ready in time.
Dream's latency
The station's inference log has 1650 Dream calls over 52 runs. The round trip had a median of 1116 ms, a 90th percentile of 2.7 s and a worst case of 8.7 s. Logged separately, the medians were 293 ms for the model, 40 ms for the network and 993 ms for Dream's gateway.
Dream's client switches to synchronous pacing when the median round trip passes 900 ms. It then waits for each 2.4 s chunk of actions to finish before it asks for the next. 1101 of the 1650 chunks ran that way, and they arrived 3.5 s apart at the median. By that arithmetic the arms had about 2.4 s of motion in every 3.5 s while in that mode. We did not check it by watching the arms.
Each request uploads a median of 1.6 MB of camera images. Our harness was also fetching 2 MB debug frames from the station. We saw round trips of 2 to 6 s fall to about 0.8 s after we switched to 640-pixel frames of about 35 KB. That is a note from the day, and we have no controlled before and after.
What we did not measure
We have no success rate for any task on the real table. The hand reviews above are small counts. We also never ran plain Dream against the harness on the station, so we cannot say how much the harness adds. The repos do not record which plan or judge ran in the final match.
Base Dream-1 never seated a plug, so the match plan dropped the plug-in stage. The local detector misses a black ball that touches the black fingers. We found that in a synthetic test, and we had no photos of the 8-ball on the table.
At the last full test run on the cloud-sims branch, 407 tests passed, 4 failed and 5 were skipped.
Team
The team was Daniel Carrera, David Carrera, Kaushik Kanna, Muhammad Mukthar and Darsh Patel.
We used Claude and Codex while writing the code.