← Darsh Patel

Robot harness for Dream-1 (Physical AI Hacks)

Hackathon · team of five · 7-8 Nov 2026 · placed 5th of 8 AI-assisted

On this page

We built a harness that runs Dream Labs' Dream-1 policy through the Hill Climb relay on two MakerMods Metal arms, at the MakerMods Physical AI Hackathon on 7 and 8 November 2026. We placed 5th of 8 teams. Dream-1 is a vision-language-action model. It takes camera frames and a text prompt and returns joint actions. It has no idea what a match is, so the harness supplies that. The harness picks the prompt for each stage, asks a camera judge whether the stage is done, and retries or stops when it is not.

The relay has three parts in a fixed order. The robot unplugs USB cables from a power strip, stacks foam blocks into a tower taller than 4 inches, then puts golf balls in a bin. As we read the rubric, touching a later part's objects early is a disqualification, and an illegal ball in the bin loses the match. The black 8-balls are illegal, and on our table the white golf ball was too.

Around the harness we built a stress test that plays whole matches against scripted faults, a MuJoCo copy of the table that runs the real Dream-1 model, and two teleoperation recorders. We recorded 170 episodes and fine-tuned Dream-1 on them. The fine-tune did worse than base Dream-1 on the table, and we traced that to the training data.

Paper (PDF) ↗ · Technical report (PDF) ↗ · hill-climb-harness (the harness) ↗ · task_simulation (simulators and cluster scripts) ↗ · follower_code (leader/follower and Quest teleoperation) ↗

A tower of blue, pink and blue foam cubes topped by a yellow stick and a pink stick, standing on a light wood table behind a black MakerMods Metal robot arm and its camera mast. The background is a flat grey fill.
The target tower for the stacking stage on Station C, 7 Nov. Three foam cubes carry a yellow stick and a pink stick. We replaced the background behind the table with flat grey to remove the people sitting there.
Top-down view of a light wood table with a black power strip, six orange USB cables, blue and pink foam cubes, yellow and pink foam sticks, four golf balls, a white plastic bin and two black robot grippers.
The relay table from above. The power strip and its orange cables are at the top, the cubes and sticks on the left, the golf balls on the right and the bin in the corner. Both grippers are at the bottom edge.
A black two-finger robot gripper holding a pink foam stick above a table with a power strip, orange USB cables, a stack of three foam cubes, three golf balls and a yellow foam stick.
A Metal gripper holding the pink stick on 7 Nov. The cables are still in the strip, and three cubes are stacked behind it.
Two black robot arms at the edge of a table that holds a power strip with orange cables, foam cubes and sticks, a yellow golf ball, and a white plastic bin with an orange golf ball inside.
The table late on 7 Nov, with an orange golf ball in the bin.
7 Nov, 20 s, no sound. The arm lowers the pink stick onto a cube tower and the tower falls. The frame is cropped to the table.

How the harness works

A plan is a list of stages. Each stage has a prompt for Dream, a yes or no question for the camera judge, a timeout, a limit on tries and recovery prompts. Between stages the harness changes the prompt on the running Dream session, because a fresh start takes 2.9 s at the median before the arms move.

A false "done" can send the robot to an object it must not touch yet. So a stage that opens the next part of the relay needs two agreeing verdicts. In the v4 plan the unplug, the two stacking stages and the tower check can each retry up to 99 times, until the match clock runs out.

Before a run, the harness checks that the station reports idle or error, that both arms are alive and that the camera frames are fresh. An arm fault, a Dream error, the time budget or Ctrl+C sends /stop. At the end of a run the bot plans send /reset to return the arms to the start pose.

With --live-guard on, a separate thread runs the local detector on both wrist cameras at 10 Hz. During a ball stage, three new frames in a row from one camera that show an illegal ball in the gripper send /stop. The camera judge takes seconds per call, which is too slow for that job.

In the prompt-engine plan the stage prompts are templates. Claude Sonnet 5.5 picks the slot values from the scene, and the lock, recovery and put-down prompts are fixed text. The v4 plan sends fixed prompts, and the hybrid plans let the judge write them. Hand tests on 8 Nov gave us our prompt rules. Naming the pink and blue cubes produced a stack. A 574-character coached cable prompt pulled out no cable in 286 s, and a stacking prompt that named the right arm sent Dream to the balls near its base.

Dream refused 4 of about 1650 station calls on safety grounds. The reasons it gave included a hand holding the plug and unplugging from a live power strip. Our strip was unplugged from the wall, so we added a sentence saying so to the cable prompts.

Dream often closed on the wire or on nothing, and it almost never pulled. The bot plan has a grip assist for that. It reads the gripper opening from the station. If the opening shows no plug, it stops Dream and re-grips with a /move command, then pulls straight up in small steps. In hand tests the plug stayed in the gripper through the pull in 2 of 2 tries, after an earlier version managed 2 of 4.

Models and classifiers

The table lists each decision in the system and what makes it. Language models answer the slow questions about the scene. Local classifiers and rules answer the ones that need an answer in milliseconds.

JobModel or methodNotes
Move the armsDream-1The bot plans run base Dream-1 (dreamlabs/dream-1-beta). The v4 plans on main still default to our fine-tune
Judge whether a stage is doneClaude Haiku 5.5Sees a frame from the start of the stage, recent overhead frames and the newest wrist frames. Returns JSON with done, progress, status, recovery, holding and guard fields
Frequent looks in the bot plansGemini 3.1 Flash-Lite through OpenRouterThe OpenRouter key hit its limit on 8 Nov and the control panel went back to Haiku
Confirming looks and prompt slotsClaude Sonnet 5.5Second look before a stage that opens the next part of the relay
Illegal ball in a gripperHSV colour thresholds plus shape rulesNo learning. Milliseconds per frame, so it can run at 10 Hz in the live guard
USB-C or USB-A cable endMobileNetV3-small on 96 by 96 cropsTrained on 891 hand-labelled crops. Draws boxes for the judge
Plug in the gripperGripper-width thresholdsSet by hand from 8 Nov readings. Drives the grip assist
Start poseJoint distance to recorded start poses, 0.35 rad toleranceBefore the first unplug, a far start homes the arms and restarts once
Joint faultsHand-written rules and a small temporal convolutional networkOn simulated faults the network caught 91% in time against 50% for the rules, with 0.2 false alarms per joint-hour against 197. Simulation only, cloud-sims branch

Claude Opus also drove the station directly. Between 11:10 and 13:04 on 8 Nov a Claude Opus session sent prompts to Dream and scripted pulls by hand, in 15 to 30 s steps, outside the harness. Its half-open grip prompt followed by a scripted 10 cm pull was the best unplug method we found that day.

We tried adding Blender renders to the cable-end training set. Real crops alone scored higher on held-out station video, 94% against 92%, and on held-out photos, 73% against 56%, so the shipped model leaves the renders out. The shipped model trains on every labelled crop, so that exact fit has no held-out score. The cable-end network and the start-pose check are on the final-harness and cloud-sims branches. The detector is on main.

Judge benchmark

On 8 Nov we ran 25 judge settings on the same questions, with real model calls. Each setting answered 60 calls, which are six fixed cases asked ten times each. Four cases are questions about one station photo. The other two are MuJoCo renders, an empty gripper and a gripper holding an 8-ball. The held 8-ball column therefore counts ten looks at one simulated image. Every setting in the table answered all 20 no-ball looks correctly. The table shows nine of the 25 settings.

JudgeRouteReasoningStage-done correct (of 60)Held 8-ball caught (of 10)Median latency
GPT-6.1 TerraOpenRouternone5992.8 s
GPT-6.1 TerraOpenRouterlow60103.7 s
Gemma 4 26BOpenRouternone5872.4 s
Gemma 4 26BOpenRouterlow601012.0 s (90th percentile 30.8 s)
Claude Haiku 5.5OpenRouternone6032.7 s
Claude Haiku 5.5OpenRouterlow6084.5 s
Claude Haiku 5.5Subscription CLIlow6078.9 s
Claude Sonnet 5.5Subscription CLIlow60107.1 s
Gemini 3.1 Flash-LiteOpenRouternone5451.8 s

GPT-6.1 Terra with low reasoning was the best trade in this test. It got every answer right at a 3.7 s median. Claude Sonnet 5.5 also got every answer right but took 7.1 s, and the CLI times include launching the process. Haiku without reasoning missed the held ball 7 times in 10. The live guard does not depend on any of these models. It uses the local detector.

Stress test

The stress test (experiments/stress_match.py) plays whole matches on a virtual clock, about 5 s for a 5-minute match. The real runner and the real monitor code run as written. The Dream station, the table, the policy and the judge call are scripted, and a referee scores rubric v4. Some faults copy the station log, such as Dream refusals, an empty action buffer and SVD failures. We injected the others, including gateway errors, frozen or corrupt camera frames, API outages and false "done" verdicts. It makes no Dream or Claude calls, so it tests the harness logic and says nothing about whether the robot can do the task.

The table shows the stress test before and after we fixed how the harness handles faults. One fix, for example, retries after a Dream refusal where the old code ended the match. Each cell lists the points for four seeds.

Scenario (scripted)Before the fixesAfter
Happy path87 74 90 11683 94 95 95
Dream refuses the first start0 0 0 0, match ended83 94 95 95
Dream refuses twice0 0 0 0, match ended78 74 95 90
Action buffer empties mid-run10 10 15 15, match ended79 94 95 100
SVD did not converge10 0 10 10, match ended83 84 95 95
Policy keeps holding the unplugged plugplugged it back in 4 of 4never
12 s of corrupt frames20 15 20 2083 94 95 95
Arm diesmatch ended, then sent /start15 10 15 15, no /start
Unplug brushes a blockdisqualified 4 of 4disqualified 4 of 4

The last row is a policy or table-layout problem. The harness only sees the contact after it happens.

In a separate six-seed test with camera frames 2 s old, the harness lost 4 of 6 matches to an illegal ball without the local detector and 2 of 6 with a simulated one.

Full stress sweep

The full sweep runs 37 scenarios with 5 seeds each, which gives 185 matches per configuration. The judge call times are inputs we chose, 6.5 to 13 s for a slow judge and 4 to 6 s for a fast one. The sim also assumes each unplug try succeeds with probability 0.8, far above what we saw on the table. So these points compare harness configurations and do not predict a real match.

ConfigurationJudge call time (input)Mean pointsLost to disqualificationLost to illegal ballMean balls binned
main, v4 plan6.5 to 13 s60.5400.00
hybrid plan6.5 to 13 s61.2410.09
hybrid plan, fast judge4 to 6 s70.54283.01
final, v4 plan, live guard4 to 6 s66.6602.37
final, hybrid plan, live guard4 to 6 s69.8402.90

A faster judge gets the robot to the golf balls, and the illegal ball is among them. The fast hybrid plan lost 28 of 185 matches to it and averaged 70.5 points. The final build with the live guard lost none and averaged 69.8. The two runs also differ in harness build, and the guard's detector in this test reads the simulated held object.

The 4 disqualifications every configuration shares come from the unplug brushing a block. The final v4 plan had 2 more, from false "done" verdicts in two scenarios. A minimum gap between confirming verdicts exists as an option on the cloud-sims branch. No plan turns it on, and we have not stress-tested it.

Dot plot of final points for simulated matches in seven harness configurations, with mean scores between 50 and 70.5 points and red crosses for lost matches.
Final points of every simulated match, one dot each, with the mean as a black bar. Red crosses are matches we would lose to a disqualification or an illegal ball in the bin. The bottom two rows use only 8 hard scenarios, so they are not comparable with the 185-match rows.

MuJoCo with the real Dream-1 model

sim_station.py serves the station's HTTP API on Dream's YAM simulator with our relay table, so the harness runs against it unchanged. The real Dream-1 model drives the simulated arms over the API. A referee reads object positions to score the rubric, and an oracle that reads the same positions stands in for the camera judge. The simulated robot is the YAM model, and we estimated the table layout from photos.

We kept 25 matches from the cluster, one of them a smoke test, and left out a batch with a referee bug.

TaskResult
UnplugA cable came out in 14 of 25 matches. A successful unplug stage took a median of 57 s, range 28 to 269 s
Put the cable downOur notes count 0 of 54 tries. The policy keeps holding the plug
StackNine matches reached a stacking stage. All 23 stacking tries in them failed
Golf ballsNever reached

Three matches were disqualified, two for touching a block before a cable was out and one for touching a ball before the tower passed. A larger cloud run stopped at 12:05 UTC on 8 Nov when our Dream account ran out of credits. Only 10 of its matches were valid, too few for us to compare configurations.

The real table on 8 November

Our station logs do not record whether a run succeeded, so we scored outcomes by hand from the joint logs and the first and last overhead frames. One review covers 101 archived rollouts with 171 prompt steps, from 09:56 to 13:29 on 8 Nov. A second replays the last 23 runs, from 13:47 to 14:28. All runs used base Dream-1 unless marked. The counts are small, and we moved objects between runs.

Prompt versus execution review (docs/prompt-comparison-2026-11-08 branch) ↗ · Replay review of the last 40 minutes (same branch) ↗

TaskWhat we saw
UnplugThe half-open grip prompt ended on the plug body in 15 of 26 steps. Of 12 scripted 10 cm pulls, 8 still held the plug after the lift. A short prompt pulled cable 1 out once by Dream alone, then none in the next 4 repeats. In the last 23 runs, 1 cable came out
Golf ballsWith the right arm and a ball near it, 2 of 5 runs ended with the ball in the bin (orange and yellow). The green ball in the middle of the table failed 3 times. In the last 23 runs, 4 balls went into the bin, all with the right arm
StackA pink cube on a blue cube worked in several runs, and a 3-high tower stood for about 30 s. The next prompt took both stacks apart
Fine-tune0 new cable grips in about 10 minutes of unplug prompts

Dream rarely pulled. It lifted a plug 4 times all day, and the other cables came out through the scripted pull. The second review's main finding is about sequencing. Nothing stopped Dream once a stack was finished, so the next prompt took it apart.

Data collection

We recorded 170 teleoperated episodes on 7 Nov, 48,413 frames and 26.9 minutes in all, in the kit's LeRobot v2.1 format that Dream's fine-tuning accepts. 97 came from a leader and follower pair of Metal arms and 73 from Quest controllers. By task, 69 are unplugs, 55 are put-downs and 19 are plug-ins. The other 27 are ball and block moves.

We wrote both recorders on top of the hackathon kit's data-collection tool. The leader/follower recorder (metal/leader_collect.py) uses one Metal arm as a gravity-compensated leader and the other as the follower. The Quest recorder (quest/quest_collect.py) adds heading calibration, a One Euro filter on each controller pose and a wrist lock.

Our first episodes showed the follower sitting 4 to 6 degrees below its command on the shoulder lift and 2.5 to 4.3 degrees on the elbow, because the follower sent no gravity torque. We added MakerMods' own Metal gravity model to the follower's commands, and the mean shoulder lift error came to +0.2 degrees on the station's arms.

In our Quest recordings, a small hand turn near joint 5's limit could swing joints 4 and 6 by 130 to 190 degrees while the tool turned only 8 to 40. On the quest-continuity branch we added a wrist-flip guard. In an offline replay of 40 fast wrist events from the episodes we designed it on, 7 swung a wrist joint more than 90 degrees with the kit's solver and 2 did with the guard. A second guard treats a controller pose that jumps 8 cm or 45 degrees in one reading as a tracking glitch and holds the arm until the clutch re-engages. We tested both guards offline. The team was recording on 8 Nov, so we put the station dry run on hold.

follower_code, quest-continuity branch ↗

Why the fine-tune failed

We fine-tuned Dream-1 on the 170 episodes. The job ran 1000 steps at learning rate 1e-5 and batch size 32, and ended at loss 0.0034. On the table it did worse than base Dream-1. The right wrist rolled about 90 degrees on the approach, and the gripper closed 11 to 16 cm above the table, where the demos grasp at 4 to 6 cm. Base Dream-1 reached 1 to 6 cm on most attempts.

Our diagnosis blames the training data. The Quest episodes were recorded without gravity feedforward, so the arm sagged below its command. On slow frames, the Quest commands put the tool 3.7 to 5.4 cm above its measured height, depending on the recording set. The leader episodes agree within 0.7 cm. Every live run has gravity feedforward, so the two sets gave the policy two answers for the same state. We suspect this caused the high grasps. We did not run the policy on held-out frames to confirm it, and a corrected fine-tune could not be ready in time.

Fine-tune diagnosis (ft-diagnosis branch) ↗

Dream's latency

The station's inference log has 1650 Dream calls over 52 runs. The round trip had a median of 1116 ms, a 90th percentile of 2.7 s and a worst case of 8.7 s. Logged separately, the medians were 293 ms for the model, 40 ms for the network and 993 ms for Dream's gateway.

Dream's client switches to synchronous pacing when the median round trip passes 900 ms. It then waits for each 2.4 s chunk of actions to finish before it asks for the next. 1101 of the 1650 chunks ran that way, and they arrived 3.5 s apart at the median. By that arithmetic the arms had about 2.4 s of motion in every 3.5 s while in that mode. We did not check it by watching the arms.

Each request uploads a median of 1.6 MB of camera images. Our harness was also fetching 2 MB debug frames from the station. We saw round trips of 2 to 6 s fall to about 0.8 s after we switched to 640-pixel frames of about 35 KB. That is a note from the day, and we have no controlled before and after.

Decision model with the pacing analysis (final-harness branch) ↗ · Final harness notes (final-harness branch) ↗

What we did not measure

We have no success rate for any task on the real table. The hand reviews above are small counts. We also never ran plain Dream against the harness on the station, so we cannot say how much the harness adds. The repos do not record which plan or judge ran in the final match.

Base Dream-1 never seated a plug, so the match plan dropped the plug-in stage. The local detector misses a black ball that touches the black fingers. We found that in a synthetic test, and we had no photos of the 8-ball on the table.

At the last full test run on the cloud-sims branch, 407 tests passed, 4 failed and 5 were skipped.

Team

The team was Daniel Carrera, David Carrera, Kaushik Kanna, Muhammad Mukthar and Darsh Patel.

We used Claude and Codex while writing the code.