Augment Stage: NVIDIA Cosmos3 Transfer on an NVIDIA Jetson AGX Thor for the Seeed Studio reBot Arm B601-RS

For the GR00T N1.7 fine-tune I borrowed someone else's dataset, prateeks-iu/rebot_b601_rs_sugar: 103 episodes of a robot arm picking up a sugar cube. All of them were recorded in the same room, on the same white table, under the same light.

A model trained on that learns the table as much as it learns the task.

The usual fix is to record more data somewhere else. The other fix is to repaint the data you already have — keep the arm's motion exactly as it was, and change everything behind it. That is what Cosmos3 Transfer does.

I had a Jetson AGX Thor on my desk and wanted to know whether it could run Cosmos3 Transfer on that dataset. Six days later: 309 episodes.

One dataset, three rooms

The source. Grey wall, white tabletop, flat light — prateeks-iu/rebot_b601_rs_sugar, 103 episodes in LeRobot format, the same dataset I fine-tuned GR00T N1.7 on.

Kitchen: "marble countertops, stainless steel appliances and timber cabinetry behind", warm overhead spots and window light.

Lab: "white epoxy workbenches, metal shelving and precision instruments", flat fluorescent panels, minimal shadows.

Workshop: "heavy timber workbenches, hand tools on a pegboard wall", industrial LED task lighting.

Those four strips are sampled at the same four timestamps — 0, 6.7, 13.3 and 20.0 seconds.

Compared properly, the arm is in the same pose at the same moment in all four. The cup has not moved, the sugar cubes have not moved. Only the room changed.

That is the point: the joint trajectories in the original parquet files should stay valid for every repainted video, because the motion is untouched.

How it holds the motion

Three things constrain generation, and they matter in this order:

Canny edge control	An edge map extracted from the source video, fed in at weight 0.9
Chunked generation	121 frames at a time, each chunk conditioned on the last frame of the previous one
Scene prompt	A structured JSON description of the room, the light and the camera

The edge map is the leash. The arm's silhouette is in it frame by frame, so the model cannot invent a different motion without contradicting its own control input. Set the control weight too low and you get a prettier video of a different robot.

Chunking is how a 16B model generates an 800-frame episode in 122 GB of unified memory at all: 121 frames, keep the last one, condition the next 121 on it. I checked a handful of episodes at the chunk boundaries by eye and found no visible seam, which I would attribute to the edge control reasserting the true arm position on every frame regardless of what the previous chunk drifted toward. By eye, on a handful — not measured across all 309.

I skipped SAM2 segmentation entirely. The reference workflow uses edge plus segmentation — but it runs edge at weight 0.9 against segmentation's 0.1. For a tenth of the control signal you install a second model and give it a chance to fail. Edge alone is a first-class control.

The settings, in full

Model	nvidia/Cosmos3-Nano — 16B, bf16, ~32 GB of weights
Mode	video2video
Control	Canny edge, weight 0.9, preset_edge_threshold: medium. No segmentation
Sampler	UniPC, 35 steps, shift 5.0
Guidance	text 4.0, control 1.5
Resolution	480 bucket at 4:3 → 736 × 544, 30 fps
Chunking	121 frames, 1 conditional frame, 0 for the first chunk
Seed	2026, fixed across all 309
Specs	309 = 103 episodes × 3 scenes

shift 5.0 is the 480p value; 10.0 is for 720p. The wrong one is a quiet quality loss, not an error, so nothing tells you. The 480 bucket is what the model was trained for — asking for more on a Jetson runs you out of memory mid-episode.

Six days at a steady 419 seconds a chunk

Every chunk the run produced, in order. 1,144 of them.

The first fifty average 426 s. The last fifty average 424 s. Mean across the whole run is 419 s, standard deviation 7.7 s, against a 429 s baseline I measured on ten clips before launching the full job.

There is one visible step, and it goes the wrong way to be a problem. Chunks 0–200 average 428 s, 200–1,100 sit at 417 s, the last 44 return to 424 s. An 11-second shift that happens once and reverses — ambient temperature over a week, not hardware degrading. Degradation is monotonic. This is not.

This is the chart I cared about. Thermal throttling costs wall-clock time and nothing else — it does not log, it does not warn. A run whose per-chunk time on day six is no worse than day zero was not losing time to heat, and that holds without reading a single temperature.

Six days at full load: 93.5 °C peak, no throttling

Sampled every 30 seconds for six days. The sawtooth is the gap between episodes: model unloads, board cools, next spec loads.

Peak junction was 93.5 °C. Thor's passive throttle is 109 °C and the fan steps are at 80, 86, 91 and 100 °C. Six days of continuous 16B diffusion sampling put it in the third fan step and left it there.

I pointed a desk fan at it for part of the week. Six-hourly mean junction temperature under load stayed between 81.9 °C and 84.5 °C for the entire run — there is no step where the fan went on. Same result as the ten-clip benchmark: external cooling bought nothing, because at 89 °C the device is already in its active fan band and losing no performance.

The desk fan on the Thor.

The fan was me being nervous, not the hardware needing help. It's an expensive box to cook: it launched at US$3,499, NVIDIA now lists it at US$5,499, and it's NZ$10,754 here in New Zealand.

What you actually get

Each of the 309 specs produces a directory:

File	Size	What it is
vision.mp4	40–80 MB	The augmented video, 736 × 544, 30 fps
control_edge.mp4	~20 MB	The edge map that constrained it
sample_args.json	~20 KB	Every argument, the full scene prompt, the source paths
sample_outputs.json	~20 KB	The same, plus what was produced

The two JSON files mean every clip carries its own provenance — how it was made sits beside it, not in a spec file elsewhere that may have moved. Frame counts matched their own specs on all 309. Worth checking: a short video fails silently in training rather than loudly here.

What it cost

Wall clock	144.7 h — 6.03 days (I predicted 132.7 h, so 9% over)
Specs	309 of 309, 0 errors
Output	13.3 GB, 1,239 files
Frame count vs spec	309 match, 0 bad
Mean draw under load	75 W GPU rail, 119 W total board
Energy	17.3 kWh — about NZ$5
Peak memory	80 GB of 122
Same job in the cloud (estimate)	~47 GPU-hours on an AWS g6e.xlarge (L40S) — about US$80 on Spot, US$87 on-demand

Five dollars of electricity, and the machine was never unavailable for anything else — it is a box on a desk, not a meter running. On AWS it would finish sooner, but my estimate for the cloud compute is about US$80 (a g6e.xlarge on Spot), against NZ$5 of electricity here.

It was not uninterrupted. Six days is long enough that something will go wrong, and on 26 October the job stopped. The driver skips any spec whose output already exists, so restarting cost one batch of five and nothing else. That property was worth more than any amount of tuning: the correct response was a single command, not a decision.

I would keep the resumability unchanged.

The one thing I would flag for anyone sizing their own run: the progress CSV lies if you read it carelessly. Field four is batch seconds, not per-spec seconds, and the driver runs batches of five, so summing the column gives 715 hours for a 143-hour run.

What I was aiming for, and what I got

I wanted more varied training data from the borrowed sugar-cube dataset. Same motion, same action labels, a different room. Each repainted video had to drop straight into the dataset in place of the real scene camera.

After the run I checked the outputs against the source, not just against their own specs. They don't match.

	Source	Generated
Frames, episode 0	1,380	690
Frame rate	60 fps	30 fps
Resolution	640 × 480	736 × 544
Field of view	full	473 of 480 rows

• The frame rate is half. My prep step downsampled to 30 fps before generating, so the video no longer lines up frame for frame with the action data.
• The size is wrong. The model wrote out its own 736 × 544.
• The view is slightly smaller. The default centre crop kept 473 of the 480 rows.
• It is video only. There are no state or action arrays with it, and no LeRobot dataset yet.

My check compared each output to its spec, so all 309 passed anyway.

So this run is not the augmentation I set out to get. I'm not calling it complete or a success.

I'll work through these in a later post.