Showing the robot, instead of retraining it

A new task should cost a few demonstrations, not a training run. Here's how we're teaching Cortex to learn from a prompt.

Teaching a robot something new has always meant collecting data and training a model. For a major new capability, that's the right investment. But most of what a warehouse throws at a robot isn't a new capability, it's a new variation: a product category it hasn't picked before, a bin arranged a little differently, a fresh SKU before peak season. For those, a full training cycle is a heavy tool. The skill is usually already within the model's reach. What's missing is a way to point at the new task and have the robot follow.

We want for manipulation what in-context learning gave language models: show the robot a handful of video demonstrations of a new task, and have it carry the task out, no retraining in between. This post is about how that works in Cortex, and why we built it the way we did.

Everything below has been tested on short, self-contained tasks, things on the order of a single pick-and-place: pulling a specified item from a tote and staging it for outbound, or pulling a labeled sample tray and setting it on a reader for a lab workflow. And it has been tested only on tasks that are similar to ones Cortex already knows, within a single domain. We have not validated it on long-horizon jobs, on tasks that combine several distinct skills, or on transfer across domains. The results we describe are the early, in-scope ones, and we're being upfront about that line.

Demonstrations are video, and they stay whole

A demonstration is a short video of the task being done. That's the whole interface: no action labels, no teleoperation traces, nothing to annotate. If you can film the task, you can specify it.

The tempting next step would be to boil those clips down into a single compact description and hand that to the model. We deliberately don't. Compressing demonstrations into one summary throws away exactly the detail that makes them useful: the texture of the approach, the way contact is made, how the motion differs at the start versus the end. Instead we keep the demonstrations whole, in a buffer the model can draw on, and retrieve from them as needed.

So the mechanism is retrieval. At each step $t$, the robot's current observation $o_t$ is the query. We embed it and compare it against the demonstration buffer $\mathcal{D}$, retrieving the $k$ most similar segments:

where $\phi$ is a frozen visual encoder and $\text{sim}$ is a similarity measure. The retrieved set $\mathcal{R}_t$ is placed in front of the model as in-context examples, the same idea as few-shot prompting in a language model: the relevant examples sit in the context, and the model attends to them while it acts.

Because $\mathcal{R}_t$ is recomputed every step, the examples the robot sees change as the task unfolds. Early in a pick, the retrieved moments are about the approach; as the gripper closes, they're about the grasp; on the way out, the placement. The guidance always comes from the part of the demonstration that matters at that instant, rather than from an averaged-out picture of the task as a whole.

A demonstration is just a short video. We keep each one whole rather than compressing it into a summary, so the texture of the approach, the contact, and the release all stay available to retrieve from.

In plain terms:

you teach the robot by showing it a few videos of the task, and it keeps those videos intact to refer back to instead of cramming them into one shorthand note.

The demonstrations reach the plan and the imagination

Cortex doesn't act straight from perception. It first forms a high-level plan, a read of what the task is and how to approach it, and then imagines how the scene is likely to unfold before committing to a motion. Two parts of the system carry that: a vision-language model that forms the plan, and a world model that imagines the futures.

For either part to do its job on a new task, it has to know what the task is. So the retrieved demonstrations go to both. Conditioning both stages on the same retrieved set, the plan $s_t$ and the imagined next observation $\hat{o}_{t+1}$ are:

The vision-language model receives the demonstrations the way a language model receives few-shot examples: the segments in $\mathcal{R}_t$ are placed in its context alongside the current scene, and it attends to them while forming the plan $s_t$. This is the well-understood path, and it's well-understood for a reason, it's how in-context learning works everywhere else. A model trained only to imitate doesn't attend to in-context examples on its own; it learns to once, in a single priming step. After that, every new task is just a matter of supplying the demonstrations, as long as that task stays close to what the model has seen before. We've leaned on that closeness deliberately: the demonstrations we've tested with are variations on familiar tasks within one domain, not jumps to genuinely new behavior.

The world model receives the same $\mathcal{R}_t$. The world model's job is to picture how the scene will play out, and on a new task those imagined futures should look like the demonstrated execution, not a generic guess. Because demonstrations are video and the world model already thinks in video, conditioning it is a natural fit: we're showing a video model the kind of footage it already works in. That turns out to be cheap to switch on, a light adaptation rather than a retrain.

The payoff from feeding both is consistency. The same $\mathcal{R}_t$ appears in both expressions, so the plan and the imagined futures are drawn from one shared source and agree on what task is being performed. The robot decides what to do and pictures how it will go from the same evidence, which is exactly what you want when the two have to line up for the motion to come out right.

Without demonstrations:

Robot gets stuck opening the box, as the lid is slightly different to previous boxes

With demonstrations:

Robot handles opening the box after being presented with demonstrations of this new box type without retraining.

In plain terms:

The robot both decides what to do and pictures how it will go, and we make sure the example videos inform both, so its plan and its expectations match.

The robot knows when it's on familiar ground

Retrieval hands us a useful by-product: a distance. When we look up the demonstration segments closest to the current view, we also learn how close they actually are. Writing $d_t^{*}$ for the nearest segment, that distance is:

This is a direct read on how well the demonstrations cover the situation the robot is in. A small $\delta_t$ means the nearest demonstration is a close match, the robot is on well-trodden ground. A large $\delta_t$ means it's in less familiar territory. That gives us a natural signal for how much the robot should deliberate: lean on what was demonstrated when the match is close, and think more carefully when it isn't. It's a clean way to let the system gauge its own confidence from the demonstrations themselves, rather than assuming it, and it fits naturally alongside how Cortex already reasons about how much to plan. In our tests this signal stays small, because the tasks sit close to familiar ground by design.

A line plot of the nearest-segment distance $\delta_t$ across the steps of one execution: low and stable while the robot is on well-trodden ground, with any spikes marking moments the demonstrations cover less well.

In plain terms:

by checking how closely the current situation matches the videos it was shown, the robot can tell whether it's on familiar ground and think harder when it isn't.

Why this fits Cortex

This builds on what Cortex already is rather than bolting something onto the side. The system's whole premise is that forming a plan and testing it against imagined futures beats reacting frame by frame. In-context learning extends that premise from how the robot acts to what it's being asked to do: the task can be set by example at runtime instead of baked in by training. The plan-then-imagine structure is unchanged. We've only changed where the task comes from, and made sure both halves of the system hear it.

We're being deliberate about scope. The near-term aim is adjacent variation, a new object category, a new bin layout, within a domain Cortex already runs, learned from a handful of video demonstrations with the weights frozen. The win is concrete and operational: a new SKU defined by a few clips, handled without a training run. The value is the training cycle we no longer have to spend.

That's the direction we're building. The demonstrations stay close to the robot as something it can draw on moment to moment, the parts that plan and imagine both see them, and how much the robot leans on them is measured rather than assumed. We're teaching the robot the way you'd teach a person a new task on the floor: show it, and let it watch.