ARX-1 Engram

Abstract

Since GPT-4o, we have been trying to let large language models operate robots directly. At first they could only manage coarse movements in roughly the right direction. It was not until Opus 4.8 that we truly connected a robot SDK to an agent, and GPT-6 Astra then showed a startling understanding of the physical world. Imitation learning moved forward in parallel: ACT made it practical, and π0.5 revealed how much potential vision-language-action models hold.

We have long asked how a robot could think and grasp the way a person does. Engram is one possible answer. It keeps deliberation and control in one network with shared weights, a single multimodal foundation model, rather than in two separately trained models that talk to each other through a text interface. Drawing on the language model's understanding of the world and on finely annotated robot data, Engram plans, remembers and acts within one model.

Engram starts from an open-weight backbone (Qwen3.5-9B) and is trained on 5,000 hours of human egocentric video and 3,000 hours of real-robot interaction. Three choices in data and supervision set it apart: demonstrations that reach the same goal in many different ways, trajectories that keep failures and recoveries and are weighted by a learned progress critic, and a verb-grounded annotation scheme that teaches the model what each physical interaction is. Together they allow Engram to follow conversational commands, plan long tasks from a single goal, generalize to new tasks zero-shot, and combine known skills into new behaviors. At our current scale it already applies what it has learned well beyond the situations in its training data.

01

Understanding the Physical World

Engram combines broad multimodal understanding with an action expert trained on real-robot data. From human and robot video it learns how the physical world behaves: how objects move, deform and respond to contact, and what each step of a task is meant to achieve. It does not map pixels straight to joint commands. It first makes sense of the scene and the task, decides what should happen next, and then generates the motion that makes it happen. The action expert attends to the same visual and language tokens that produce the subtask, so the plan and the motion come from one representation.

Engram brings together four capabilities: multimodal perception, which fuses the camera views, the robot's state and recent history; language understanding, which reads instructions, dialogue and task context; physical reasoning, which anticipates how the scene will change as the task unfolds; and action generation, in which an action expert inside the model turns the current subtask into continuous control. Robot episodes supervise vision, language, proprioception and action together, and human egocentric video adds breadth across scenes, objects and strategies, labeled with the same task, subtask and verb layers.

Language ties these together. The subtask Engram announces is the subtask that drives its actions, so what it says it is doing and what the robot actually does stay aligned.

Episodic task memory

Engram deliberates at a lower rate than it acts. Between deliberative steps, the action expert keeps producing action chunks for the current subtask. Each deliberative step outputs a natural-language reply when there is something to say to the user, together with three structured fields: think, a grounded assessment of the scene against the goal; memory, a first-person summary of what has been done and what is in progress, carried forward unchanged unless the scene shows that something has happened; and subtask, one short, verb-led instruction for the next step, which conditions the action expert. The memory is plain language, so a person can read, audit and correct it. It also lets the model tell apart moments that look the same but call for different actions. For example, an empty gripper above the box can mean "not started yet" or "already finished".

The same memory holds what the cameras can no longer see. In the two runs below, an object is hidden and shuffled while Engram watches. It keeps track of where the object ends up and goes straight to it.

02

Language as the Interface

Engram handles dialogue, deliberation and control in one model, so a person can work with it at whatever level of abstraction suits them. They can delegate an entire goal and let the model plan the steps. They can steer it one instruction at a time, as a remote operator who uses words instead of joysticks. They can also redirect it mid-task, and it replans from where it is.

Whenever a turn calls for action, Engram issues a well-formed subtask that conditions its action stream, and it adds a natural-language reply whenever there is something to confirm, explain or ask. Because the subtask is written in plain language, the person always sees what the robot is about to do before it does it.

In the run below, Engram is given one goal, "arrange the tableware neatly". It breaks the job into subtasks on its own, including a handoff of the cup between its two arms, keeps the goal and everything already in place in its memory, and works through the subtasks one by one until the table is set.

03

Human + Robot Data

We hold a simple conviction: whatever the architecture, learning to control a physical robot requires real-robot data. Only real interaction captures the embodiment that a deployed policy has to master: contact forces, actuator latency, sensor noise and the robot's actual kinematics. Real-robot data has two limits, though. It is expensive to collect, and the environments it covers are narrow, because a robot can only be in so many kitchens, offices and warehouses.

Human egocentric video fills this gap. It is cheap to record at scale and spans a very wide range of scenes, objects, lighting conditions and manipulation styles. It also shows dexterous strategies refined over a lifetime of everyday experience. Engram uses the two sources for complementary purposes. Human video teaches it how the world behaves and how tasks are commonly approached, mainly through visual and language objectives. Robot data teaches it how this body achieves those effects.

Outcome-equivalent, strategy-diverse collection

Ask ten people to fold a shirt or pack a bag and you will see ten different procedures that all end in the same state. Conventional robot data collection removes this variety: operators are drilled on a single scripted motion, and the policy ends up learning that script. We regard this as a fundamental mistake. There is no single correct way to grasp an object or complete a task. The right choice depends on the object's affordances, its pose and its surroundings.

For every task we therefore collected many outcome-equivalent trajectories. They reach the same goal state through different grasp choices, sub-goal orderings and manipulation strategies. Operators were told what the result should be, not how to achieve it.

Full-spectrum trajectories

Clean demonstrations show what success looks like. They never show how to get back to success after something goes wrong. Engram's corpus deliberately keeps four classes of episodes:

Mixing these classes naively would teach the model to imitate mistakes. We therefore train a progress critic. For every segment of every episode it estimates how much closer to completion the task has moved and whether the segment helped or hurt. During policy training these estimates reweight the imitation signal. Segments that advance the task are emphasized. Segments that set it back are down-weighted but still kept, so the model learns to recognize failure and what should follow it.

Below is the critic's progress estimate on a real "fold the clothes" episode. The two dips are slips where the cloth leaves a gripper, and progress resumes once the robot recovers.

04

Verb-Grounded Annotation

Today's foundation models, the largest language models included, still have a shallow understanding of physical interaction. Data volume alone does not close this gap. For a policy to understand what it is doing, the supervision also has to state what is being done, precisely.

We observed that almost every manipulation, from folding laundry to assembling parts, is built from a small set of recurring physical verbs: fold, pull, pick, place, grasp, lift, rotate, twist and others. Models trained only on task-level text learn that they are "packing a bag". They do not learn that at this moment packing means pinch → pick → insert → release. Beyond task and subtask labels, every episode in our corpus, human egocentric and robot alike, is therefore labeled with verbs and contact phases, on four aligned layers.

The manipulation verb lexicon

Each verb has an operational definition rather than a dictionary one: the contact it requires and the observable condition that ends it. Neighboring verbs are separated by the same physical events, such as contact made, contact broken or the object leaving its support, which keeps labels consistent across annotators and gives the model a shared action vocabulary that carries over between tasks. The definitions are written for a gripper. In human video the hand takes the gripper's place, and the same events mark the boundaries. Select a verb to inspect its definition.

05

Emergence at Scale

As the data scale grows, Engram's improvements stop being uniform. Seen-task success rises early and then saturates. Zero-shot and compositional success improve only slowly at first, then climb steeply once the training data passes a little over 5k hours. We read this as a sign of emergence: given enough data, the model shifts from memorizing trajectories to reusing primitives it understands.

Beyond the demonstrations

None of the behaviors below has a training objective of its own. Each one applies something Engram learned to situations that its training data does not cover.

06

Evaluation

All evaluations are run on the ARX OpenX5. We report success rates on three task families: Seen (tasks and objects in the training data), Zero-shot (new tasks described only in language) and Compositional (new tasks that need a chain of 3–6 verbs that never appear together in training). Table 2 adds two diagnostic sets: Novel objects (seen tasks with objects that are not in the training data) and Recovery (seen tasks with a human-injected perturbation). Results for the final model and for the supervision ablations in Figure 6 are the mean of three independent evaluation runs of 80 real-world trials each (240 trials per family). The other scaling points in Figure 5, the other training checkpoints in Figure 7 and the other data-design variants in Table 2 are evaluated in a single run of 80 trials per family. Wherever the final model appears in these figures (the 8k-hour point in Figure 5, the 40k-step point in Figure 7 and the Engram row in Table 2), we reuse its 240-trial result, and the last checkpoint of the model without verb labels in Figure 7 reuses the 240-trial "+ Subtask" result from Figure 6. The 95% confidence interval is about ±4.5 points near 85% success and about ±6.3 points near 50% with 240 trials, and about ±7.8 and ±11 points with 80 trials.

Supervision granularity drives generalization

We trained otherwise identical models with progressively richer supervision targets. Memory labels lift seen-task success from 56.25% to 67.92%, because the model knows what it has already done, but they barely move zero-shot and compositional success. Subtask labels raise seen tasks to 81.25% and give the first large step on zero-shot and compositional tasks, because the model learns to break a goal into steps. Verb-level labels add a little more on seen tasks and another large jump on zero-shot and compositional tasks, because they turn whole tasks into reusable primitives.

Robot data deepens embodied understanding

Robot training gives Engram an embodied understanding that general-purpose models lack. We probe that understanding with held-out scenes and fixed question templates.

Table 1 Embodied language understanding. Subtask: next-subtask prediction accuracy. Memory: correctness of the predicted progress state. Verb F1: recognition of the verb currently being executed. Dialogue: end-to-end success of multi-turn natural-language commands on the robot.

Data design ablation

Table 2 Data design ablation (success / recovery rate, %). Recovery is the fraction of trials that still succeed after a human-injected perturbation, such as knocking the object out of the gripper or moving the target during execution.

07

Demo

Engram on the ARX OpenX5: it plans long tasks on its own, works alongside people through language and gestures, grasps and places everyday objects with care, and repeats a skill after watching a person do it once. Every run is fully autonomous and shown at 1× speed, with no task-specific fine-tuning. Zero-shot tasks are marked in the top-left corner.

08

Outlook

We are steadily expanding both data streams, with more scenes, more objects and more task families, so that Engram can take on a wider range of tasks and finer-grained manipulation. We are also bringing Engram to partners who build on ARX hardware. The behaviors described in Section 05 appeared at our current scale, and we expect more to follow.

Citation

@misc{arx2026engram,
  title        = {Engram: A Multimodal Foundation Model for Real-World Robot Tasks},
  author       = {{ARX Robotics}},
  year         = {2026},
  howpublished = {ARX Research Blog}
}