V-JEPA 2-AC

V-JEPA 2-AC post-trains a V-JEPA 2 visual encoder as a latent, action-conditioned world model. Each frame is encoded into its own spatial token grid — the three-dimensional patch embedding wants $2$ frames, so a frame is repeated to fill one tubelet. The predictor interleaves an action token and a state token before every grid, applies a frame-causal transformer, and emits the next latent grid. The visual target is kept in representation space, so the model predicts the consequences of an action without rendering pixels.

The released action-conditioned checkpoint contains one ViT-g/16 variant with a 1024-wide, 24-layer predictor and seven-dimensional robot action and state vectors. Its encoder uses the ViT-g wide MLP ratio, while the predictor retains the released four-times expansion.

Attention is frame-causal rather than token-causal: every token of a step attends to every token of that step and of the ones before it, and to nothing after. A step is therefore a unit of time rather than a position in a sequence, and the conditioning tokens sit inside it — action first, then state, then the spatial grid. The action and state tokens are rotated only along the time axis, since they have no place in the spatial grid, which is what separates this attention from the encoder's.

The training signal is next-frame prediction in representation space: the predictor is given the latents of frames up to $t$ and the action taken at $t$, and is scored against the latents of frame $t + 1$, both sides layer-normalised. Scoring against the same frame instead would be satisfied by copying the input, which the causal mask already permits. Planning and controller search are consumers of this latent transition model, not additional network layers — the forward is one transition, and a rollout is that forward applied to its own output.