Joint Embedding Predictive Architecture (JEPA) shift

Joint Embedding Predictive Architecture (JEPA) shift: text-to-video outcome prediction versus video-to-physical action

2026-05-25 · ai-architecture agentic-ai consciousness-cognition benchmarks-eval formal-methods · medium · source → · wiki →

key claims

Text-token prediction and JEPA video-outcome prediction are both passive prediction paradigms: neither requires the model to select or commit to actions that affect a physical environment during training or inferenceLeCun (2022)Assran et al. (2023)Assran et al. (2025)
Reinforcement learning theory formally distinguishes prediction problems (estimating the value of a fixed policy) from control problems (optimising the policy itself), identifying them as structurally different classes with different algorithmic requirements and feedback dependenciesIncompleteideas (n.d.)
V-JEPA 2 (2025) required a separate action-conditioned predictor trained on 62 hours of labeled robot-trajectory data after Stage 1 video pre-training, demonstrating that passive video JEPA prediction does not by itself yield physical action capabilityAssran et al. (2025)
JEPA representations learned from passive internet video transfer to the action-conditioned prediction stage efficiently, enabling zero-shot manipulation in new environments with minimal interaction data, meaning JEPA video pre-training is a necessary but not sufficient precondition for action groundingAssran et al. (2025)
Physical action introduces three requirements absent from passive video prediction: action-conditioned interaction data, closed-loop feedback at inference time using model predictive control (MPC), and irreversibility constraints that prevent simple error correction through retrainingHafner et al. (2023)Assran et al. (2025)
The text-to-video-JEPA transition is an improvement in representation quality within the passive prediction class: it captures motion dynamics and causal structure rather than surface token statistics, but the fundamental problem structure (passive, observational, corpus-trained) is unchangedLeCun (2022)Assran et al. (2023)
Cognitive science research on predictive processing distinguishes passive perceptual prediction-error minimisation from active inference (action selection to confirm world-model predictions), independently supporting the inference that passive prediction and physical action belong to different problem classes even when they share a representational foundationFriston (2010)Clark (2013)
For research planning purposes, improving video JEPA pre-training is primarily a data and compute scaling problem, while grounding JEPA representations in physical action requires safe interaction-data collection infrastructure, closed-loop evaluation environments, and robot-specific designAssran et al. (2025)Hafner et al. (2023)

Research Question

Is the shift from text-token prediction to Joint Embedding Predictive Architecture (JEPA)-style video outcome prediction the same class of problem as the shift from video prediction to physically grounded action in the real world?

Findings

Executive Summary

The shift from text-token prediction to Joint Embedding Predictive Architecture (JEPA)-style video outcome prediction is not the same class of problem as the shift from video prediction to physically grounded action. Both text-token prediction and JEPA video-outcome prediction are passive observational prediction tasks: neither requires the model to select actions that affect the world, and both are trained on corpora without environment interaction. The video-JEPA to physical-action shift crosses a structural boundary formalised in reinforcement learning (RL) theory as the prediction/control distinction: prediction estimates what will happen under a fixed policy, while control selects actions that change outcomes, requiring action-conditioned training data, closed-loop feedback, and irreversibility management. V-JEPA 2 (Assran et al., 2025) confirms this empirically by requiring a separate action-conditioned training stage beyond video pre-training before it can plan robot actions. JEPA representations transfer efficiently to the action-conditioned stage, reducing the interaction data needed, but this efficiency gain does not collapse the class distinction.

Key Findings

Text-token prediction and JEPA video-outcome prediction are both passive prediction paradigms: neither requires the model to select or commit to actions that affect a physical environment during training or inference.
Reinforcement learning theory formally distinguishes prediction problems (estimating the value of a fixed policy) from control problems (optimising the policy itself), identifying them as structurally different classes with different algorithmic requirements and feedback dependencies.
V-JEPA 2 (2025) required a separate action-conditioned predictor trained on 62 hours of labeled robot-trajectory data after Stage 1 video pre-training, demonstrating that passive video JEPA prediction does not by itself yield physical action capability.
JEPA representations learned from passive internet video transfer to the action-conditioned prediction stage efficiently, enabling zero-shot manipulation in new environments with minimal interaction data, meaning JEPA video pre-training is a necessary but not sufficient precondition for action grounding.
Physical action introduces three requirements absent from passive video prediction: action-conditioned interaction data, closed-loop feedback at inference time using model predictive control (MPC), and irreversibility constraints that prevent simple error correction through retraining.
The text-to-video-JEPA transition is an improvement in representation quality within the passive prediction class: it captures motion dynamics and causal structure rather than surface token statistics, but the fundamental problem structure (passive, observational, corpus-trained) is unchanged.
Cognitive science research on predictive processing distinguishes passive perceptual prediction-error minimisation from active inference (action selection to confirm world-model predictions), independently supporting the inference that passive prediction and physical action belong to different problem classes even when they share a representational foundation.
For research planning purposes, improving video JEPA pre-training is primarily a data and compute scaling problem, while grounding JEPA representations in physical action requires safe interaction-data collection infrastructure, closed-loop evaluation environments, and robot-specific design.

Assumptions

Assumption: The text→video-JEPA transition is best classified by the structure of the prediction objective (passive observational loss) rather than by the difficulty or quality of the resulting representations. Justification: LeCun's JEPA framework characterises both text and JEPA prediction as variants of self-supervised prediction in latent space; the structural differentiator used here is whether the model must select actions that affect the world (source: LeCun (2022) A Path Towards Autonomous Machine Intelligence
Assumption: JEPA Stage 1 representations are reusable as the foundation for Stage 2 action-conditioned prediction without modification. Justification: V-JEPA 2-AC freezes the Stage 1 encoder and trains only the action-conditioned predictor on top; this design choice implies that Stage 1 representations encode sufficient world-dynamic structure to support action consequence prediction (source: Assran et al. (2025) V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning
Assumption: Sutton and Barto's prediction/control distinction maps onto the JEPA/action-conditioning distinction in deep learning world models. Justification: Both distinctions apply the same structural test: does the model's output select actions that affect the environment? The concepts are directly analogous, though the neural network context adds considerations about convergence that tabular RL theory does not fully address (source: http://incompleteideas.net/book/the-book-2nd.html).

Analysis

Two converging lines of evidence support the non-equivalence conclusion. The structural line comes from objective function analysis: LLM cross-entropy, JEPA representation prediction, and action-conditioned world model training are each describable in the RL framework as estimation under a fixed policy (the corpus), estimation under passive observation (the video corpus), and active policy optimisation respectively. This maps directly to the prediction/control distinction. The empirical line comes from V-JEPA 2's two-stage architecture: if video JEPA and action-conditioned prediction were the same class, a single training stage would suffice. The fact that Meta/FAIR needed a separate stage with different data and a different objective is strong evidence of a structural boundary.

The primary counter-argument is a continuum view: text, video, and action can be seen as progressively richer forms of grounding on a single dimension of world-model completeness. This view has partial support in LeCun's hierarchical world-model framework, which treats all stages as variants of configurable predictive architectures. However, the continuum interpretation does not fully account for the data-type discontinuity: passive video and action-labeled trajectories are categorically different data types that cannot be trivially interconverted. Even granting the continuum framing, the data-type discontinuity means the two transitions have different practical requirements, which is the decision-relevant conclusion for research planning.

The class-boundary claim refers to the structural requirements of the objective function and feedback mechanism, not to the difficulty of the tasks. The text→video-JEPA transition may be empirically harder (more data, more compute, more engineering) than the JEPA→action transition in cases where abundant robot interaction data is available. Structural class membership is orthogonal to empirical difficulty ordering.

Risks, Gaps, and Uncertainties

V-JEPA 2-AC results come from a single lab and have not been independently replicated. The claim that 62 hours of robot data is sufficient for zero-shot manipulation may not generalise across robot morphologies, task types, or operating environments.
The V-JEPA 2 paper does not include ablations comparing JEPA-pre-trained versus randomly-initialised action-conditioned models; the magnitude of the transfer benefit is not precisely quantified.
The extent to which JEPA video representations capture counterfactual world dynamics (how the world would have evolved under unobserved actions) is not established by the V-JEPA 2 paper. This gap limits the strength of the causal-grounding claim.
Sutton and Barto's prediction/control distinction was developed for tabular and linear-approximation RL settings; its application to deep neural network world models involves additional assumptions about representational power and convergence that are not fully settled in the literature.
The cognitive science analogy (predictive processing / active inference) is an approximate parallel, not a formal derivation. Friston's Free Energy Principle applies to biological systems with specific homeostatic constraints; mapping it precisely to neural network architectures requires additional bridging work.

Open Questions

Does JEPA video pre-training capture counterfactual causal structure (how the world responds to interventions not observed in the training video), or only conditional predictive structure (what the world will look like given observed context)? This distinction is central to whether JEPA representations can support intervention planning without additional interaction data.
Can the prediction/control class boundary be crossed through synthetic action annotation of internet video, for example by inferring plausible motor commands from dense motion fields in video? If so, the interaction-data requirement might be addressable without physical robots.
Does the text→video-JEPA transition produce any qualitative capability gains in physical-world reasoning tasks (not only video understanding benchmarks), or do the gains remain confined to visual domains?
At what scale of video pre-training does JEPA representation quality plateau for downstream robot manipulation, and what does this imply for the cost-efficiency of the two-stage V-JEPA 2 approach?