JEPA is very similar to Model Predictive Control. Here, you swap the model of the system with black-box simulator (a neural net), apply some version of dynamic programming / Bellman equation to find the optimal action over planning horizon.

In the planning stage, by factoring out the optimization step from the rollout step, you can free the model to focus on learning the dynamics in latent space. (The other case, where the rollout+optimization happen in the learning loop, just becomes model-free RL.)

My background straddles classical & data-driven controls. This was the first similarity that popped out to me when I first started learning about world models.

Is JEPA just MPC but where the "M" is a latent-space model?

▲

kooi 5 hours ago | parent | next [-]

I think this is why it has a lot of potential. I'm also controls background, so I'm going to just throw stuff at the wall.

I'm thinking it's both M and P. M for the reasons you've mentioned. Then there is P where a forward horizon prediction of both plant and the world.

I think JEPA also touches on the perception tasks: What to do with sensor data? Classical control objectives are quite simple, drive the state to stable, follow a trajectory, stay safe. It's very deterministic: Sensor data is used to determine state. State is used to update dynamics, dynamics is used for the forward horizon.

So. MPC works for things like self-driving because the car model is fairly knowable and world model is predictable (rules of the road, knowable dynamics of other cars).

JEPA seems like a fusion of control, sensing, world model into an embedding space where the learning that occurs is a model free approach to how these are all related to each other.

	▲	LdBaringSwallow an hour ago \| parent \| next [-]
		Yeah not sure where the C comes from; models lack access to knobs and switches of the system they're on and taking over the OS isn't control over the hardware still since it can't upgrade its RAM, or GPU But the M+P aspect def fits.
	▲	nextaccountic 3 hours ago \| parent \| prev [-]
		Yeah it looks like humans, other animals, and life in general are much more willing to instinctively control things in the transient regime, away from equilibrium I guess this is the new frontier. Rather than doing safe, predictable, boring control, push the envelope on what trajectories are possible, so that control becomes more lifelike and has a degree of autonomy to recover from error states

▲

ainch 4 hours ago | parent | prev | next [-]

Yes absolutely. JEPA is far from the first approach to apply a learned dynamics model to MPC - it's just been well-marketed.

If you're interested, I'd recommend the TD-MPC series of papers, which also learn a latent world model, but achieves much better task performance in general. Unlike JEPA, TD-MPC learns to predict the task reward (what you'd call the cost in optimal control), so it's got a much better ability to rank trajectories during planning.

▲

nsingh2 5 hours ago | parent | prev [-]

In LLM parlance, I guess you could say MPC rollout and planning is the "harness" around a JEPA style world model.