There’s a repeated theme here that I don’t understand. For example:

> (3) That's where I decided to build a harder test, where each task takes its start from one demonstration and its goal from another

> (4) I also built an interactive demo where I could place the block anywhere and pick any goal. [but] The goals in the harder test are frames from expert demonstrations

Why lean so hard on the demonstration examples? Surely the actual task here is to move between any two arbitrary poses.

So if the demonstration set is statistically strange in any way, you keep banging into that. And even if you get great results, you still haven’t shown that it generalizes to the generic system. And the one thing you know 100% about a physical robot system is that you will never encounter exactly the same situation twice.

Is there actually a reason to pick eval examples from the demonstrations instead of independent samples from the underlying task?

I’m pretty sure that the T-shoving problem doesn’t have goal reachability issues. And if it did, test 4 could run into them since the start pose is arbitrary.

Agreed, I don't know much about JEPA but i wish this article had explained / motivated the memory graph more. "3. Fill gaps in memory with transformed memories" in particular seemed really really damning for the models—when you removed the examples from the demonstration that were near the goal state, the model completely broke. It seems like the only way you could make the model work again would be to manually flip and transform the memories it did have so there was something in the vicinity of the goal location.

That seems like a really massive failure of the model to generalize? This doesn't seem like a new technique, it seems like you've discovered a pretty big issue with the fundamental setup