Here's my two cents as a mere paper reader;

JEPA is really just a generalized encoder, so the JEPA created latents should be fed into a transformer trained with either user data or RL policy.

Although the above might not work great either because you said that vertical position was not predicted well!

Great article and I hope that you can carry on with the JEPA research