Everything is coming from PSRAM including frame buffer (at 15 fps) so performance is going to be abysmal.

You should be able to cache hot code and data in the SRAM. Although it'd significantly increase complexity.