>At what point of optimization does it turn into 'real' high performance programming?

somewhat past optimizing the frame count of an entirely empty scene.

on that matter : is it a game engine if there isn't a game?

I totally agree with other comments though -- if there is no pressure to meet specific metrics or accomplish certain things with the product then there is no real pressure to improve past a window or framebuffer drawn to video, just declare it's a game engine that makes a million FPS and throw it on the portfolio.

game engine work gets tough (and rewarding) with 1) goals and 2) constraints -- without those two it's more or less just spherical-cow style work that is too ambiguous or vague for real application.