> One can easily serve pages faster than an average AMP page if they wanted to

This is incorrect. You cannot beat prerendered. It does not make sense to implement AMP for people visiting your website directly. AMP is for link aggregators like search engines, news aggregators, and social media websites.