That implies that the A/B testing will lead to a new version that is substantially more engaging. Maybe so, but it seems that the most successful social media platforms arrived at their optimal version more or less immediately. For instance, I (naively?) doubt that much A/B testing went into designing HN. Yet, no other site holds as much of a grip on me.

Facebook seems to think it will, since they employ A/B testing across pretty much much every facet of the company. Check out Airlock for just one example:

https://engineering.fb.com/2014/01/09/android/airlock-facebo...