I like the points and I'll probably link to this.

I'll add one from my experience as a PM dealing with very "testy" peers in early stage startups: don't do any of this if you don't have {enough} users -- rely on intuition and focus on the core product.

I was waiting for that comment to appear.

If your core product isn't any good, A/B testing seems like rearranging the deck chairs on the Titanic.