> most of the time you should settup a small database with just enough data for that test, which hopefully is fast.

Yeah, this way you're not going to notice performance issues with your data model or application. Test databases should be of sizes comparable to what's expected in production, specifically after several years in production.