How do you know if you're making the right optimizations?

I struggle with this a lot with Marginalia's index. Where I identify a hot method in a prod profiler run, try to replicate it on a test machine where I can never get the same cache characteristics because everything in this space is like an onion of caching layers that you affect the real performance of the system. I may get it to run significantly faster, but that only sometimes makes the production profiler sample move its needle.

e.g. I've recently been experimenting with using a cursed hybrid model in Marginalia's index, where based on a mincore probe, I switch between mmap and io_uring for reading a cluster of pseudoadjacent data. There are real tanglible benefits both in the test machine and in prod with this, but the numbers do not agree at all about how the needle moves :P

▲

malisper 2 hours ago | parent [-]

I would probably dig into the reasons for the differences in the benefit on the test machine and in prod

I had an issue like this for optimizing pgrust. I had an optimization that showed no impact on my test machine (c8g.4xl) and showed a 20% improvement when ran on my mac. It turns out the issue was the instruction cache on the c8g.4xl was being saturated on the test machine but not on my laptop, moving the bottleneck to a different place

If you can consistently reproduce the performance difference, you're already half way there

	▲	marginalia_nu 2 hours ago \| parent [-]
		I'm pretty sure the reason for the difference is that production machine exists in a state of mixed memory residency and low grade resource contention that is incredibly hard to replicate in a test scenario (as the moment you start making queries the pages warm up, and the test becomes unreliable). The hard part about optimizing this type of code, IMO, is that there are so many cache layers, both in the CPU and the OS and sometimes in the storage medium. You can warm all of those caches up, but then you're testing a nonsense scenario that will basically never happen in a realistic scenario, where wall clock time is what matters, and not much the CPU is working or how many IOPS you're pushing.