The author is spot on. As an HPC performance engineer and computational artist, I could tell numerous stories about it, but I'll summarize three:

The common method for evaluating MPI performance on a supercomputer is with the OSU microbenchmarks, and with the -z option will print tail latencies. These are far more useful values to predict real-world performance on the big machines.

Second, while means DO say something useful, they are the top part of a profiling hierarchy. Deeper levels are sampling experiments (with minor effects on performance) to tracing (potentially larger effects on the code being measured). Seeing parallel timeline traces of a complex app has helped me and my team debug many performance problems. Look at HPCToolkit, Score-P, rocprof, and other tools.

Lastly, the reason I do computer artwork in the first place was because of a bug in an old fortran77 code which simulated turbulent mixing in 3d. I couldn't find our code mistake at first, but then I visualized the computational elements with a raytracer I had been playing with, and the result was not only shockingly beautiful, but pointed me to the line of code with the bug.

▲

gtirloni 2 days ago | parent [-]

If you don't mind me asking, how does one get into HPC performance these days?

	▲	markstock 2 days ago \| parent [-]
		The nice thing is that every gpu is like a little distributed-memory machine, and programming it for optimal memory locality (registers, shared, global) and bandwidth is an essential skill. One step up is using multi-node distributed memory machines, which you could set up at home with openmpi, but is often easier to get access in an academic setting. In short, learn how to use CUDA and MPI. That's my personal take on "HPC", though obviously it can mean different things to different people.