Median also lies more than 99th/100th percentiles because in many workloads the same client is making multiple parallel requests and often waiting for the slowest.

See this piece from Jeff Dean [0]

Quote:

""" Component-Level Variability Amplified By Scale

A common technique for reducing latency in large-scale online services is to parallelize sub-operations across many different machines, where each sub-operation is co-located with its portion of a large dataset. Parallelization happens by fanning out a request from a root to a large number of leaf servers and merging responses via a request-distribution tree. These sub-operations must all complete within a strict deadline for the service to feel responsive.

Variability in the latency distribution of individual components is magnified at the service level; for example, consider a system where each server typically responds in 10ms but with a 99th-percentile latency of one second. If a user request is handled on just one such server, one user request in 100 will be slow (one second). The figure here outlines how service-level latency in this hypothetical scenario is affected by very modest fractions of latency outliers. If a user request must collect responses from 100 such servers in parallel, then 63% of user requests will take more than one second (marked “x” in the figure). Even for services with only one in 10,000 requests experiencing more than one-second latencies at the single-server level, a service with 2,000 such servers will see almost one in five user requests taking more than one second (marked “o” in the figure).

"""

[0] https://cacm.acm.org/research/the-tail-at-scale/#body-4

It's little surprising that the author was doing perf work and not already comparing distributions. As the rest of the article outlines, you learn a lot more with more data!

It's complicated, but really worth learning how prometheus and grafana heatmaps combine if you want dashboards for real time service data. Multimodal distributions are basically the expected outcome given all the caching done in distributed systems.

Instead of the CDF I like to use one minus the CDF, the fraction of requests not yet completed at any given time. Then you can make it a log log plot showing the entire tail with details invisible in the CDF because in the CDF the tail is essentially a horizontal line at one.

An interesting statistical example if you haven’t seen it before is the Anscombe Quartet :

https://en.wikipedia.org/wiki/Anscombe%27s_quartet

Four sets of X,Y datapoints that have exactly (or very close) common statistical parameters (mean, variance, correlation, linear regression, R^2), but with vastly different spatial distributions and “behavior” when looked at visually.

Imo, average has to be one of the biggest net negative stats you can collect. It is much better to collect just total throughput stat since that is the only thing that average is useful for, but unlike average stat, it doesn't cause confusion and tons of mistakes.

There are so many issues with average stats, not just the fact that people misunderstand it often, but also in collection and graphing, people often end up having final graph that ends up being some type of average of averages which becomes even more useless and completely meaningless.

If your metrics/monitoring system doesn't allow you to collect and graph proper distributions/percentiles, you really should change that.

The author is spot on. As an HPC performance engineer and computational artist, I could tell numerous stories about it, but I'll summarize three:

The common method for evaluating MPI performance on a supercomputer is with the OSU microbenchmarks, and with the -z option will print tail latencies. These are far more useful values to predict real-world performance on the big machines.

Second, while means DO say something useful, they are the top part of a profiling hierarchy. Deeper levels are sampling experiments (with minor effects on performance) to tracing (potentially larger effects on the code being measured). Seeing parallel timeline traces of a complex app has helped me and my team debug many performance problems. Look at HPCToolkit, Score-P, rocprof, and other tools.

Lastly, the reason I do computer artwork in the first place was because of a bug in an old fortran77 code which simulated turbulent mixing in 3d. I couldn't find our code mistake at first, but then I visualized the computational elements with a raytracer I had been playing with, and the result was not only shockingly beautiful, but pointed me to the line of code with the bug.

"The “after” curve is higher than the “before” curve to the right of 140ms which means more requests are finishing slower than before."

Think they made a typo - "before" curve is higher than the "after" curve to the right of 140ms, which means more requests take longer than before to finish.

Or am I the one misunderstanding?

My preference for visualising disributions is violin plots: https://en.wikipedia.org/wiki/Violin_plot

Similar to the ridgeline plot, you can see the distribution better without having to interpret derivatives as you do with the CDF plots.

With only having read the headline (as it is customary), I dare say: Aggregation without distribution always means nothing.

Headline is unnecessary clickbait, the mean was still signaling that something was wrong.

[flagged]

[dead]