> CI runs use loop devices on shared ephemeral VMs (one VM per filesystem): compare shapes and ratios, not absolute MB/s. Each job records a host-calibration anchor — see the table.

I think if you're not using baremetal for such tests, it's likely that the results are simply not comparable at all? What if another tenant is also using the disk?

▲

walrus01 4 hours ago | parent [-]

It's a fair point but it's also possible the person running the tests has a dedicated test hypervisor for this , so that different configurations of filesystems and VMs can be created and destroyed quickly in an automated manner.

If it's something as simple as a KVM hypervisor that only runs 1 test VM at a time (with no other load from anything else other than the basic systemd daemons, ssh daemon etc running on the hypervisor), the results could be very close to bare metal.

I can see it being very time consuming and annoying to do repeated manual bare metal OS installs and new partitioning/filesystem creation for such a large variety of tests.

The author does also say that performance isn't really the main thing but rather, data integrity:

https://github.com/fenio/modern-fs-benchmark

▲

toast0 3 hours ago | parent | next [-]

> I can see it being very time consuming and annoying to do repeated manual bare metal OS installs.

Well don't do that then. There's lots of other options. Probably the simplest is a single bare metal install on a simple filesystem on one device. run the filesystems under test on other storage dedicated to testing.

You could also boot into a network install and use local storage exclusively for testing.

▲

Farmadupe 3 hours ago | parent [-]

Yes exactly. The issueThe epherrality of the VMs isn't an issue, it's the _shared_ part that's the concern here. Going by the fact that the kernel is listed as "kernel 7.0.0-1012-azure" I feel like it's a fair risk that there may have been noisy neighbours.

	▲	fenio 3 hours ago \| parent [-]
		Obviously they could and probably were noisy neighbours. I'm not trying to hide that fact. Initial step for every benchmark is test of underlying device to at least reject completely unlucky cases. Also after almost 600 runs average is probably more or less correct... Also take a look at tests on real hardware. There are not many of them but there are some. I pointed to them in my first answer.

▲

fenio 3 hours ago | parent | prev | next [-]

For the main linked benchmark there are no dedicated test hypervisors. It's all based on GH runners with all the limitations and quirks that came with it.

Real hardware is used in: https://bartosz.fenski.pl/modern-fs-benchmark/real-hw/ https://bartosz.fenski.pl/modern-fs-benchmark/sas-hdd/

But unfortunatelly it's much more limited number of actual runs. sas-hdd is still in progress so numbers for it should increase over time.

▲

Farmadupe 3 hours ago | parent | prev [-]

> compare shapes and ratios, not absolute MB/s

In this case, given that the author's own disclaimer (above) already disclaims the numeric readings, I'm not sure how it's possible to make any inference on "shapes and ratios" derived from the numeric readings.