Anyone running ripgrep on a an HPC cluster against a large cluster filesystem needs to stop and redesign their workflow. This generates high amounts of small I/O which is the Achilles heel of any large cluster filesystem. You are exporting your workload onto the metadata mechanisms of the filesystem rather keeping it within the higher bandwidth capable memory subsystem on your cluster. It doesn't take but a couple users running these types of jobs simultaneously to bring a high-bandwidth filesystem to its knees. Just stop it already.

▲

btown 4 hours ago | parent [-]

Similarly, I've often wondered if this is the root cause of why GitHub has become fragile of late. All of a sudden you have operations on billions of tiny files that are amplified by AI usage. And object graphs are so fragmented by nature that it's hard to e.g. prefetch a page into memory and ensure that common git operations mainly touch that page's set of objects.

And if https://isolveproblems.substack.com/p/how-microsoft-vaporize... is to be believed even in the slightest, any unoptimized paths in Azure's filesystem abstractions could cause usage spikes to have massive splash radius.

	▲	gsu2 3 hours ago \| parent [-]
		> All of a sudden you have operations on billions of tiny files that are amplified by AI usage. And object graphs are so fragmented by nature that it's hard to e.g. prefetch a page into memory and ensure that common git operations mainly touch that page's set of objects. In theory, that's what git's packfiles are for: loose objects get packed together into a file, and if the file is well-arranged then objects that tend to get read together will also end up getting paged-in together (see also: https://www.kernel.org/pub/software/scm/git/docs/technical/p...). GitHub in particular has spent a lot of time and effort on packing things well (e.g. https://github.blog/open-source/git/scaling-monorepo-mainten...), so I doubt that the "billions of tiny files" problem is at the root of their recent instability.