This trick is actually used by some banking apps.

They fill app their mobile apps with junk data just to make the APK/IPA bigger. So if they need to push an urgent update, they won't have users that can't update because their phones are full to the brim.

I know two Italian banks that do it, Unicredit and Intesa. The latter was on the news when a user found out that one of the filler files was a burp recording [1].

[1] https://www.ilfattoquotidiano.it/2024/12/20/intesa-san-paolo... (in Italian)

Better fill those files with random bytes, to ensure the filesystem doesn’t apply some “I don’t actually have to store all-zero blocks” sparse-file optimization. To my knowledge no non-compressing file system currently does this, but who knows about the future.

Similarly, I always leave some space unallocated on LMV volume groups. It means that I can temporarily expand a volume easily if needed.

It also serves to leave some space unused to help out the wear-levelling on the SSDs on which the RAID array that is the PV¹ for LVM. I'm, not 100% sure this is needed any more² but I've not looked into that sufficiently so until I do I'll keep the habit.

--------

[1] if there are multiple PVs, from different drives/arrays, in the VG, then you might need to manually skip a bit on each one because LVM will naturally fill one before using the next. Just allocate a small LV specially on each and don't use it. You can remove one/all of them and add the extents to the fill LV if/when needed. Giving it a useful name also reminds you why that bit of space is carved out.

[2] drives under-allocate by default IIRC

> A neat trick I was told is to always have ballast files on your systems.

ZFS has a "reservation" mechanism that's handy:

> The minimum amount of space guaranteed to a dataset, not including its descendants. When the amount of space used is below this value, the dataset is treated as if it were taking up the amount of space specified by refreservation. The refreservation reservation is accounted for in the parent datasets' space used, and counts against the parent datasets' quotas and reservations.

* https://openzfs.github.io/openzfs-docs/man/master/7/zfsprops...

Quotas prevent users/groups/directories (ZFS datasets) from using too much space, but reservations ensure that particular areas always have a minimum amount set aside for them.

I always called it a “bit-mass”. Like a thermal mass used in freezers in places where the power is not very stable.

I knew I didn’t invent the concept, as there’s so many systems that cannot recover if the disk is totally full. (a write may be required in many systems in order to execute an instruction to remove things gracefully).

The latest thing I found with this issue is Unreal Engines Horde build system, its so tightly coupled with caches, object files and database references: that a manual clean up is extremely difficult and likely to create an unstable system. But you can configure it to have fewer build artefacts kept around and then it will clear itself out gracefully. - but it needs to be able to write to the disk to do it.

Now that I think about it, I don’t do this for inodes, but you can run out of those too and end up in a weird “out of disk” situation despite having lots of usable capacity left.

This saved us a couple times. At least until I had time to add monitoring to their old system to track disk usage. It was also helpful to use a tool called ncdu. It helps you visualize where most disk space is getting used up to track down the problem.

Also good for stopping phone-home auto firmware updates

I did this too, but i also zipped the file, turns out it had great packing ratio!

ext2/ext3/ext4 all automatically reserve an amount of space on a partition. 5% iirc

Interesting strategy, can't believe I've never heard of this one before.

Would it be more pragmatic to allocate a swap file instead? Something that provides a theoretical benefit in the short term vs a static reservation.

This is why I never empty the Rubbish Bin/trash Can on my Linux laptop until the disk fills.

This is my snippet i used alot. In doubt when even rm wont work just reboot.

Disc Space Insurance File

    fallocate -l 8G /tmp/DELETE_IF_OUT_OF_SPACE.img

Great idea, thanks!

Sounds like something straight out of Dilbert

Surely a 50% warning alarm on disk usage covers this without manual intervention?

Similar to the old game development trick of hiding some memory away and then freeing it up near the end of development when the budget starts getting tight.

I did this recently, aka, docker images prune. Can confirm, saved the day.

Love the simplicity and pragmatism of this solution

Some filesystems can be unable to delete a file if full. Something to be a bit worried about.

> A neat trick I was told is to always have sleep statements in your code. Just a few sleep statements that you can delete in cases like this. This won't fix the problem, but will buy you time and free up latency for stuff like slow algorithms so you can get faster code.

FTFY ;)

Would another way be to drop the reserved space (typically 1% to 5% on an ext file system)?

Even if your authorization is so sophisticated that nginx cannot do it, a common pattern I’ve seen is to support a special HTTP response header for the reverse proxy to read directly from disk after your custom authorization code completes. This trick dates back to at least 2010. The nginx version of this seemed to be called X-Accel-Redirect from a quick search.

Yeah it's a bit odd to use a Haskell server to serve a static file which nginx then needs to buffer. You'd do much much better just serving the file out of nginx. You could authenticate requests using the very simple auth_request module:

https://nginx.org/en/docs/http/ngx_http_auth_request_module....

gdu is really nice but ncdu, though slower, is very useful and is usually available on distro repos.

I can't recommend `dust` enough: https://github.com/bootandy/dust

I have an alias named usage with this in it:

    du -hs -- * .??* 2> /dev/null |  sort -h | tail -$LINES

Have you used ncdu? I wonder how this compares.

I use dust for this, but gdu looks nice, I'll give it a try. Thanks for sharing.

I also discovered gdu recently. It's really good. It saves me running du -h --max-depth=1 | sort -h a million times trying to find where the space has gone while you're stressing about production being down.

My fav line from the post is one above it. There is something very blunt and funny about it.

> It’s difficult to reason under pressure. Experience, that I didn’t have here, would have helped.

I was thinking the same thing - wouldn't blob storage or a CDN help?

Or NTP, if something is not working df -h and date are the first commands I input.

It's always lupu... I mean NTP or disk space.

Came here to say this

X-Accel-Redirect (Nginx sendfile), if supported by Haskell is the way, it is zero copy and will dramatically help in many cases.

If you are modifying the body is one of the cases where it doesn’t work.

mkfs.ext4 defaults to 5% reserved for root. -m 0 to turn it off.

Old war story: I had an old Sun 4/260 with 2 1G drives - I had SunOS on 1 and Gentoo on the other - my initial Gentoo install worked for a while but then the portage directory used all the configured iNodes - really weird errors and I could not figure it out at the time; error msgs maybe should mention inodes? I had to do #gentoo-sun IRC and someone suggested df -i which was indeed the issues (solve: you can configure extN filesystems to have more iNodes)

That happened to me exactly once in my 20-year career. It was on a web server (maybe even NGINX) that had too many cached files.

Even though it only happened once, I still set up monitoring for inode exhaustion.