You really need to define how much you are willing to spend on logging/observability as a percentage of total budget. IMHO 5% is bare minimum 10% is better. I've worked for a company that had a dedicated storage array just for logging with Splunk and it was amazing and very much worth the money.

Good automatic tiering for logs is very useful as the most recent logs tend to be the most useful. I like NVMe -> hard disk -> tape library. LTO tape storage is cheap enough you don't need to delete data until it is VERY old.