> It generated something like 300-400 megabytes per hour. Despite the insane volume we made this work really well using just grep and other basic CLI tools.

400MB of logs an hour is nothing at all, that's why a naive grep can work. You don't even need to rotate your log files frequently in this situation.

Did you mean GB or TB otherwise you’re in the very low end of log volume

YEAH! If its less than a 100TB as second I don't even get out of bed!