Splunk is expensive but it makes searching logs so much faster and more effective. I think of it as SQL for unstructured data.
loki works great too and is FOSS
We really need an open-source implementation of the Splunk Query Language. The query language is what lets you actually find the few dozen relevant lines out of the billions of lines logged.