I’ve only used Splunk with one set of devs and maybe we were doing it wrong, but it didn’t feel fast to me.

Several of us were working hard to move everything into Prometheus that made any sense to be in Prometheus instead of Splunk.

Notably any time we had a production issue that it was unclear which team was responsible, Splunk became the bottleneck because we started exceeding quotas immediately.

▲

UltraSane 2 days ago | parent [-]

Splunk is one of the best software I've ever used but it HAS to be used with very fast storage to be effective. I've only used it on enterprise grade storage arrays and servers with lots of RAM for caches. On modern PCIe 5.0 NVMe drives it is stupid fast.

I'm not sure what you mean by exceed quotas because Splunk is normally licensed on GB ingested per day. This can lead to bitter fights between teams over how this is allocated.

The good thing about this license model is that you can use as much hardware as you want for no extra license cost.

	▲	hinkley 2 days ago \| parent [-]
		> used with very fast storage That’s sounds like self hosting. Which is not the only product they offer. But you still have hardware that can only run so many queries at once and then starts queuing any additional request, yeah? Once you have a dozen people on a call it went to shit. Only occasionally ran into problems like this with graphite. But you need a lot of people looking at a very large dashboard to start feeling refresh delays.