We’re running standard Prometheus on Kubernetes (14 onprem Talos clusters, total of 191 nodes, 1.1k cpu cores, 4.75TiB memory and 4k pods). We use Thanos to store metrics in self-hosted S3 (seaweedfs) with 30 days retention, aggressively downsample after 3 days.

It works pretty good tbh. I’m excited about upgrading to version 3, as is does take a lot of resources to keep going, especially on clusters with a lot of pods being spawned all the time.

Using Victoria Metrics here. Very easy to set up and run. I monitor under 100 hosts and resource usage is low, performance is good.

One gripe is that they recently stopped publishing tarballs for LTS versions, which caused some grumbling in the community. Fair enough since they are developing for free, but felt like a bait and switch.

I am curious to hear from people on this forum, at what point will people practically cross the limits of prometheus, and straightforward division (eg, different prometheus across clusters and environments) does not work?

Regular Prometheus inside clusters for collection and alerting, Thanos for cross-cluster aggregation and long retention.

Nope. Mimir. Before that, Thanos.

Mimir

It's usually meant to monitor infrastructure, like memory and CPU usage of various processes and machines, request per seconds for servers, number of HTTP errors returned, etc.

https://prometheus.io/docs/introduction/overview/#what-are-m...

Apache 2 is scary? Maybe I'm missing something.

Victoria Metrics is well worth the migration. Much better performance and lower resource utilization in my experience.

It's a reminder to us all that when we think: "Hey, why sweating over this memory layout or that extra CPU expenditure, it's small and nobody will notice", there will be times when everybody will notice. Maybe notice as much as to switch to our competitors' products.

VM was a game changer for us, 7x reduction in memory and 3x CPU, plus the scaling flexibility.

Before reading the entire page did you consider clicking the header. This will bring you to the main landing page of the product/project which more often than not contains a helpful summary of what it is and why it exists. You can also apply this pattern to other unknown things you come across.

Sir, for thought leaders and CTOs they have a home page.

This is for DevOps minions.

if you can't tell what it is but still read the whole thing... i think the problem is on you?

https://prometheus.io/

I think most people that read release notes and changelogs want the text to be concise and easy to interpret when they're doing due diligence to decide when to start rolling out upgrades. They know what the software is about and don't care for some sales pitch.