Sometimes a company just isn't able to manage a data warehouse; I've worked for a few.

I even phone screened with this one but couldn't cover their crazy quilt of requirements; at that time they were also looking for someone to rewrite the clickhouse optimizer, and they were stuck trying to reshard their cluster, which should take an afternoon.

I still like the company because their business prop reminds me of what we were doing in 2009, a kind of federated analytics organization on a centralized data warehouse that enables a lot of tools to be developed quickly at large scale.

But goddammit they keep picking these half-working platforms and flip-flopping from one to another. We were crushing millions of queries against 2T-row tables on Vertica in 2010, and it's only gotten better for customer-facing data apps, which is what Posthog is. The company doesn't seem to have a baseline of what to expect; the market is too saturated with slow tools that lower expectations.

Author here. This is not about our internal warehouse but the warehouse product we offer to customers. This is still running on our centralized Clickhouse. Long-running data modelling queries and multi-tenancy is a challenge on Clickhouse.

I worked at a place with a couple of overly optimistic principals who were so happy about their computationally convoluted system responding to queries in 600 ms, but it practically meant we needed almost half a CPU per request, and then pre-autoscaling, a strange requirement to be able to failover from taking a region offline so we ran each at around 60% utilization, putting us at about 1 CPU per req/s. Which is just fucking embarrassing.

The first big system I worked on is the one where I learned not to generate expensive responses from a read mostly system, instead you should make writes expensive so reads are cheap. That project had an almost equally convoluted responses, and yet we were doing 30 req/s per instance on, IIRC, 4 core boxes. Adjusted for hot spare capacity, that ancient system was servicing 4x as many requests per core, 8x as configured.

A big part of the problem, besides selling a “premium” service to customers, was drinking the flavor of Knuth’s “premature optimization” misquoted koolaid that convinces people that all substantial speed improvements come from major architectural shifts. Which is how in the time I was on that team of 6-9 people, I delivered over half of the response time and cluster size improvements. If you count the work done to support upgrading the runtime, which brought perf improvements of its own, then more like 2/3.

Sounds a bit like this team might have the same disease.

The problem with internal service teams, of which this seems to be one, is that dealing with noisy neighbors is always a bigger problem than anyone is willing to tackle because it involves confrontation. Team X gets punished every time Team Y introduces a new feature that leans on the shared resource harder. It’s a multi tenant system without the fairness guarantees that we usually build into proper multi tenant systems.

Sometimes the easiest way to solve this is sharding. You end up deploying the same service many times and routing different classes of requests to different machines so if Team Y keeps trying to wreck the Commons they slow themselves down faster than they slow everyone else down. Those sped up consequences create a feedback loop that almost works instead of an amorphous one with consequences so far removed from actions as to become academic.

On that second team I mentioned we had an unfortunate situation of introducing things that took 100 ms off of response time and then three months later we were right back where we had been due to new features and creeping complexity of user content. It was demotivating for at least half of the team. My biggest contributions lowered the slope of that line, which allowed us to finally carry improvements over from year to year instead of being on the Red Queen problem.