Am I missing something here, or is the only technical detail in the post "a trillion rows is a lot of data, maybe trying trinio will work this time"?

It feels like a lot of churn and not a lot of measurements or engineering.

▲ Deukhoofd 2 hours ago | parent | next [-]

If I was a user of this product I'd be really worried about the "We rewrote our software 5 months ago, but that didn't solve the issue we wanted to solve so we're doing it again, maybe this time we'll fix it" message of this post.

▲ roboben an hour ago | parent | next [-]

[Author] it was more like: we underestimated the problem and complexity of workloads our customers want to run. We did not rewrite anything but developing a new stack we can give to our customers doing warehousing. Agreed that this context is not really clear from the blog post.

▲ eddythompson80 2 hours ago | parent | prev [-]

Unironically, I think we will see a lot more of this given how engineering is being done currently. I have been involved in 2 green field projects in the last 2 years. Both have been rewritten or had massive "ship of theseus" type changes multiple times over that time window.

Engineers working on new products, thanks to LLMs, have very little understanding of the technical details of the product. They have the same level of understanding a traditional high level manager had of products they owned. If you ever worked in an engineering org with an M2 or M3 who always attended your outage review, and in every outage review something has to involve some issue with Postgres or your Kafka queue, that disconnected M3 would conclude:

    "Almost every outage involves some problem with Postgres. My conclusion, observed over the last 6-12 months, Postgres is bad. I see this new FancyDB or DynamoDB thing is popular on Twitter. We should try that"

I have lost count of the number of idiotic takes like that from M2s or M3s who have no idea how things work but instead focus on keywords they can latch onto that seems to be in the intersection of all issues. Then the logical solution is just to remove that and see what happens. This used to be very expensive, but if you can throw an LLM at a code base and say "Replace Postgres with FancyDB, split it to a list of logical checkpoints adding regression testing when missing to each checkpoint and open stacked PRs for all checkpoints", let it churn for an after noon, and 46k-line change PR (and $800 token bill) later you are ready to try FancyDB, why would you not give it a try? Then if you have issues with FancyDB, are you gonna spend the time to understand what's going on or just say "lets try DynamoDB next"?

	▲	vcarl an hour ago \| parent \| next [-]
		I worked at so many companies that operated this way, the tool is new but the approach is not. Those failed startups you hear about usually did a couple of these before hitting the ground
	▲	antonvs an hour ago \| parent \| prev \| next [-]
		> lets try DynamoDB next This is what this new feature called “loops” are for. Tell your agent to identify a list of databases and try them all, in prod, until it finds a good one. Be sure to tell it “no outages”.
	▲	twoodfin an hour ago \| parent \| prev [-]
		I have lost count of the number of idiotic takes like that from M2s or M3s who have no idea how things work but instead focus on keywords they can latch onto that seems to be in the intersection of all issues. This phenomenon is at the root of the hardest organizational & architectural challenges I’ve been involved in. It really needs a name so it can be tagged as a leadership anti-pattern. Good leaders find real connections to explore & strengthen. Bad leaders reach conclusions at the fuzzy pattern matching stage.

▲ lokar 3 hours ago | parent | prev | next [-]

To me to key “insight” is that if you are going to spin up single VMs (or even groups) per-customer and then route queries, you need to be able to estimate the query cost first.

I know snowflake does this, they do the query optimization before sending it for execution. So it knows about how many resources will be needed (it can obviously be wrong).

▲

roboben an hour ago | parent | next [-]

[Author] Basically yes, but we spun up pods on EKS. It was hard for the customer to estimate how much they might need and hard for us to know how much capacity to provide. Given our plans on pricing on CPUHours (or similar) the logic was kinda broken.

▲

gonzalohm 3 hours ago | parent | prev [-]

How does Snowflake use that information? You still need to pick the warehouse. It would be cool if there was an "elastic warehouse" where heavier queries are routed to bigger Warehouses, but that's not the case

▲

roboben an hour ago | parent | next [-]

[Author] Basically what we are doing with the help of Trino: a distributed query engine which can use as many resources as the query needs dynamically.

▲

lokar an hour ago | parent | prev [-]

There is an elastic option, I think . Also, even with a specific warehouse, not every query uses every node in the warehouse.

	▲	gonzalohm an hour ago \| parent [-]
		Right, there is an option to scale horizontally, if a lot of queries are queued, Snowflake can spin up another node. But that doesn't help if you query is spilling to remote storage. You need a bigger node to speed up those queries. Snowflake has an option to add more resources, which is supposed to be vertical scaling, but it doesn't work as well as increasing the size of the warehouse

▲ roboben an hour ago | parent | prev | next [-]

Hi! Author here.

We engineered, then we measured and then we tried to make it work, probably for a too long of a time before we decided to completely scrap our current architecture and go back to the drawing board. Happy to answer more questions.

▲

a34729t an hour ago | parent | next [-]

Why didn't you just use Iceberg and Trino from the get go (or Paimon if you are fancy)? This has been the accepted answer for quite some time now. The fast OLAP stores just don't scale as well as going object store to a proper query engine like Trino.

	▲	roboben an hour ago \| parent [-]
		We tried Iceberg and it was too slow writing data is the short story. >The fast OLAP stores just don't scale as well as going object store to a proper query engine like Trino this exactly why we started with this project.

▲

benrutter an hour ago | parent | prev [-]

> Happy to answer more questions.

Ah yay, I appreciate it! I feel like its rare to get a look at how these decisions cone about.

What did you measure? How'd you realise there was a problem? And did you measure anything to give you confidence that Trino won't hit the same issues?

	▲	roboben 44 minutes ago \| parent [-]
		With DuckDB and the queries we got, we had to provision such huge pods/nodes that we made it work but could never get to an economical viability. In the end, most of our customers had pods running 46 vCPUs / 360GB and asked for more. DuckDB is amazing, it really does everything right on a single node to make your query run successfully to the end almost always but it just doesn't work in our cloud env. Imagine when a query is running for 12h with this since and then another 10 from the same customer come in :) Trino on the other hand has a much more convenient distributed query concept, so you can much more dynamically assign resources.

▲ coolgoose 4 hours ago | parent | prev | next [-]

I don't get the blog post either.

▲ huqedato 3 hours ago | parent | prev | next [-]

This sounds more like: shoot first ask later. (OMG!)

▲ dominotw 3 hours ago | parent | prev [-]

> It feels like a lot of churn and not a lot of measurements or engineering.

They are big on shipping slop at lightnigg speed. So this is not surprising. Checkout posts by their ceo on cringedin.

	▲	sleepyfran an hour ago \| parent \| next [-]
		I don't know about their development or business practices, but if you think that their CEO posts on LinkedIn are serious and not satire I'm sorry to say the jokes are flying above your head a bit.
	▲	andor 2 hours ago \| parent \| prev [-]
		The entire website is a bit of a red flag to me, starting with the cookie banner featuring an image of Ursula von der Leyen and some misinformation (cookie banners are not legally required).