| ▲ | vira28 2 hours ago | |
Author here. For context, I was the tech lead for the Postgres team at Cloudflare, and this came directly out of a challenge I kept hitting there: BI and dashboard teams needed to run long-running analytical queries, and the answer was always to spin up another bespoke read replica or stand up an ETL dump into an analytical database and query that. So the question I started with was: what's the fewest components I could get away with? That led to the architecture here — Streambed connects to Postgres as a logical replication subscriber (same mechanism as a read replica) and streams WAL changes straight into Apache Iceberg on S3, queryable from psql via an embedded DuckDB. There are a lot of edge cases to handle, and it's very much early days. Welcome any feedback. | ||
| ▲ | kikimora 28 minutes ago | parent | next [-] | |
To me being able to query over psql is secondary. I’m fine with any SQL. What is very important is being able to transform the data to better suite analytical queries. That is, define custom transformations, define how data sectioned and what indices available. | ||
| ▲ | ashtuchkin an hour ago | parent | prev [-] | |
Just wanted to say thank you! Very relevant to our use cases. I'll report if I find any issues. | ||