I've similarly spent a decade in the streaming space, started a stream processing startup, built three streaming platforms at various large companies... and I basically agree.

Streaming always felt like it was about to happen, and it kept not happening, in a year of the linux desktop sort of way. I (and others, who also optimistically started stream processing startups) thought it was a technology problem but ultimately it's a demand problem: very few companies actually need low latency processing. And continuously running, stateful systems are inherently more complex to operate and evolve compared to batch systems.

Now that relatively low-latency ingest into warehouses and datalakes is easily achieved, it's really hard to make the argument to invest in complicated streaming systems.

That said, it still feels like we've landed in a suboptimal spot. Stream processors (in particular, those following the dataflow model) solve two problems that are hard in batch systems:

1. Determining and signaling completeness for a time period (when have I likely received all of the data for 12:01pm, such that I can now safely process it)

2. Avoiding expensive recomputations for periodic queries

I do think long term we'll see some amount of streaming influence moving back into the batch systems, particularly as object storage gives us the ability to run stateful workloads with less operational headache.

(If anyone else finds these problems interesting, I'm hiring for my stream processing team at Cloudflare. Email in my profile)

▲

scott_s 3 hours ago | parent | next [-]

Agreed agreed.

During my time in the space, my pithy saying about the system I worked on [1] was that we could scale up or down. If you wanted to do streaming packet filtering with microsecond latency, we could do that. If you wanted to do complex analytics on structured data, we could do that. We did have deployments that "scaled down" and were more stream processing rather than streaming analytics. But analytics is by far the dominant use case, and SQL and relational databases are the better abstraction there. And for the stream processing cases, folks tend to stick to their existing lower-level stacks.

[1] I worked on IBM Streams, https://www.ibm.com/docs/en/streams/4.3.0?topic=welcome-intr..., which had its own language, compiler and runtime system. IBM sold this technology in 2023: https://21cs.com/en/resources/articles/2023/10/10/21cs-acqui....

▲

convolvatron 3 hours ago | parent | prev [-]

the fact that traditional database infrastructure handles many people's needs doesn't really obviate dataflow style architectures. personally I think (3) composition is a real potential win. I would also note that streaming is a really great base on which to build a distributed database that looks more like Postgres on top.

I guess I'm mostly confused about the idea that streaming systems should be trying to supplant sql databases, or that a failure to do so implies that they don't have utility. it looks like the GP has done some really interesting work on automatic parallelism. that isn't pointless just because most people building operational systems still reflexively reach for PG.

	▲	scott_s 3 hours ago \| parent [-]
		That work is still relevant! It's just that end-users don't need to be aware of it. The position of the paper I submitted, which I basically agree with, is that "streaming" shouldn't need to be something end-users care about. It's something the system does based on needs. Databases already have a dataflow style architecture: that's how they implement queries. Because SQL is relational, SQL queries become dataflow execution plans. One way to think about the programming model I worked on is that it was like exposing a query plan API directly to users, instead of giving them SQL.