| ▲ | tombert an hour ago | |
Yeah, as I said I wouldn't mind reimplementing it anyway. I still more or less remember what I did so I don't think it would be too hard for me to reimplement if I get enough free time. Kafka Streams is one of those things that I think is like 95% cool, but there are some design decisions I have mixed feelings on. One big thing is that while it does give an API for making new state stores out of the box and it's not too hard to write that, it is all dependent on blocking IO. This isn't that big of a deal with the built in RocksDB store because the latency isn't that high for it, so you can get away with everything being blocking, but if you want to substitute another store (e.g. Redis or even something like PostgreSQL), the naive version has you dealing with round trip latency for every item for a join, which can two or three orders of magnitude more expensive. Less naively you can implement batching and the like in your driver, which is what my first version did, but you do eventually have to add blocking, and the Kafka Streams library doesn't really utilize virtual threads so you're paying the full cost of it at the end. Eventually I just found it more elegant to make Kafka Streams non-blocking-aware and adding built-in semantics for batching to automatically amortize the cost of these things. Anyway, sorry, just kind of miss working on that project. I really should redo it. | ||