Remix.run Logo
▲ necubi 5 hours ago

I'm the author of the post and tech lead for K2. Happy to answer any questions!

▲samtp 3 hours ago | parent | next [-]

After reading the post, still don't fully understand when you would use CF Queues vs K2. Can you help to elaborate a bit more?

▲necubi an hour ago | parent | next [-]

Sure! There are definitely some overlapping use cases, and we've seen folks using/abusing queues for use cases that are more appropriate to something like k2.

Queues are great when you have a unit of work that needs to be completed, retried, and tracked individually. For example, a shop might need to call a payment processor API that can fail or timeout, and retry it until it succeeds, while polling on the frontend for the state of that particular message. With a queue, you can insert a message tracking that payment, and have a queue processor that keeps getting sent it until it succeeds or has failed too many times.

In a queue each item is its own thing that's important to someone, and queues give you APIs to interact with that particular item.

K2 is for moving large volumes of data around. Pricing is per GB, not per message. Records are produced and consumed in bulk, and what matters is that all records are processed, but no one is querying the state of a particular record. K2 also supports multiple consumers for the same record, and long term retention. For example, all of your applications may emit events when things happen, and those events need to be read by an alerting system, a system that durably stores them, and a system that uses them to build ML features.

▲hasyimibhar 17 minutes ago | parent | prev [-]

Queues are for actions (do this), K2/event streams are for events (this happened).

▲e1g 2 hours ago | parent | prev | next [-]

I'm doing some testing, and while my writes are <1s, I'm seeing E2E delivery latency of p95=2.5s / p99=7.5s. Is this roughly expected?

▲aniketsauravv 5 hours ago | parent | prev [-]

Great product, congratulation on launch. Are you using this internally in any way?

▲necubi 5 hours ago | parent [-]

Yes! We originally built K2 to serve as the ingestion layer for Basin Pipelines [0], our stream processing product. We have a number of other teams building new products on top of it at the moment which I can't talk about yet :)

[0] https://developers.cloudflare.com/basin-pipelines/