Cloudflare K2: serverless event streams

loufe

If I was a serious Cloudflare customer I would be seriously concerned about the security of my infrastructure with them. Yes LLMs can code fast but this is an almost frenetic pace of releasing new products, all with fewer staff.

necubi

I'm the author of the post and tech lead for K2. Happy to answer any questions!

theredsix

What's the benefits of this over traditional GKE pub/sub, kafka or another queuing service?

psanford

Object store is quickly becoming the new core data substrate. Lets build kafka, but on s3. Lets build github, but on s3. It feels like we going to see more and more "object-store first" systems in the next few years. I am excited about this future. Give me stateless servers and a storage bucket over having to manage systems with disks any day. I do wonder if we will see an expansion of the s3 api to support more of these use cases. S3 added a janky file append operation to their new express-one-zone bucket type, and limited to 10k total file append operations. I wonder what else we will get in the next few years.

ipkstef

i was trying to understand why i would use this over their current offerings of queues, and im really sick so my brain isn't working. so i ran it through ai You need... | Use -------------------------------------------------------|------- “Make sure this job gets done” | Queue retries / dead-letter handling | Queue delayed jobs | Queue distribute jobs among workers | Queue “Record that this event happened” | K2 multiple independent systems reading the same events | K2 replay old events | K2 ordered event streams | K2 Kafka-like architecture | K2

thepaulmcbride

This sounds like a super interesting product, but I'm always reluctant to build on anything that isn't a portable industry standard.

kirillkosolapov

How does this solution differ from AutoMQ and WarpStream? I’ve worked with one of them, and it is indeed a serverless solution built on top of S3. As far as I know, Kafka itself already supports offloading some data to S3 for long-term storage. Based on the articles—which I didn't fully grasp—I’m wondering if there are additional benefits mentioned, such as multi-region distribution (though I find it hard to imagine how that would be implemented).

addisonj

Congrats on the launch! Stream/event-based system are really powerful, but they are also just pretty complex, in no small part because the modeling of streams for most people today is really modeling Kafka topic/partitions which has a whole bunch of foot-guns and complexity. Making the individual stream really cheap and easy is a big simplification, especially if flexibly consuming a stream for both ordered and unordered use-cases is made simple. This looks to work for unordered, be curious to see how the managed dividing the work for the mentioned key-based ordering.

senderista

Minor correction: it is not true that no major object stores support appends--Azure Blob Storage has from the beginning.

senderista

Not a single mention of AWS Kinesis? That seems like the closest competitor.

eivindga

Congratulations on the launch! Looks very interesting indeed! As someone who enjoys writing Kafka streams applications I am also looking forward to the day you support the Kafka APIs. Having a cost efficient fully serverless Kafka compatible service would be great, and something I think many businesses would find useful. Great work!

sarkarghya

i think nats jetstream is superoverlooked in this space. the amount of throughput is more then enough for any medium scale enterprise as long as you know what you are doing

vira28

I wonder how many of data infra startups are wrapper on top of S3. Also, the boundary between OLTP and OLAP is blurring every day. For folks who want an off shelf version of this you may be interested in https://github.com/viggy28/streambed (Disclaimer: I am one of the committers)

wg0

Has anyone built anything on top of D1 + Durable objects for multi tenancy? How has been the experience?

max8539

Cloudflare is catching up to be a full AWS/GCP/Azure. At start it was just a few services around but they are adding everything else

pcthrowaway

Rather than requiring consumers to ack the batch, why not just have them submit the ID of the batch tail on `consume` requests?

elendilm

Nice. Our Monolog is similar in spirit. Instead of building Monolog on object storage (R2, S3), we built it on our Dip, thus achieving extreme low latency and parallelism for ingestion and consumption. Our novel architecture enables scaling to infinite consumers without upfront partitions (no magic, different tradeoff). This plays nicely with Slyp's data locality and application architecture. Monolog is built on Rust, is lightweight, and runs on mobiles and servers alike. The original reason for building Monolog was comically outlandish, Kafka was slow and required JVM, Redpanda was eating too much RAM - 2GB min which is absurd for our use case. Redpanda was also consuming so much CPU that the disgusting CPU fan noise had us feel emotional pain. We would very much like to build object storage on Dip, but we are currently preoccupied and hence don't have any immediate plans to build one in the short term. Hence, it is R2 for now.

nnx

The pricing is surprising, Data Produced at $0.04/GB seems reasonable (when compared to other clouds event streams) but Data Consumed being at the same $0.04/GB is rather steep... This means actual usage is $0.08/GB in the simplest case (one consumer) but fan-out consumer strategies get very expensive very fast.