> Why they'd need Kafka was always our first question, never got a good answer from a single one of them

"To follow the hype train, Bro" is often the real answer.

> If you need a queue, great, go get RabbitMQ, ZMQ, Redis, SQS, named pipes, pretty anything but Kafka.

Or just freaking MQTT.

MQTT has been battle-proven for 25 years, is simple and does perfectly the job if you do not ship GBs of blobs through your messaging system (which you should not do anyway).

▲

atomicnumber3 4 days ago | parent | next [-]

It's resume-driven development. It honestly can make sense for both company and employee.

Companies get standard tech stacks people are happy to work with, because working with them gets people experience with tech stacks that are standard at many companies. It's a virtuous cycle.

And sure even if you need just a specific thing, it's often better to go slightly overkill for something that's got millions of stack overflow solutions for common issues figured out. Vs picking some niche thing that you are now 1 of like six total people in the entire world using in prod.

Obviously the dose makes the poison and don't use kafka for your small internal app thing and don't use k8s where docker will do, but also, probably use k8s if you need more than docker instead of using some weird other thing nobody will know about.

▲

laughing_man 3 days ago | parent [-]

That's what happened where I worked. The people making the tech decisions were worried they weren't "keeping up" and committed us all-in on kafka. That decision cost the company millions.

▲

adev_ 3 days ago | parent [-]

> That decision cost the company millions.

And 5 years later the responsible of the decision left the company with a giant pile of mess behind him/her.

But let's see things positively: he can now add "Kafka at scale" on the CV.

	▲	laughing_man 3 days ago \| parent [-]
		That is exactly what happened.

▲

munksbeer 3 days ago | parent | prev | next [-]

> Or just freaking MQTT.

Disclaimer: I'm a dev and I'm not very familiar with the actual maintenance of kafka clusters. But we run the aws managed service version (MSK), and it seems to just pretty much work.

We send terrabytes of data through kafka asynchronously, because of its HA properties and persistent log, allowing consumers to consume in their own time and put the data where it needs to be. So imagine, many apps across our entire stack have the same basic requirement, publish a lot of data which people want to analyse somewhere later. Kafka gives us a single mechanism to do that.

So now my question. I've never used MQTT before. What are the benefits of using MQTT in our setup vs using kafka?

▲

cowanon2222 3 days ago | parent [-]

I use MQTT daily. I'm not sure why the commenter suggested it; it is orthogonal to queueing or log streams.

MQTT is a publish/subscribe protocol for large scale distributed messaging, often used in small embedded devices or factories. It is made for efficient transfer of small, often byte sized payloads of IoT device data. It does not replace Kafka or RabbitMQ - messages should be read off of the MQTT broker as quickly as possible. ( I know this from experience - MQTT brokers get bogged down rapidly if there are too many messages "in flight")

A very common pattern is to use MQTT for communications, and then Kafka or RabbitMq for large scale queuing of those messages for downstream applications.

	▲	adev_ 3 days ago \| parent \| next [-]
		> it is orthogonal to queueing or log streams. That is currently the problem. A lot of usage of Kafka I have seen in the wild are not for log stream or queing but deployed as a simple pub/sub messaging service because "why not".
	▲	munksbeer 3 days ago \| parent \| prev [-]
		Thank you.

▲

mdaniel 3 days ago | parent | prev | next [-]

I presume one will want to use https://github.com/eclipse-mosquitto/mosquitto if going that route because I seem to recall the "mainstream" MQTT project doing a rugpull but since I'm not deeply in that community, I don't have substantiating links handy

▲

physicles 3 days ago | parent | prev [-]

MQTT and Kafka solve different problems. At my current company, we use both.

Kafka isn’t a queue. It’s overkill to use it as one.

Kafka is a great place to persist data for minutes, hours or days before it’s processed. It fully decouples producers and consumers. It’s also stupidly complex and very hard to operate reliably in an HA configuration.

MQTT is good for when data needs to leave or enter your cloud, but persistence is bolted on (at least it is in mosquitto), so a crash means lost data even though you got a PUBACK.