Why Was Apache Kafka Created?

The look of the page -- that's Substack's default UI, you can't control it too much. The other images are created by me.

I'm simply curious what parts give that "cheap" look so I can improve. On Reddit I've had massive amounts of downvotes because they guess the content is AI, when in fact no AI is used in the creation process at all.

One guess I have is the bullet points + bolding combo. Most AIs use a ton of that, and rightly so, because it aids in readability.

Companies get standard tech stacks people are happy to work with, because working with them gets people experience with tech stacks that are standard at many companies. It's a virtuous cycle.

And sure even if you need just a specific thing, it's often better to go slightly overkill for something that's got millions of stack overflow solutions for common issues figured out. Vs picking some niche thing that you are now 1 of like six total people in the entire world using in prod.

Obviously the dose makes the poison and don't use kafka for your small internal app thing and don't use k8s where docker will do, but also, probably use k8s if you need more than docker instead of using some weird other thing nobody will know about.

Disclaimer: I'm a dev and I'm not very familiar with the actual maintenance of kafka clusters. But we run the aws managed service version (MSK), and it seems to just pretty much work.

We send terrabytes of data through kafka asynchronously, because of its HA properties and persistent log, allowing consumers to consume in their own time and put the data where it needs to be. So imagine, many apps across our entire stack have the same basic requirement, publish a lot of data which people want to analyse somewhere later. Kafka gives us a single mechanism to do that.

So now my question. I've never used MQTT before. What are the benefits of using MQTT in our setup vs using kafka?

Kafka isn’t a queue. It’s overkill to use it as one.

Kafka is a great place to persist data for minutes, hours or days before it’s processed. It fully decouples producers and consumers. It’s also stupidly complex and very hard to operate reliably in an HA configuration.

MQTT is good for when data needs to leave or enter your cloud, but persistence is bolted on (at least it is in mosquitto), so a crash means lost data even though you got a PUBACK.

Sure, it can do it. But it's not efficient or what it's good at.

The dimensions we focus on are number of streams (unlimited, so you can do granular streams like per user or session), internet accessibility (you can generate finely-scoped access tokens that can be safely used from clients like CLIs or browsers), and soon also massive read fanout for feed-like use cases.

jokes aside, we got a shift in the industry where many java programs were replaced by electron-like programs which now take 20x memory

Is C++ bloated because of the memory Chrome uses?

With over 15 years of professional experience since then, my perspective has shifted: Java demonstrates its strength when stability, performance, and scalability are required (e.g. bloody enterprise)

A common misconception comes from superficial benchmarking. Many focus solely on memory consumption, which often provides a distorted picture of actual system efficiency.

I can point to EU-scale platforms that have reliably served over 100 million users for more than a decade without significant issues. The bottleneck is rarely the language itself, it is the depth of the team’s experience.

The JVM tends to hold onto memory in order to make things faster when it does wind up needing that memory for actual stuff. However, how much it holds on to, how the GC is setup, etc are all tunable parameters. Further, if it's holding onto memory that's not being used, these are prime candidates to be stored in virtual memory which is effectively free.

Nonetheless, tracing GCs do have some memory overhead in exchange for better throughput. This is basically the same concept as using a buffer.

-----

And can you tell which of these websites use Java from "the feel"? AWS cloud infra, a significant chunk of Google, Apple's backends, Alibaba, Netflix?

I haven't worked with too much Java, but I suspect that the distaste many have for it is due to its wide adoption by large organizations and the obfuscating "dressed up" tendency of the coding idioms used in large organizations.

The runtime isn't inherently slow, but maybe it's easier to write slow programs in Java.

Maybe I have been lucky, or that the practice is more common in certain countries or eco systems? Java has been a very productive language for me, and the code has been far from the forced pattern usage that I have read horror stories about.

Golang has little to distinguish itself technically. It has a more modern std lib (for now) and isn't Oracle.

Which aren't trivial, but they aren't Trump cards.

1: https://cwiki.apache.org/confluence/display/kafka/kip-500:+r...

Go back to the article, it wasn't about event-sourcing or replacing a DB for application code.

Dismissing this as «just a social network» understates the real constraints: enormous scale, global privacy rules, graph queries, near-real-time feeds and abuse controls. Periodic DB queries can work at small scale, but at high volume they either arrive late or create bursts that starve the primary. Capturing changes once and pushing them through a distributed transaction log such as Kafka evens out load, improves data timeliness and lets multiple consumers process events safely and independently. It does add operational duties – schema contracts, idempotency and retention – yet those are well-understood trade-offs. The question is not push versus pull in the abstract, but which approach meets the timeliness, fan-out and reliability required.

> You don’t need push, it’s just a performance optimization that almost never justifies using a whole new tool.

It is not about drama but about fit for purpose at scale.

Pull can work well for modest workloads or narrow deltas, especially with DB features such as incremental materialised views or change tables. At large scale, periodic querying becomes costly and late: you either poll frequently and hammer the primary, or poll infrequently and accept stale data. Even with cursoring and jitter, polls create bursty load and poor tail latencies.

Push via change data capture into a distributed log such as Kafka addresses such pain points. The log decouples producers from consumers, smooths load, improves timeliness and lets multiple processors scale independently and replay for backfills. It also keeps the OLTP database focused on transactions rather than fan-out reads.

This is not free: push introduces operational work and design care – schema contracts, ordering being per-partition, duplicate delivery and idempotency, back-pressure and retention governance including data-protection deletes. The usual mitigations are the outbox pattern, idempotent consumers, DLQ's and documented data contracts. The data processing complexity now belongs in each consumer, not the data processing engine (e.g. a DB).

Compute–storage separation in modern databases raises single-cluster ceilings for storage and read scale, yet it does not solve single-writer limits or multi-region active-active writes. For heavy write fan-out and near-real-time propagation, a CDC-to-log pipeline remains the safer bet.

To sum it up, both pull and push are valid – engineering is all about each specific use case assessment and the trade-off analysis. For small or bounded scopes, a well-designed pull loop is simpler. As scale, fan-out and timeliness requirements grow, push delivers better timeliness, correctness and operability.

"Unavailable Due to the UK Online Safety Act"

:(

The issue in the docs was that there are no available Jetstream numbers, so I talked over a video call to the field CTO, who cited the base NATs numbers to me, and when I pressed him on if it was with Jetstream he said that it was without: so I asked for them with Jetstream enabled and he cited the same numbers back to me. Even when I pressed him again that “you just said those numbers are without Jetstream” he said that it was not an issue.

So, I got a bit miffed after the call ended, we spent about 45 minutes on the call and this was the main reason to have the call in the first place so I am a bit bent about it. Maybe its better now, this was a year ago.