The Kafka Replication Protocol with Kip-966

I'm writing a distributed DB rn ; and all the algs and tech they are using, SWIM ; hash ring ; the questions they are asking themselves - are pretty much standard ; it is not easy but it's a great field with different depth ; lot of successful companies don't dive that deep because it's uncharted experimentation failure & document process that is very very expensive to justify in a traditional context. I don't see how I would sell what i'm doing rn without getting it right first - and there is no point in proposing a new technology if there is no operational guarantees - And tbh a lot of people who are using kafka or doing distributed work are mostly doing the equivalent of setting up kafa consumers and publishers and that is enough and *it is fine*.

I'm a big fan of https://nats.io/ as a lightweight alternative to Kafka. I think of it like Nomad vs. Kubernetes: you can get 80% of the functionality for the cost of 25% of the complexity.

It is impossible to know what Northguard can and cannot do since it's an internal products.

Besides the implementation effort in building from scratch, it also opens you up to unknown unknowns, while the limitations of mature software are better defined. Just because Jepsen didn't test other alternatives doesn't mean they are free from issues.

Regarding the Kafka issues pointed out:

* 2 only affect you if you are using transactions (and I would be interested in knowing of alternatives that better handle exactly-once semantics);

* Consumer.close() is likely off the critical path of your applications (personally I never encountered these hangs);

* Aborted reads and torn transactions were mentioned as being fixed by KIP-890, of which the mentioned ongoing work was completed since that article was published: https://issues.apache.org/jira/browse/KAFKA-14402