Event Sourcing, Cqrs and Micro Services: Real Fintech Example

This! I've recently was hired to fix a company that tried to build their own niche ERP over the last 10 years and they had totally drowned in the confluent kool-aid. There is very very few projects where event sourcing is the best solution.

Yea I think that's a fair take.

If you peer underneath the covers of a lot of financial stuff, and it's effectively double entry accounting. Which is a giant ledger (or ledgers) of events

I'd call it a trade-off. I've done event sourcing for systems that didn't really need it, and the ability to fix things retroactively by just replaying the log through the corrected pipeline of processors was marvelous. It lends itself to functional programming with effects instead of banging on global state wherever and whenever, and that pays dividends. But there's no free lunch, and pipelines could get complex by needing to track "epochs" and keeping bug-compatible behavior for old messages that would be patched downstream. Since the system just built persisted state at the end, I could checkpoint it at a new state and retire the older epochs. Doubtful that would that pass a rigorous audit, but if that's not one of your requirements, you really can do event sourcing halfway and still reap a lot of the benefits.

Yes. And that's usually anything that touches money or is limited by it.

But you don't need to decide to use it. The people describing the requirements will tell you, insist on it, and threaten you if you don't do it.

Yeah, that's a big NO. Events are immutable. If an event is wrong, you post an event with an amendment. Then yes, rebuild the app state.

Instead of modifying the original (and incorrect) event, you can add a manual correction event with the info of who did it and why, and replay the events. This is how we dealt with such corrections with event sourcing.

It's poorly phrased but I'm not sure they meant "mutate the past". The keyword is "adjust" which could mean "append a correction".

No, that definitely happens.

There are two kinds of adjustments: an adjustment transaction (pontual), or re-interpreting what happened (systemic). The event sourcing pattern is useful on both situations.

Sometimes you need to replay events to have a correct report because your interpretation at the time was incorrect or it needs to change for whatever reason (external).

Auditing isn't about not changing anything, but being able to trace back and explain how you arrived at the result. You can have as many "versions" as you want of the final state, though.

The argument I've always heard for this was issues with code, not the event. If for a period of time you have a bug in your code, with event sourcing, you can fix the bug and replay all the events to correct current projections of state.

It would be outside of the normal exceptional cases, yes.

Like buggy data that crashes the system.

If you have the old events there, you can "measure twice, cut once", in the sense that you can keep re-running your old events and compare them to the new events under unit-test conditions, and be absolutely sure that your history re-writing won't break anything else.

It's not for just doing a refund or something.

Do you mean inter-system? Intra-system would mean on the same db, so a simple transaction would do.

For inter-system consistency, you’d probably need a reconciliation mechanism or some kind of 2 phase commit

Create an event type for transfer with amount, source account and destination account.

You're getting hung up on the "double" entries. That's mostly for reducing accidental and deliberate (fraud) calculation errors if you have humans crunching the numbers. The two entries can be in the same row if you like.

As for the inter-system transfers, It's not possible in the general case. You might not have a network cable between the two systems. And if you do, you run into CAP, two-generals, etc.

ACID is out of the question because the two parties don't share a DB. And if they did, some tech lead would turn down the acidity level for performance reasons.

The best you can do is intelligently interpret the information that has been given to you. That's all ledgers are: a list of things that some system knows. 'UPDATE CustomerBalance...' (CRUD) is not a fact, but 'Customer paid...' is a fact, and that's all event-sourcing is.

Any problem with event sourcing can be solved with more events (said semi-sarcastically).

In this case it’s XTransactionStarted, XTransactionDepositConfirmed, and XTransactionCreditConfirmed or something along those lines. External interactions tend to follow that kind of pattern where it tracks success/failure in the domain events.

The command side of CQRS tends to be the services that guarantee ordered events either via the backing database or with master-slave topology.

Data sync is one scenario where it can be useful.

What did you find to be the most problematic?

They appear to be ESL and if in Germany they use "IT" the same way we do in France, I can understand why they skipped the "industry" part.

For me the most important ideas is an immutable ledger and isolating the primary OLTP database from the secondary support services. Reporting, user management, and notifications etc. can be satisfied with a stale copy of the transactional data.

The answer is rarely so clearly cut. This is a natural extension of the bank system of having both account balance and account transactions. How would you feel if the bank only knows your account balance but not transactions? It has its uses. It's just a bit overused in places where it's unnecessary.

> proven double-entry ledger approach.

A double-entry ledger is a combination of a process and a view that was mistaken for a data model centuries ago, and that mistake became embedded.

Fundamentally, you’re dealing with a sequence of events. The double-entry ledger is a particular result of processing those events - a view. There are many other useful views.

This is well understood in academic accounting. See e.g. https://en.wikipedia.org/wiki/Resources,_Events,_Agents for an alternative system that doesn’t make the same mistake.

* Most single-db deployments give up on ACID for performance reasons (see READ_COMMITTED)

* Even if you have ACID, it's not sufficient for distributed systems. Its guarantees will keep one node consistent with itself. No transactionality between the customers app and your db.

Maybe you're one bank with all the customers, but as soon as you want to talk to other banks, are you really going to share the one ACID instance? Who's the DBA?

Currently the state of fintech is 90% of devs being in denial that they're in a distributed system.

> proven double-entry ledger approach.

Yes. In that language, the ledger is the list of events. If today's devs were around 300 years ago they'd be calling for 'balances' instead of 'ledgers' because they're simpler.

I think they needed to be clearer about what the actual requirement was.

If the requirement is, "Show the balance _as it was_ at that point in time", this system doesn't fulfil it. They even say so in the article: if something is wrong, throw away the state and re-run the events. That's necessarily different behaviour. To do this requirement, you actually have to audit every enquiry and say what you thought the result was, including the various errors/miscalculations.

If the requirement is, "Show the balance as it should have been at that point in time", then it's fine.

> They use running totals with audit trails, which is exactly what temporal tables provide.

In the author's case, they separate writes and reads into different DBs. The read-optimized DB has aggregated balances stored, not events. This is not materially different, and the trade-offs regarding staleness of data will be mostly the same.

Not only overkill, but error-prone. I had to suffer to work on a massive financial system based in serialized Python objects. And expensive as hell.

Related: https://vvvvalvalval.github.io/posts/2018-11-12-datomic-even...

I suspect there is a bit of knee-jerk because so often this pattern is misapplied. I actually quite like the example in the article although I'm basically allergic to CQRS in general.

I think your point about write-ahead logging etc is a good one. If you need a decent transactional system, you're probably using a system with some kind of WAL. If you're event sourcing and putting events into something which already implements a WAL, you need to give your head a wobble - why is the same thing being implemented twice? There can be great reasons, but I've seen (a few times) people using a perfectly fine transactional DB of some kind to implement an event store, effectively throwing away all the guarantees of the system underneath.

> Not sure why there is so much hate on this thread.

1) "Kafka is resume-driven-development" is a meme.

2) Devs are in denial about being in a distributed system, and think that single-threaded thinking (in proximity to a DB that calls itself ACID) leads to correct results in a distributed setting.

Isn't it simple? Create events from past relational data and add them to the log?

We are in a similar situation, and here's what I did:

1) Learned the basic concepts of double-entry bookkeeping. 2) Told ChatGPT about my business domain and requested an example Chart of Accounts (CoA) tailored to it.

Feel free to reach out to me, I’d love to exchange ideas.