An Adventure in Writing Compatible Systems

Posted4 months agoActive4 months ago

Original: An adventure in writing compatible systems

ezekg

47 points

12 comments

turso.techTech Discussionstory

informativepositive

Debate

20/100

System DesignCompatibilityProgramming

Key topics

System Design

Compatibility

Programming

Diving into the complexities of writing compatible systems, a developer's journey to recreate SQLite revealed a critical bug that only surfaced after hitting the 1 GB mark due to a lock-page at that exact offset. Commenters were surprised it took the team so long to inspect the source code, with some sarcastically praising the developer's superhuman abilities for doing so. Suggestions poured in on how to catch such issues in the future, including automated tests that randomly perform database operations and switch between implementations. The discussion also touched on the challenges of rewriting established, rock-solid software while striving for full compatibility.

Snapshot generated from the HN discussion

Discussion Activity

Moderate engagement

First comment

Peak period

84-96h

Avg / period

Comment distribution12 data points

Loading chart...

Based on 12 loaded comments

Key moments

01Story posted
Aug 28, 2025 at 11:38 PM EDT
4 months ago
Step 01
02First comment
Sep 1, 2025 at 1:43 PM EDT
4d after posting
Step 02
03Peak activity
7 comments in 84-96h
Hottest window of the conversation
Step 03
04Latest activity
Sep 2, 2025 at 8:36 AM EDT
4 months ago
Step 04

Generating AI Summary...

Analyzing up to 500 comments to identify key contributors and discussion patterns

Discussion (12 comments)

Showing 12 comments

ozgrakkurt

4 months ago

1 reply

Curious if having the automated tests do db operations on a persistent db would catch this.

It would do inserts/deletes/updates/create_table/alter_column etc. randomly for a long time, continue even if an operation fails unless there is a crash. This way the db could grow without bound and queries can get very slow

swiftcoder

4 months ago

1 reply

It only catches it if you re-load the DB into sqlite after every write by turso. At which point you'd flag pretty quickly that it was the next write after 1 GB that messed things up

ozgrakkurt

4 months ago

Could also make switching an operation. So it randomly switches between implementations and keeps using it until the next switch. And can keep running two instances so it is always kind of cross checking

soanvig

4 months ago

2 replies

I wonder why it took so much time among entire team to actually look into the source code. That's probably third thing I would do after encountering that kind of issue. I would even look for a word "gigabyte" considering it is well documented, because bizarre things should be documented.

petcat

4 months ago

1 reply

Yeah. Is their developer super human because he...reads the source code?? This should be obvious.

Now I would be genuinely worried about what other issues are lurking in their rewrite.

yencabulator

4 months ago

Previously in this saga: https://turso.tech/blog/a-deep-look-into-our-new-massive-mul...

They broke SQLite compatibility even before they started writing in Rust: https://news.ycombinator.com/item?id=42386894

Joker_vD

4 months ago

The worst thing is, it's also documented in the SQLite's documentation. The file format is documented, and it describes the existence of this lock-page located at 1 GiB mark. How they've managed to implement support for SQLite DB file format without reading the file format description is, frankly, beyond me.

rawxtl

4 months ago

1 reply

I don't even understand the point in rewriting something that is rock solid. And on top of that you guys are asking for full compatibility. This is just nonsense.

vendiddy

4 months ago

I think it's a valid question, but it's better to assume they had their reasons and try to understand why before drawing conclusions.

Joker_vD

4 months ago

1 reply

This. This exact bloody attitude of people who claim "we want to write a compatible system" but can not be bothered to read — not even the source code — the freaking docs, freely available, is why we can't have any decently compatible with each other systems nowadays. There, look: [0], it's explicitly documented right on the SQLite's "Database File Format" page, and has been for a decade at least. How the fuck did you manage to write a clone of SQLite without reading the description of its file format? Like, seriously, how?

[0] https://sqlite.org/fileformat2.html#the_lock_byte_page

nesarkvechnep

4 months ago

We live in a SOCIETY!!!

yencabulator

4 months ago

> To the point that we have a challenge [...]: if anyone can find a bug that leads to data corruption *and improve our simulator to catch it next time*, we will pay you a cash prize.

Uhh, you should be paying people if you get them to work for you.

View full discussion on Hacker News

ID: 45059888Type: storyLast synced: 11/20/2025, 6:27:41 PM

Want the full context?

Jump to the original sources

Read the primary article or dive into the live Hacker News thread when you're ready.

Open link View on HN