Haydex: From Zero to 178.6b Rows a Second in 30 Days
Posted3 months agoActive3 months ago
axiom.coTechstory
skepticalmixed
Debate
60/100
Database PerformanceData ProcessingAlgorithm Optimization
Key topics
Database Performance
Data Processing
Algorithm Optimization
The article discusses how Haydex achieved 178.6B rows per second in 30 days, but the discussion revolves around understanding the actual achievement and the techniques used.
Snapshot generated from the HN discussion
Discussion Activity
Active discussionFirst comment
53m
Peak period
14
0-6h
Avg / period
5
Comment distribution20 data points
Loading chart...
Based on 20 loaded comments
Key moments
- 01Story posted
Sep 25, 2025 at 2:07 PM EDT
3 months ago
Step 01 - 02First comment
Sep 25, 2025 at 3:00 PM EDT
53m after posting
Step 02 - 03Peak activity
14 comments in 0-6h
Hottest window of the conversation
Step 03 - 04Latest activity
Sep 28, 2025 at 5:08 PM EDT
3 months ago
Step 04
Generating AI Summary...
Analyzing up to 500 comments to identify key contributors and discussion patterns
ID: 45376559Type: storyLast synced: 11/20/2025, 2:27:16 PM
Want the full context?
Jump to the original sources
Read the primary article or dive into the live Hacker News thread when you're ready.
They did not process 178 billion rows per second. They did a search that found something in a large data set by eliminating the parts of the data set that could not have contained the item. Same way I did by picking one grocery store and going straight to the shelf.
So, the analogy doesn't really hold true unless you actually have these trillions of alternate products stored in your brain and manage to cite the matching subset on demand.
If I have 10 billion rows in an SQL database, with a UNIQUE index, and do SELECT * FROM t1 WHERE pk=<number>, then I have “processed” 10 billion rows.
If I do 10k of these queries per second, I have processed 100T rows per second.
It's not just a tech blog post - it's a thriller. ;)
https://en.wikipedia.org/wiki/Bloom_filter
In the 80's or so when I thought I was being really clever I came up with another variation on this and I recall being quite annoyed when someone on HN pointed out (many years later) that this was a staple of computing science for longer than that I had been busy with computers. So much for having original thoughts...