An Mvcc-Like Columnar Table on S3 with Constant-Time Deletes

Posted3 months agoActive3 months ago

shayonj

45 points

9 comments

shayon.devTechstory

calmpositive

Debate

20/100

Database DesignCloud StorageMvcc

Key topics

Database Design

Cloud Storage

Mvcc

The post discusses a novel columnar table design on S3 that achieves constant-time deletes using an MVCC-like approach, sparking discussion on its potential applications and trade-offs.

Snapshot generated from the HN discussion

Discussion Activity

Light discussion

First comment

Peak period

Day 5

Avg / period

Key moments

01Story posted
Oct 6, 2025 at 12:33 PM EDT
3 months ago
Step 01
02First comment
Oct 10, 2025 at 9:49 AM EDT
4d after posting
Step 02
03Peak activity
5 comments in Day 5
Hottest window of the conversation
Step 03
04Latest activity
Oct 15, 2025 at 6:24 PM EDT
3 months ago
Step 04

Generating AI Summary...

Analyzing up to 500 comments to identify key contributors and discussion patterns

Discussion (9 comments)

Showing 9 comments

simlevesque

3 months ago

2 replies

I just want to be able to append metadata to a Parquet file at the end without rewriting the whole file. Tombstones could be baked in the parquet file this way.

It does work with "one more file" but it's not good for performance.

shayonjAuthor

3 months ago

1 reply

Yeah. Or just sub out the data with null bytes. Something like that could be nice too.

simlevesque

3 months ago

1 reply

Are you familiar with Parquet ? you can't do that at all, you need to rewrite the whole file.

shayonjAuthor

3 months ago

Yeah , I poorly phrased it - I meant in an ideal situation with the benefits of parquet like columnar file structure. I very much understand that it’s not possible on parquet today for the reasons you mentioned and others.

cpard

3 months ago

That’s the whole reason of existence of Iceberg, Delta and Hudi right?

Not as easy as just appending metadata to a parquet file but in the other hand, parquet was never and probably shouldn’t be designed with that functionality in mind.

simonw

3 months ago

2 replies

This is a really clever design.

The cost estimates are particularly notable: if they're right that's a cost of about $3/day for 6TB/day of written data, 2TB/day of deletes and 50K read queries.

Storing all those TBs of data in S3 is where the real cost lies. I think it costs $5520 to store 8TB*30 = 240TB in S3, and if you retain all data your monthly cost goes up by $5520 every month.

xyzzy_plugh

3 months ago

1 reply

I think the idea is that the deletes would eventually be compacted, so it's ultimately half as much, but I digress.

The cost isn't that bad all things considered. Hot, durable and available data ain't that cheap, especially in the cloud. Self-hosting is within an order of magnitude.

shayonjAuthor

3 months ago

I think ideally you could map retention of cold data to file objects itself and using key space naming strategy and lifecycle rules, expire the data that is not needed, thus saving on the storage costs (as much as possible hopefully)

shayonjAuthor

3 months ago

Here is another take on deletes through by just updating the row groups in Parquet file through multi part upload and UploadPartCopy - https://www.shayon.dev/post/2025/285/mutable-atomic-deletes-...

View full discussion on Hacker News

ID: 45493158Type: storyLast synced: 11/20/2025, 2:40:40 PM

Want the full context?

Jump to the original sources

Read the primary article or dive into the live Hacker News thread when you're ready.

Open link View on HN