I Built Binary Serializer for Pydantic Models That's Up to 7× Smaller Than JSON

Posted2 months agoActive2 months ago

sijokun

2 points

5 comments

github.comTechstory

supportivepositive

Debate

20/100

PydanticSerializationBinary Data

Key topics

Pydantic

Serialization

Binary Data

The author presents PyByntic, a binary serializer for Pydantic models that outperforms JSON in size, sparking interest and discussion about its potential applications and comparisons to existing solutions.

Snapshot generated from the HN discussion

Discussion Activity

Light discussion

First comment

N/A

Peak period

0-2h

Avg / period

2.5

Key moments

01Story posted
Oct 25, 2025 at 4:29 AM EDT
2 months ago
Step 01
02First comment
Oct 25, 2025 at 4:29 AM EDT
0s after posting
Step 02
03Peak activity
4 comments in 0-2h
Hottest window of the conversation
Step 03
04Latest activity
Oct 26, 2025 at 5:28 AM EDT
2 months ago
Step 04

Generating AI Summary...

Analyzing up to 500 comments to identify key contributors and discussion patterns

Discussion (5 comments)

Showing 5 comments

sijokunAuthor

2 months ago

1 reply

I ran into memory issues with a high-load project so I built a compact binary encoder/decoder for Pydantic models. It cut in-RAM object size by up to 7× vs json.dumps(), and ended up saving the whole service from collapsing.

GitHub: https://github.com/sijokun/PyByntic

Works with annotated Pydantic models and gives you: – .serialize() -> bytes – .deserialize(bytes) -> Model

Curious to hear whether others here have hit similar problems and how you solved it?

P.S. Project was a Telegram MiniApp with 10m+ MAU, we were storing cached user objects in Redis Cluster

Locutus_

2 months ago

2 replies

I've had exactly the same situation, ~2M MAU service with REDIS as the only persistence system, all data being JSON serialized Pydantic models. The storage overhead was just terrible and cost real money.

This would have been a super nice to have back then.

I wonder though how much sense it would make to get something like this mainlined into upstream Pydantic? as having this downstream would give many continuity and dependency lock concerns. And having it as part of the main library would significantly drive adoption rate.

sijokunAuthor

2 months ago

1 reply

Found in issue where maintainer answered that they are only going to support JSON

https://github.com/pydantic/pydantic/discussions/4097

Locutus_

2 months ago

The answer honestly feels a bit lacking, and evasive.

sijokunAuthor

2 months ago

Thanks, your Redis story sounds exactly like my motivation. When the cache layer becomes the primary persistence and you’re pushing around millions of JSON-encoded Pydantic objects, the overhead hits both RAM and the bill very quickly. That pain is real.

Regarding getting this into Pydantic core: I doubt the maintainers would want to ship and maintain a binary serialization layer inside the main project. That said, it might still be worth opening an issue to hear their perspective - worst case, it becomes a documented "won’t include".

As for continuity and lock-in concerns - my primary production workload depends on this library, so I fully intend to keep it updated alongside new Pydantic and Python releases. I also intentionally designed it with minimal dependencies: core functionality uses only pydantic and typing-extensions. The only extra is leb128, and that’s just for UInt128, which almost nobody truly needs - I’m considering moving that to an optional extra to keep the base install even leaner.

View full discussion on Hacker News

ID: 45702216Type: storyLast synced: 11/17/2025, 8:03:05 AM

Want the full context?

Jump to the original sources

Read the primary article or dive into the live Hacker News thread when you're ready.

Open link View on HN