I Solved a Distributed Queue Problem After 15 Years

Posted4 months agoActive4 months ago

jedberg

39 points

19 comments

dbos.devTechstory

calmmixed

Debate

60/100

Distributed SystemsQueueing SystemsDatabase Scalability

Key topics

Distributed Systems

Queueing Systems

Database Scalability

The author describes solving a distributed queue problem using durable queues, sparking a discussion on the trade-offs between using OLTP for queueing versus specialized systems like Kafka.

Snapshot generated from the HN discussion

Discussion Activity

Active discussion

First comment

Peak period

84-96h

Avg / period

4.8

Comment distribution19 data points

Loading chart...

Based on 19 loaded comments

Key moments

01Story posted
Sep 4, 2025 at 1:56 PM EDT
4 months ago
Step 01
02First comment
Sep 8, 2025 at 12:05 AM EDT
3d after posting
Step 02
03Peak activity
12 comments in 84-96h
Hottest window of the conversation
Step 03
04Latest activity
Sep 9, 2025 at 2:53 PM EDT
4 months ago
Step 04

Generating AI Summary...

Analyzing up to 500 comments to identify key contributors and discussion patterns

Discussion (19 comments)

Showing 19 comments

noident

4 months ago

1 reply

Hi, OP. Any thoughts on why to prefer using your OLTP for queueing as opposed to Kafka? In my mind it would be a drop in ease of observability (though there is still KSQL and a number of UI wrappers) in exchange for much better performance. I'm interested if you had other reasons.

KraftyOne

4 months ago

Observability is a big advantage, another advantage (in the context of DBOS specifically) is integration with durable workflows, so you can write a large workflow that enqueues and manages many smaller tasks.

belZaah

4 months ago

4 replies

I was architecture team lead at Skype 2005-2011 and persistent queues were the only ones around. Basically, because they knew how to scale databases, our DBA team (Hannu Krosing comes to mind) built queues into and on top of Postgres. It happened not just at Skype, too: eBay guys (Dan Prichett in particular) had built a very cool Oracle-based queue solution. Scaling persistent queues seems to be something that needs to be reinvented periodically. Maybe it’s too nuanced of a problem?

jedbergAuthor

4 months ago

2 replies

I actually worked on security for Skype in 2005/2006. And I knew Dan at eBay too.

Using databases for queues isn't a new idea, but a couple things make it different now. One is that I think this is the first time a solution has been open-sourced (not 100% sure on that).

And two, Postgres added SKIP LOCKED in 9.5 (around 2016), which was the performance unlock to make this work without needing a whole DBA team like Skype had.

arthurcolle

4 months ago

1 reply

is this referring to Dan Tarman?

sshine

4 months ago

GP wrote Dan Prichett.

fmajid

4 months ago

Skytools/PgQ was amazing technology that made clever use of Postgres MVCC to provide a high performance transactional message queue. But yes, it was not low-touch to administer.

asplake

4 months ago

1 reply

I know from firsthand experience that there were investment banks doing this at least as early as the mid 1990s - on top of Sybase in the particular case I have in mind. Perhaps not the best performance, but it was trivially easy to inspect, and you could integrate with other database features, for example stored procedures.

fmajid

4 months ago

Back when I worked for France Telecom circa 1996–99, Sybase Replication Server was the fastest message queue we tested for transporting billing events, by a wide margin, in a way that predated Postgres logical replication by 3 decades.

danieldc

4 months ago

1 reply

FWIW, SQL Server had built in queuing support from ~2005. The feature is called Service Broker (https://learn.microsoft.com/en-us/sql/database-engine/servic...)

fmajid

4 months ago

Much longer than that, the legendary Jim Gray developed MSMQ aka Falcon for them.

BSVogler

4 months ago

This article resonated with my experience building what was essentially a distributed task queue using Redis+PostgreSQL with Python workers in Kubernetes. It seems like these systems naturally evolve different patterns based on their specific use cases. The logic of our queue was intertwined with a rule engine. I wrote about building a rule engine here [1]. Another difference to this article is that it did not report back to the client as the events were delivered via web hooks.

There are some different approaches here and there which come from making it application specific, e.g. we added a periodic reconciliation check. I also built a debouncer into the queue to give special treatment to burst in the load.

[1] https://blog.benediktsvogler.com/blog/building-a-distributed...

na4ma4

4 months ago

1 reply

The article is interesting, but what does the title have to do with it ?

> What we really needed to make distributed task queueing robust are durable queues ...

> Durable queues were rare when I was at Reddit, but they’re more and more popular now.

It sounds like the answer was known at the time but there wasn't the resources to solve it ?

jedbergAuthor

4 months ago

Back then, you couldn't build a performant queue in a database without a huge number of resources, both hardware wise and people wise. That's why only huge enterprises were using them then.

Only recently (last decade or so), has the performance of an open source database on modest hardware caught up to the alternatives.

redditor98654

4 months ago

2 replies

From what I recall, Reddit uses AWS extensively. Could they not have replaced RabbitMQ with SQS? You get the near unlimited horizontal scalability, extremely good uptime, guaranteed at least once message delivery and for the case of a worker crash, the messages will become visible again after the visibility timeout (since they wouldn’t have been deleted by the worker).

4ndrewl

4 months ago

I think there is a hard limit on the number of in-flight requests (that is items that have been dequeued by a worker, but whose job has not been completed). I wouldn't be surprised if Reddit hit those sorts of volumes.

jedbergAuthor

4 months ago

SQS could not handle the volume or latency requirements we had, and it was too expensive compared to running it ourselves at the time.

yencabulator

4 months ago

Utterly content-free marketing blogspam.

_1tan

4 months ago

We're a MySQL/Hibernate shop. Anything similar possible with that stack?

View full discussion on Hacker News

ID: 45130143Type: storyLast synced: 11/20/2025, 4:44:33 PM

Want the full context?