Matrix.org Service Offline: Corrupted Database
Posted4 months agoActive4 months ago
status.matrix.orgTechstory
calmmixed
Debate
40/100
MatrixFederated ServicesDecentralized Communication
Key topics
Matrix
Federated Services
Decentralized Communication
The Matrix.org service experienced an outage due to a corrupted database, sparking discussions about the usability and resilience of Matrix and its clients, such as Element.
Snapshot generated from the HN discussion
Discussion Activity
Light discussionFirst comment
11h
Peak period
3
15-18h
Avg / period
2
Comment distribution14 data points
Loading chart...
Based on 14 loaded comments
Key moments
- 01Story posted
Sep 2, 2025 at 3:48 PM EDT
4 months ago
Step 01 - 02First comment
Sep 3, 2025 at 2:37 AM EDT
11h after posting
Step 02 - 03Peak activity
3 comments in 15-18h
Hottest window of the conversation
Step 03 - 04Latest activity
Sep 4, 2025 at 6:25 PM EDT
4 months ago
Step 04
Generating AI Summary...
Analyzing up to 500 comments to identify key contributors and discussion patterns
ID: 45108131Type: storyLast synced: 11/20/2025, 4:38:28 PM
Want the full context?
Jump to the original sources
Read the primary article or dive into the live Hacker News thread when you're ready.
I turn it off and use Discord again. I know they are fundamentally distinct, but I am not willing to trade in ease of use (which is really REALLY awful) for privacy, and certainly not trading anything for having a distributed system.
You can't even do direct file transfer. There's a feature request open for YEARS... Yet nobody bothers to push it through.
I feel like that’s completely stopped.
For example, I’m still waiting for reasonable documentation to be published about how to deploy and use Element Call, which has apparently been generally available for over a year.
Fwiw, the DB isn't corrupted - the database 2ndary dropped its RAID array on having new disks added (the hw raid controller incorrectly added them into the array, breaking it)... and a few hours later we lost the primary db too. The outage is caused by the time taken to restore & rebuild a 55T db from nightly snapshot.
In terms of lack of documentation for running Element Call: i published a tutorial & video run-through myself back in November: https://element.io/blog/experimenting-with-matrix-2-0-using-... and https://github.com/element-hq/element-docker-demo and https://youtu.be/6iMi5BiQcoI. Or you could just run it via Element Server Suite: https://element.io/server-suite/community
In terms of "Element X has a 10th of the functionality of classic Element" - with respect, this is bullshit. The only features folks complain about missing are Threads & Spaces, both of which are have implementations behind feature flags and will land shortly. In all other respects Element X is a wild improvement over classic Element.
Fwiw, there's another HN thread on this over at https://news.ycombinator.com/item?id=45107696
... oof.
You really, truly should look at ZFS.
What happened and why? Any pointers to read more on this?
Perhaps the network of servers could even have some redundancy for the last two weeks history of chat messages, minus images.
It's entirely just user accounts that are tied to a homeserver currently. There's a proposal to make it possible for clients to fully manage their account identity (https://github.com/matrix-org/matrix-spec-proposals/pull/408...) but that doesn't look particularly active.