Bcachefs Goes to "Externally Maintained"
Key topics
The Linux kernel's bcachefs file system has been relegated to "Externally Maintained" status, sparking debate about the implications for users and developers. Commenters weighed in on the potential consequences, with some lamenting the loss of a kernel-maintained file system and others pointing out that DKMS (Dynamic Kernel Module Support) isn't the only way to distribute out-of-tree modules. A consensus emerged that the change might be more about managing the mercurial creator of bcachefs, Kent, and his sometimes prickly interactions with other kernel developers, rather than any technical issues with the file system itself. As one commenter noted, Kent's history of "drama" often involves clashing with rigid workflows and environments, suggesting that his approach may be at the heart of the issue.
Snapshot generated from the HN discussion
Discussion Activity
Very active discussionFirst comment
4h
Peak period
123
Day 1
Avg / period
22.9
Based on 160 loaded comments
Key moments
- 01Story posted
Aug 30, 2025 at 9:07 AM EDT
4 months ago
Step 01 - 02First comment
Aug 30, 2025 at 1:08 PM EDT
4h after posting
Step 02 - 03Peak activity
123 comments in Day 1
Hottest window of the conversation
Step 03 - 04Latest activity
Sep 13, 2025 at 2:01 AM EDT
4 months ago
Step 04
Generating AI Summary...
Analyzing up to 500 comments to identify key contributors and discussion patterns
Want the full context?
Jump to the original sources
Read the primary article or dive into the live Hacker News thread when you're ready.
Your distro could very easily include bcachefs if it wishes? Although I think the ZFS + Linux situation is mostly Linux religiosity gone wild, that very particular problem doesn't exist re: bcachefs?
The problem with bcachefs is the problem with btrfs. It mostly still doesn't work to solve the problems ZFS already solves.
I can think of non-religious reasons to want to avoid legal fights with Oracle.
Oh, certainly, but that's really not the problem posed right now. Oracle's own lawyers have said they see no problem with the combination, and Ubuntu has shipped the Linux + ZFS combination for years without a lawsuit.[0][1]
In the 1990s, Microsoft/SCO, like you, would also fear monger about open source and lawsuits, and we in the OSS community mostly called this "FUD" (fear, uncertainty, and doubt). Whereas now, almost 10 years into this experiment, we know more about the ZFS + Linux combination, and what Oracle will do about it, than most other open questions in OSS, and the answer is that some in OSS community have chosen to instead participate in the same kind of FUD because of a very online, very uninformed internecine licensing debate.
[0]: https://www.youtube.com/watch?v=PFMPjt_RgXA&t=2260s [1]: https://ubuntu.com/blog/zfs-is-the-fs-for-containers-in-ubun...
I think the Linux Kernel just doesn't want to be potentially in violation of Oracle's copyrights. That really doesn't seem that unreasonable to me, even if it feels pointless to you.
* Another kernel Dev takes over management and they tread it as a fork (highly unlikely according to their estimate)
* Kent hires someone to upstream the changes for him and Kent stops complaining wrt when it's getting merged
* Bcachefs gets no maintenance and will likely be removed in the next major release
I do not know him personally, but most interactions I've read online by him sounded grounded and not particularly offensive, so I'm abstaining from making any kind of judgement on it.
But while I have no stake in this, Drama really does seem to follow Kent around for one reason or another. And it's never his fault if you take him by his public statements - which I want to repeat: he sounds very grounded and not offensive to me whatsoever.
https://lore.kernel.org/lkml/CAHk-=wiLE9BkSiq8F-mFW5NOtPzYrt...
https://lore.kernel.org/all/citv2v6f33hoidq75xd2spaqxf7nl5wb...
The second has one offensive remark:
> Get your head examined. And get the fuck out of here with this shit.
which I thought he admitted was out of line and - said sorry for. Or do I misremember? I admit once again, I'm still completely uninvolved and merely saw it play out on the internet.
I had high hopes for bcachefs. sigh
He’s not super offensive, but he will tell a Debian package maintainer that their process sucks, and the should change it and they are being stupid by following that process. Overall, he seems a bit entitled, and unwilling to compromise with others. It’s not just Kent though, the areas that seem to be the most problematic for him, are when it’s an unstoppable force (Kent), and an immovable wall (Linux / Debian).
Working in the Linux kernel is well known for its frustrations and the personal conflict that it creates, to the point that there are almost no linux kernel devs/maintainers that aren’t paid to do the work. You can see a similar set of events happen with Rust4Linux people, Asahi linux project and their R4L drivers, etc.
Both Linus and Kent drive a hard bargain, and it's not as simple as finding someone else to blindly forward bcachefs patches. At the first sign of conflict, the poor person in the middle would have no power, no way to make anyone back down, and we'd be back to square one.
It's in limbo, and there is still time, but if left to bitrot it will be removed eventually.
The issue is that I have never seen Kent back down a single time. Kent will explain in details why the rules are bullshit and don't apply in this particular case, every single time, without any room for compromise.
If the only problem was when to send patches, that would be one thing. But disagreements over patches aren't just a timing problem that can be routed around.
The point of contention here was a patch within fs/bcachefs/, which was repair code to make sure users didn't lose data.
If we can't have clear boundaries and delineations of responsibility, there really is no future for bcachefs in the kernel; my core mission is a rock solid commitment to reliability and robustness, including being responsive to issues users hit, and we've seen repeatedly that the kernel process does not share those priorities.
To some extent drawing clear boundaries is good as a last resort when people cannot agree, but it can't be the main way to resolve disagreements. Thinking in terms of who owns what and has the final say is not the same as trying to understand the requirements from the other side to find a solution that works for everyone.
I don't think the right answer is to blindly follow whatever Linus or other people say. I don't mean you should automatically back down without technical reasons, because authority says so. But I notice I can't remember an email where concessions where made, or attemps to find a middle grounds by understanding the other side. Maybe someone can find counterexamples.
But this idea of using ownership to decide who has more authority and can impose their vision, that can't be the only way to collaborate. It really is uncompromising.
Agreed 100%. In an ideal world, we'd be sitting down together, figuring out what our shared priorities are, and working from there.
Unfortunately, that hasn't been possible, and I have no idea what Linus's priorities except that they definitely aren't a bulletproof filesystem and safeguarding user data; his response to journal_rewind demonstrated that quite definitively.
So that's where we're at, and given the history with other local filesystems I think I have good reason not to concede. I don't want to see bcachefs run off the rails, but given all the times I've talked about process and the way I'm doing things I think that's exactly what would happen if I started conceding on these points. It's my life's work, after all.
You'd think bcachefs's track record (e.g. bug tracker, syzbot) and the response it gets from users would be enough, but apparently not, sadly. But given the way the kernel burns people out and outright ejects them, not too surprising.
Remarks like this come across as extremely patronizing, as you completely ignore what the other party says and instead project your own conclusions about the other persons motives and beliefs.
> his response to journal_rewind demonstrated that quite definitively
No, no it did not in any shape way or form do that. You had multiple other perfectly valid options to help the affected users besides getting that code shipped in the kernel there and then. Getting it shipped in the kernel was merely a convenience.
If bcachefs was established and stable it would be a different matter. But it's an experimental file system. Per definition data loss is to be expected, even if recovery is preferable.
If we had the fuse driver done that would have worked, though. Still not completely ideal because we're at the mercy of distros to make sure they're getting -tools updates out in a timely manner, they're not always as consistent with that as the kernel. Most are good, though).
Just making it available in a git repo was not an option because lots of bcachefs users are getting it from their distro kernel and have never built a kernel before (yes, I've had to help users with building kernels for the first time; it's slow and we always look for other options), and even if you know how, if your primary machine is offline the last thing you want to have to do is build a custom rescue image with a custom kernel.
And there was really nothing special about this than any other bugfix, besides needing to use a new option (which is also something that occasionally happens with hotfixes).
Bugs are just a fact of life, every filesystem has bugs and occasionally has to get hotfixes out quickly. It's just not remotely feasible or sane to be coming up with our own parallel release process for hotfixes.
I think there's room to have your cake and eat it too, but I certainly can't blame you for caring about quality, that much is sure.
Has he ever even been involved with a bcachefs bug? No, aside from arguing against shipping bugfixes.
Has he contributed in any way, besides merging code? No...
Has he set rules or guidelines that benefited bcachefs reliability? No, but he has shouted down talk about automated testing.
I think you're confusing power with responsibility.
You are one of those maintainers (not any more). Your code can be taken into the bundle (not any more), but on the bundle's schedule, not yours. You have consistently failed to understand that the train doesn't wait for you - if you are late, you get on the next one. If you don't want to get on the next one, then don't be late. Normal people, after missing a train once or twice, would adjust their schedule accordingly so they won't miss it next time. But your exclusive, repeated reaction has been to yell at the train driver and the station master, which is why you've been kicked out of the station.
Have you ever ridden a train, by the way? Were you on time? (Deutsche Bahn doesn't count because they're not on time)
https://github.com/chimera-linux/ckms
Also see https://www.phoronix.com/news/Josef-Bacik-Leaves-Meta
It's important to note that striping and mirroring works just fine. It's only the 5/6 modes that are unstable: https://btrfs.readthedocs.io/en/stable/Status.html#block-gro...
Ad hominem. My thinkpad ssd is massive.
How can this be a stable filesystem if parity is unstable and risks data loss?
How has this been allowed to happen?
It just seems so profoundly unserious to me.
In practice RAIDZ2 works great.
The fact that btrfs isn't stable after 15+ years for parity setups is, IMO, unreasonable.
I also have had to deal with thousands of nodes kernel panicing due to a btrfs bug in linux kernel 6.8 (stable ubuntu release).
The md metadata is not adequately protected. Btrfs checksums can tell you when a file has gone bad but not self-heal. And I'm sure there are going to be caching/perf benefits left on the table not having btrfs manage all the block storage itself.
We are using a fairly simple config, but under certain heavy load patterns the kernel would panic: https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux...
I hear people say all the time how btrfs is stable now and people are just complaining about issues when btrfs is new, but please explain to me how the bug I linked is OK in a stable version of the most popular linux distro?
The man page for mkfs.btrfs says:
> Warning: RAID5/6 has known problems and should not be used in production.
When you actually tell it to use raid5 or raid6, mkfs.btrfs will also print a large warning:
> WARNING: RAID5/6 support has known problems is strongly discouraged to be used besides testing or evaluation.
Single, dup, raid0, raid1, raid10 have been usable and stable for a decade or more.
Disaster recovery isn't obvious on any setup I've worked with. I have to RTFM to understand each system's idiosyncrasies.
The idea that some filesystems have no bugs is absurd. The idea that filesystems can mitigate all bugs in drive firmware or elsewhere in the storage stack is also absurd.
My anecdota: hundreds of intentional sabotaging of Btrfs while writing, single drive and raid1 configurations, and physically disconnecting a drive. Not one time have I encountered an inconsistent filesystem or data loss once it was on stable media. Not one. It always mounted without needing a filesystem check. This is on consumer hardware.
There's always some data loss in the single drive case no matter the filesystem. Some of the data or fs metadata isn't yet persistent. Raid1 helps with that, because so long as the hardware problem that affected the 1st drive is isolated, the data is written to stable media.
Of course, I'm no more a scientific sample than you are. And also my tests are rudimentary compared to the many thousands of synthetic tests fstests performs on Linux filesystems, both generic and fs specific, every cycle. But it is a real world test, and suggests no per se problem that inevitably means data loss as you describe.
All the anecdotes I see tend to be “my drive didn’t mount, and I tried nothing before giving up because everyone knows BTRFS sux lol”. My professional experience meanwhile is that I’ve never once been able to not (very easily!) recover a BTRFS drive someone else has given up for dead… just by running its standard recovery tools.
Now you've piqued my curiosity; what uses that many filesystems/subvolumes? (Not an attack; I believe you, I'm just trying to figure out where it comes up)
The main criticism in this thread about btrfs involves multidisk setups, which aren't relevant for me, since I'm working on cloud systems and disk storage is abstracted away as a single block device.
Also, the out-of-band deduplication for btrfs using https://github.com/Zygo/bees is very impressive and flexible, in a way that ZFS just doesn't match.
*caveat: I’m using RAID 10, not a parity RAID. It could have problems with parity RAID. So? If you really really want RAID 5, then just use md to make your RAID 5 device and put btrfs on top.
https://news.ycombinator.com/item?id=42221564 - 2024-11-23, 103 comments
I don’t doubt that people on all sides have made mis-steps, but from the outside it mostly just seems like Kent doesn’t want to play by the rules (despite having been given years of patience).
I agree that the kernel community can be a hostile environment.
Though I’d argue that people _have_ tried to explain things to Kent, multiple times. At least a few have been calm, respectful attempts.
Sadly, Kent responds to everything in an email except the key part that is being pointed out to him (usually his behavior). Or deflects by going on the attack. And generally refuses to apologise.
Behaviour sounds like the least important part of code contributions. I smell overpowered, should've-been-a-kindergarten-teacher code of conduct person overreach.
CoC isn't even the issue, he constantly breaks kernel development rules relating to the actual code, then starts arguments with everyone up to and including Linus when he gets called out, and aggressively misses the point every time. Then starts the same argument all over again 6 weeks later.
And, like, if you don't like some rules, then you can have that discussion, but submitting patches you know will be rejected and then re-litigating your dislike of the rules is a waste of everyone's time.
[0] https://lore.kernel.org/lkml/6740fc3aabec0_5eb129497@dwillia...
Example of eye-rolling post, above:
> Sadly, Kent responds to everything in an email except the key part that is being pointed out to him (usually his behavior). Or deflects by going on the attack. And generally refuses to apologise.
And there's an email thread linked somewhere here where a CoC member repeatedly replies to Kent's emails with demands for a formal apology. All of this soft, subtle stuff adds up to an impression in people's heads, even though the main output of these projects should be highly complex software, and not bike-shedding email mediation.
I just think that while, yes, the kernel folks have tried to explain, they didn't explain well. The "why" of it is a people thing. Linus needs to be able to trust that people he's delegated some authority will respect its limits. The maintainers need to be able to trust that each other maintainer will respect the area that they have been delegated authority over. I think that Kent genuinely doesn't get this.
You can't win a rules-lawyer argument with the rulemaker.
unfortunately that's either due to lack of investigation by yourself or a bit dishonest.
Kent just does not listen. Every time the discussion starts from the top. Even if you do agree on some compromise, in a month or two he'll just do the same thing again and all the same arguments start again.
You can't expect people to detail about four or five years of context in every single engagement for the benefit of interested 3rd parties like you or me.
Kent seems very patient in explaining his position (and frustrations arising from other people introducing bugs to his code) and the kernel & debian folks are performing a smearing campaign instead of replying to what I see are genuine problems in the process. As an example, the quotes that are referenced by user paravoid are, imho, taken out of context (judging by reading the provided links).
There probably is a lot more history to it, but judging from that thread it's not Kent who looks like a bad guy.
(Debian's rules aren't worthless, it's part of how they can make something that's pretty suitable for 'boring infrastructure' systems because they can keep a system with a known and stable set of behavior up to date with critical security fixes for a long time, but boy do they result in some dumb situations sometimes)
This is one of the problems: Kent is frequently unable to accept that things don't go his way. He will keep bringing it up again and again and he just grinds people down with it. If you see just one bit of it then it may seem somewhat reasonable, but it's really not because this is the umpteenth time this exact discussion is happening and it's groundhog day once again.
This is a major reason why people burn out on Kent. You can't just have a disagreement/conflict and resolve it. Everything is a discussion with Kent. He can't just shrug and say "well, I think that's a bit silly, but okay, I can work with it, I guess". The options are 1) Kent gets his way, or 2) he will keep pushing it (not infrequently ignoring previous compromises, restarting the discussion from square one). Here too, the Debian people have this entire discussion (again) forced upon them by Kent's comments in a way that's just completely unnecessary and does nothing to resolve anything.
Even as an interested onlooker who is otherwise uninvolved and generally more willing to accept difficult behaviour than most people, I've rather soured on Kent over time.
And there's a real connection to the issue that sparked all this drama in the kernel and the Debian drama: critical system components (the kernel, the filesystem, and others) absolutely need to be able to get bugfixes in a timely manner. That's not optional.
With Debian, we had a package maintainer who decided that unbundling Rust dependencies was more important than getting out updates, and then we couldn't get a bugfix out for mount option handling. This was a non-issue for every other distro with working processes because the bug was fixed in a few days, but a lot of Debian users weren't able to mount in degraded mode and lost access to their filesystems.
In the kernel drama, Linus threw a fit over a repair code to recover from a serious bug and make sure users didn't lose data, and he's repeatedly picked fights over bugfixes (and even called pushing for getting bugfixes out "whining" in the past).
There are a lot of issues that there can be give and take on, but getting fixes out in a timely manner is just part of the baseline set of expectations for any serious project.
But there are also reasons why things are the way they are, and that is also not unreasonable. And at the end of the day: Linus is the boss. It really does come down to that. He has dozens of other subsystem maintainers to deal with and this is the process that works for him.
Similar stuff applies to Debian. Personally, I deeply dislike Debian's inflexible and outmoded policy and lack of pragmatism. But you know, the policy is the policy, and at some point you just need to accept that and work with it the best you can.
It's okay to make all the arguments you've made. It's okay to make them forcefully (within some limits of reason). It's not okay to keep repeating them again and again until everyone gets tired of it and seemingly just completely fail to listen to what people are sating. This is where you are being unreasonable.
I mean, you *can* do that, I guess, but look at where things are now. No one is happy with this – certainly not you. And it's really not a surprise, I already said this in November last year: "I wouldn't be surprised to see bcachefs removed from the kernel at some point".[1] To be clear: I didn't want that to happen – I think you've done great work with bcachefs and I really want it to succeed every which way. But everyone could see this coming from miles.
[1]: https://news.ycombinator.com/item?id=42225345
It is unreasonable if it leads to users losing data. At this point, the only reasonable thing is to either completely remove support for bcachefs or give timely fixes for critical bugs, there's no middle position that won't willfully lead to users losing their data.
This used to be the default for distributions like Debian some time ago. You only supported foundational software if you were willing to also distribute critical fixes in a timely manner. If not, why bother?
For all other issues, I guess we can accept that things are the way they are.
Changing the kernel development process to allow adding new features willy-nilly late in the RC cycle will lead to much worse things than a few people using an experimental file system losing their data in the long term.
The process exists for a reason, and the kernel is a massive project that includes more than just one file system, no matter how special its developers and users believe it is.
This blowup was entirely unnecessary.
And this is happening even though it's common for Debian to package the same C library multiple times, like, libfuse2 and libfuse3. This could be done for Rust libraries if they wanted to.
Anyway see the discussion and the relevant article here https://news.ycombinator.com/item?id=41407768 and https://jonathancarter.org/2024/08/29/orphaning-bcachefs-too...
And carrying multiple versions is problematic too as it causes increased burdens for the downstream maintainers.
I'd argue that libfuse is a bit of a special case since the API between 2 & 3 changed substantially, and not all dependencies have moved to version 3 (or can move, since if you move the v3 then you break on other platforms like BSD and macOS that still only support the v2 API).
Rust and especially Golang are both a massive pile of instability because the developers don't seem to understand that long term stable APIs are a benefit. You have to put in a bit of care and attention rather than always chasing the new thing and bundling everything.
XFS has burned through maintainers, citing "upstream burnout". It's not just bcachefs that things are broken for.
And it was burning me out, too. We need a functioning release process, and we haven't had that; instead I've been getting a ton of drama that's boiled over into the bcachefs community, oftentimes completely drowning out all the calmer, more technical conversations that we want.
It's not great. It would have been much better if this could have been worked out. But at this point, cutting ties with the kernel community and shipping as a DKMS module is really the only path forwards.
It's not the end of the world. Same with Debian; we haven't had those issues in any other distros, so eventually we'll get a better package maintainer who can work the process or they'll figure out that their Rust policy actually isn't as smart as they think it is as Rust adoption goes up.
I'm just going to push for doing things right, and if one route or option fails there's always others.
It's the people we meet along (and get along with) not the fucking data.
Cherish people who want to work with you, not one more byte saved.
Users are responsible for their own backups. If you want to be really responsible educate them. (As you already helped some of them to build their own kernel, you can nudge them to keep proper backups too!)
If you have problems with downstream being slow then offer workarounds.
How ridiculous is all this, really? Instead of hosting a shell script that sets up a PPA or a cron job with updates or whatever you try to brute force things through Debian and Linus?
If users are that important for you give them your phone number or whatever.
Seriously. Based on all of what you wrote you need to put effort into having a direct line to your users. (Which is completely fine nowadays. Discord and Twitch/YT office hours and whatnot are all the hype nowadays.)
Stop projecting your needs onto other maintainers.
> critical system components (the kernel, the filesystem, and others) absolutely need to be able to get bugfixes in a timely manner. That's not optional.
that's not how these projects are set up. (and even though your code is upstream to them, your users were (are) downstream to them. they were hosting your project for their users, they can (and did) decide to stop hosting your project.)
data integrity and data-loss-prevention are not considered security updates - where we have a culture of out of bad updates. (because the workaround for them is the standing order is to have working backups.)
So either Kent is on a righteous crusade against unreasonable processes within the Kernel, Debian, and every other large software project he interacts with. Or there's something about the way Kent interacts with these projects that causes friction.
I like Bcachefs, I think Kent is a very talented developer, but I'm not going to pretend that he is innocent in all this.
But the problem with comparisons is that even if you're better than nuclear waste being dumped into the aquifer, you still might be enough to light a river on fire.
However, sometimes, a certain detachment can help when looking at what is, in the end, a "cultural disagreement" more suited to an elementary school's playground.
Whenever I see open source spats like this, and then see a Dev harangued and chased from forum to forum by what looks like a coordinated group ("groupies"?), all accusing him/her of rude behaviour, while they keep making attacks on his/her personality, character or temperament...
it leads me to think rather poorly of this "wild west posse".
Anyway, bottom line, Kent is writing open source software to benefit others (and people didn't have qualms about taking his previous bcache & using it to build out storage solutions to make millions), so perhaps he doesn't quite deserve all the abuse and ganging up, no matter whose feathers he ruffled, and how.
IMHO, what his communications show is an unwillingness to acknowledge that other projects that include his work have focus, priorities, and policies that are not the same as that of his project. Also, expecting exceptions to be made for his case, since exceptions have been made in other cases.
Again IMHO, I think he would be better off developing apart with an announcement mailing list. When urgent changes are made, send to the announcement list. Let other interested parties sort out the process of getting those changes into the kernel and distributions.
If people come with bug reports from old versions distributed by others, let them know how to get the most up to date version from his repository, and maybe gently poke the distributors.
Yes, that means users will have older versions and not get fixes immediately. But what he's doing isn't working to get fixes to users immediately either.
Almost makes me think the distros light-forking it to just change the name (IceWeasel style) so the support requests don't get to him will help… probably not, though, because people will still go there because they want to recover their data.
There is no 'modern' ZFS-like fs in Linux nowadays.
zfs is out of tree leaving it as an unviable option for many people. This news means that bcachefs is going to be in a very weird state in-kernel, which leaves only btrfs as the only other in-tree ‘modern’ filesystem.
This news about bcachefs has ramifications about the state of ‘modern’ FSes in Linux, and I’d say this news about the btrfs maintainer taking a step back is related to this.
1. The dm layer gives you cow/snapshots for any filesystem you want already and has for more than a decade. Some implementations actually use it for clever trickery like updates, even. Anyone who has software requirements in this space (as distinct from "wants to yell on the internet about it") is very well served.
2. Compression seems silly in the modern world. Virtually everything is already compressed. To first approximation, every byte in persistent storage anywhere in the world is in a lossy media format. And the ones that aren't are in some other cooked format. The only workloads where you see significant use of losslessly-compressible data are in situations (databases) where you have app-managed storage performance (and who see little value from filesystem choice) or ones (software building, data science, ML training) where there's lots of ephemeral intermediate files being produced. And again those are usages where fancy filesystems are poorly deployed, you're going to throw it all away within hours to days anyway.
Filesystems are a solved problem. If ZFS disappeared from the world today... really who would even care? Only those of us still around trying to shout on the internet.
Yeah nah, have you tried processing terabytes of data every day and storing them? It gets better now with DDR5 but bit flips do actually happen.
And maybe below it.
And backups.
Backups make a lot of this minor.
You can certainly add verification above and below your filesystem, but the filesystem seems like a good layer to have verification. Capturing a checksum while writing and verifying it while reading seems appropriate; zfs scrub is a convenient way to check everything on a regular basis. Personally, my data feels important enough to make that level of effort, but not important enough to do anything else.
> Personally, my data feels important enough to make that level of effort, but not important enough to do anything else.
OMG. Backups! You need backups! Worry about polishing your geek cred once your data is on physically separate storage. Seriously, this is not a technology choice problem. Go to Amazon and buy an exfat stick, whatever. By far the most important thing you're ever going to do for your data is Back. It. Up.
Filesystem choice is, and I repeat, very much a yell-on-the-internet kind of thing. It makes you feel smart on HN. Backups to junky Chinese flash sticks are what are going to save you from losing data.
I wouldn't comment but I feel like I'm naturally on your side of the argument and want to see it articulated well.
My goal was actually the same though: to try to short-circuit the inevitable platform flame by calling it out explicitly and pointing out that the technical details are sort of a solved problem.
ZFS argumentation gets exhausting, and has ever since it was released. It ends up as a proxy for Sun vs. Linux, GNU vs. BSD, Apple vs. Google, hippy free software vs. corporate open source, pick your side. Everyone has an opinion, everyone thinks it's crucially important, and as a result of that hyperbole everyone ends up thinking that ZFS (dtrace gets a lot of the same treatment) is some kind of magically irreplaceable technology.
And... it's really not. Like I said above if it disappeared from the universe and everyone had to use dm/lvm for the actual problems they need to solve with storage management[1], no one would really care.
[1] Itself an increasingly vanishing problem area! I mean, at scale and at the performance limit, virtually everything lives behind a cloud-adjacent API barrier these days, and the backends there worry much more about driver and hardware complexity than they do about mere "filesystems". Dithering about individual files on individual systems in the professional world is mostly limited to optimizing boot and update time on client OSes. And outside the professional world it's a bunch of us nerds trying to optimize our movie collections on local networks; realistically we could be doing that on something as awful NTFS if we had to.
But I don't usually verify the backups, so there's that. And everything is in the same zip code for the most part, so one big disaster and I'll lose everything. C'est la vie.
https://wiki.archlinux.org/title/Dm-integrity
> It uses journaling for guaranteeing write atomicity by default, which effectively halves the write speed
I'd really rather not do that, thanks.
No, I read the official kernel docs too; the Arch wiki just happened happened to be a quicker way to describe it.
From https://docs.kernel.org/admin-guide/device-mapper/dm-integri... -
> The dm-integrity target can also be used as a standalone target, in this mode it calculates and verifies the integrity tag internally. In this mode, the dm-integrity target can be used to detect silent data corruption on the disk or in the I/O path.
> There’s an alternate mode of operation where dm-integrity uses a bitmap instead of a journal. If a bit in the bitmap is 1, the corresponding region’s data and integrity tags are not synchronized - if the machine crashes, the unsynchronized regions will be recalculated. The bitmap mode is faster than the journal mode, because we don’t have to write the data twice, but it is also less reliable, because if data corruption happens when the machine crashes, it may not be detected.
This is more clearly presented lower down in the list of modes, in which most options describe how they don't actually protect against crashes, except for journal mode:
> J - journaled writes
> data and integrity tags are written to the journal and atomicity is guaranteed. In case of crash, either both data and tag or none of them are written. The journaled mode degrades write throughput twice because the data have to be written twice.
On further reflection, I grant that that might only be talking about the integrity metadata, in which case we just don't know about the impact to data writes and it would be useful to go benchmark to see what the hit is in practice.
EDIT: So I went looking to see if anyone had done that benchmarking and found https://github.com/t13a/dm-integrity-benchmarks which seems to show that actually yes dm-integrity is that bad on data writes. Of course, its possible saving grace is that everything else with the same features also had a performance hit. I also found https://www.reddit.com/r/linuxadmin/comments/1crtggd/why_dmi... talking about it.
It only falls behind, and very signficantly so, on the 1M sequential write test, exactly the situation where you'd expect there to be the least delta between systems! I'm going to bet anything that's a misconfigured RAID.
Frankly looking at that from a "will this work best for my general purpose filesystem used mostly to handle giant software builds and Zephyr test suites" it seems like a no brainer to pick dm, especially so given the simplicity argument.
based on my own testing, dm has a lot of footguns and, with some kernels, as little as 100 bytes of corruption to the underlying disk could render a dm-integrity volume completely unusable (requiring a full rebuild) https://github.com/khimaros/raid-explorations
O_o
Apparently I've been living under a rock, can you please show us a link about this? I was just recently (casually) looking into bolting ZFS/BTRFS-like partial snapshot features to simulate my own atomic distro where I am able to freely roll back if an update goes bad. Think Linux's Timeshift with something little extra.
https://docs.kernel.org/admin-guide/device-mapper/snapshot.h...
A block level cache like bcache (not fs) and dm-cache handles it less ideally, and doesn't leave the SSD space as usable space. As a home user, 2TB of SSDs is 2TB of space I'd rather have. ZFS's ZIL is similar, not leaving it as usable space. Btrfs has some recent work in differentiating drives to store metadata on the faster drives (allocator hints), but that only does metadata as there is no handling of moving data to HDDs over time. Even Microsoft's ReFS does tiered storage I believe.
I just want to have 1 or 2 SSDs, with 1 or 2 HDDs in a single filesystem that gets the advantages of SSDs with recently used files and new writes, and moves all the LRU files to the HDDs. And probably keep all the metadata on the SSDs too.
You were not alone. However, things changed, namely SSD continued to become cheaper and grew in capacity. I'd think most active data is these days on SSDs (certainly in most desktops, most servers which aren't explicit file or DB servers and all mobile and embedded devices), the role of spinning rust being more and more archiving (if found in a system at all).
Also for video games, where performance matters, game sizes are huge, and it's nice to have a bunch of games installed.
IIRC my laptop's zpool has a 1.2x compression ratio; it's worth doing. At a previous job, we had over a petabyte of postgres on ZFS and saved real money with compression. Hilariously, on some servers we also improved performance because ZFS could decompress reads faster than the disk could read.
This is my favorite side effect of compression in the right scenarios. I remember getting a huge speed up in a proprietary in-memory data structure by using LZO (or one of those fast algorithms) which outperformed memcpy, and this was already in memory so no disk io involved! And used less than a third of the memory.
The dm stuff is one key for the entire partition and you can't check it for bitrot or repair it without the key.
As for the snapshots, things like LVM snapshots are pretty coarse, especially for someone like me where I run dm-crypt on top of LVM
I’d say zfs would be pretty well missed with its data integrity features. I’ve heard that btrfs is worse in that aspect, so given that btrfs saved my bacon with a dying ssd, I can only imagine what zfs does.
Maybe, if you never create anything. I make a lot of game art source and much of that is in uncompressed formats. Like blend files, obj files, even DDS can compress, depending on the format and data, due to the mip maps inside them. Without FS compression it would be using GBs more space.
I'm not going to individually go through and micromanage file compression even with a tool. What a waste of time, let the FS do it.
*Coming from the extremely well thought out and documented zfs utilities to btrfs will have you wondering wtf fairly frequently while you learn your way around.
I spent some time researching this topic, and in all benchmarks I've seen and my personal tests btrfs is faster or much faster: https://www.reddit.com/r/zfs/comments/1i3yjpt/very_poor_perf...
"Note that increasing iodepth beyond 1 will not affect synchronous ioengines"[1]
Is there a reason you used that ioengine as opposed to, for example, "libaio" with a "--direct=1" flag?
[1] https://fio.readthedocs.io/en/latest/fio_doc.html#cmdoption-...
A ZFS pool will remain available even in degraded mode, and correct me if I'm wrong but with BTRFS you mount the array through one of the volume that is part of the array and not the array itself.. so if that specific mounted volume happens to go down, the array becomes unavailable unmounted until you remount another available volume that is part of the array which isn't great for availability.
I thought about mitigating that by making an mdadm RAID1 formatted with BTRFS and mount the virtual volume instwad, but then you lose the ability to prevent bit rot, since BTRFS lose that visibility if it doesn't manage the array natively.
I don't think btrfs has a concept of having only some subvolumes usable. Either you can mount the filesystem or you can't. What may have confused you is that you can mount a btrfs filesystem by referring to any individual block device that it uses, and the kernel will track down the others. But if the one device you have listed in /etc/fstab goes missing, you won't be able to mount the filesystem without fixing that issue. You can prevent the issue in the first place by identifying the filesystem by UUID instead of by an individual block device.
You can still mount the BTRFS array as degraded if you specify it during mount. But then this lead to some others issues like the missing data written while degraded will not be automatically be copied over without doing a scrub, while ZFS will resilver it automatically, etc
> You can prevent the issue in the first place by identifying the filesystem by UUID instead of by an individual block device.
I tried that, but all it does is select the first available block device during mount, so if that device goes down, the mount also goes down.
AFAIU, btrfs effectively absolves itself of responsibility in these cases, claiming the issue is buggy drive firmware.
https://zfsonlinux.org/
223 more comments available on Hacker News