The Latest Linux File-System: Ternfs

(disclaimer: CTO of XTX)

These limits aren't quite as strict as they first seem.

Our median file size is 2MB, which means 50% of our files are <2MB. Realistically if you've got an exabyte of data with an average file size of a few kilobytes then this is the wrong tool for the job (you need something more like a database), but otherwise it should be just fine. We actually have a nice little optimisation where very small files are stored inline in the metadata.

It works out of the box with "normal" tools like rsync, python, etc despite the immutability. The reality is that most things don't actually modify files, even text editors tend to save a new version and rename over the top. We had to update relatively little of our massive code base when switching over to this. For us that was a big win, moving to an S3-like interface would have required updating a lot of code.

Directory creation/deletion is "slow", currenly limited to about 10,000 operations per second. We don't current need to create more than 10,000 directories per second so we just haven't prioritised improving that. There is an issue open, #28, which would get this up to 100,000 per second. This is the sort of thing that, like access control, I would love to have had in an initial open source release, but we prioritised open sourcing what we have over getting it perfect.

So, it competes more with S3/minio than NFS it seems ?

What's wrong with NFS?

Nfs is cheap and simple. We are using it for over 15 years in our business. Sering 10s of million daily users. I yet have to find a replacement.

(disclaimer: CTO of XTX)

It was a long long time ago that we were only using NFS, it ran on top of a Solaris machine running ZFS. It did its job at the very beginning, but you don't build up hundreds of petabytes of data on an NFS server.

We did try various solutions in between NFS and developing TernFS, both open source and properietary. However we didn't name these specifically in the blog post because there's little point in bad mouthing what didn't work out for us.

There’s definitely a space for these highly-specialised filesystems. You wouldn’t want to use this as your /home FS, nor would you want to use ext4 or something similar for what they’re trying to do.

No, they aren't. Especially not distributed filesystems which really aren't yet a "solved problem", which in part explains why there are all these proprietary competing ones still around and companies everywhere using all different ones. NFS, BeeGFS, Weka, Ceph, Lustre, GPFS, GoogleFS, Coda/AFS, and more, each with their own flavor of crap.

For local filesystems, the average PC user shouldn't really care though. Just use whatever your installer defaults. But this story is about a distributed filesystem.

I don't have great hopes for one capable of such massive scale being good and usable (low overhead, low complexity, low adminst cost) in very small configurations, but we can always hope.