Nfs at 40 – Remembering the Sun Microsystems Network File System

plusses were security (kerberos), better administrative controls and global file space.

minuses were generally poor performance, middling small file support and awful large file support. substantial administrative overhead. the wide-area performance was so bad the global namespace thing wasn't really useful.

I guess it didn't cause as many actual multi-hour outages NFS, but we used it primarily for home/working directories and left the servers alone, whereas the accepted practice at the time was to use NFS for roots and to cross mount everything so that it easily got into a 'help I've fallen and can't get up' situation.

[0]: https://openafs.org/

[1]: https://www.auristor.com/filesystem/

It's one of those tools that, unless you already know what you're doing, you can expect to sink several hours into trying to get the damn thing working correctly.

It's not the kind of thing you can throw at a junior and expect them to get working in an afternoon.

Whereas NFS and sshfs "just work". Albeit I will concede that NFSv4 was annoying to get working back when that was new too. But that's, thankfully, a distant memory.

SMBv1 has a reputation for being an extremely chatty protocol. Novell ipx/spx easily dwarfed it in the early days. Microsoft now disables it by default, but some utilities (curl) do not support more advanced versions.

SMBv2 increases efficiency by bundling multiple messages into a single transmission. It is clear text only.

SMBv3 supports optional encryption.

Apple dropped the Samba project from MacOS due to gplv3, and developed their own SMB implementation that is not used elsewhere AFAIK. If you don't care for Apple's implementation, then perhaps installing Samba is a better option.

NFSv3 relies solely on uid/gid mapping by default, while NFSv4 requires idmapd to run to avoid squashing. I sometimes use both at the same time.

Samba can be. Especially when compared with NFS

> NFS support was lacking on windows when I last tried.

If you need to connect from Windows then your options are very limited, unfortunately.

I do agree that object storage is a nice option. I wonder if a FUSE-like object storage wrapper would work well here. I've seen mixed results for S3 but for local instances, it might be a different story.

- POSIX compliant, including dotting the i's. As opposed to, say, NFS which isn't cache coherent.

- performance and scalability. 1 TB/s+ sequential IO to a single file is what you'd expect on a large HPC system these days.

- Metadata performance has gotten a lot better over the past decade or so, beating most(all?) other parallel filesystems.

Downsides:

- Lots of pieces in a Lustre cluster (typically nodes are paired in sort-of active/active HA configs). And lots of cables, switches etc. So a fairly decent chance something breaks every now and then.

- When something breaks, Lustre is weird and different compared to many other filesystems. Tools are rudimentary and different.

To get a feel for what 'life with Lustre' could be, see e.g. various 'site reports' from workshops. E.g. for a couple somewhat recent ones: https://www.eofs.eu/wp-content/uploads/2024/09/cscs_site_rep... and https://www.eofs.eu/wp-content/uploads/2024/09/LAD-24-Luster...

The caveat is a lot of software is written to assume things like fread(), fopen() etc will either quickly fail or work. However, if the file is over a network obviously things can go wrong so the common default behaviour is to wait for the server to come back online. Same issue applies to any other network filesystem, different OS's (and even the same OS with different configs) handle the situation differently.

> Hi, could you give some pointers about this? Thanks!

* https://man.archlinux.org/man/nfs.5.en#soft

* https://kb.netapp.com/on-prem/ontap/da/NAS/NAS-KBs/What_are_...

Synchronous I/O may be simple, but it falls down hard at the "complex things should be possible" bit. And people have been doing async I/O for decades before they got handholding constructs like 'async' and 'await'. Programming the Amiga, for instance, was done entirely around async I/O to and from the custom chips. The CPU needn't do much at all to blow away the PC at many tasks; just initiate DMA transfers to Paula, Denise, and Agnus.

Quote[0]:

> In Ingo's view, there are only two solutions to any operating system problem which are of interest: (1) the one which is easiest to program with, and (2) the one that performs the best. In the I/O space, he claims, the easiest approach is synchronous I/O calls and user-space processes. The fastest approach will be "a pure, minimal state machine" optimized for the specific task; his Tux web server is given as an example.

Granted, most software is not developed for the Linux kernel, but neither is asynchronous programming black magic. I think the software space has rather been negatively impacted by being slow to adopt asynchronous programming, among other old practices.

[0] https://lwn.net/Articles/219954/

The AFS consistency model is fairly strong. Each client (aka cache manager) is only permitted to access the data/metadata of a vnode if it has been issued a callback promise from the AFS fileserver. File lock transitions, metdata modifications, and data modifications as well as volume transactions cause the fileserver to break the promise. At which point the client is required to fetch updated status information before it can decide it is safe to reuse the locally cached data.

Unlike optimistic locking models, the AFS model permits cached data to be validated after an extended period of time by requesting up to date metadata and a new callback promise.

An AFS fileserver will not permit a client to perform a state changing operation as long as there exist broken callback promises which have yet to be successfully delivered to the client.

I looked up at one point whatever happened to AFS and it turns out that it has some Amdahl’s Law glass ceiling that ultimately limits the aggregate bandwidth to something around 1 GBps, which was fine when it was young but not fine when 100Mb Ethernet was ubiquitous and gigabit was obtainable with deep enough pockets. If adding more hardware can’t make the filesystem faster you’re dead.

I don’t know if or how openAFS has avoided these issues.

https://www.usenix.org/legacy/publications/library/proceedin...

https://workshop.openafs.org/afsbpw08/talks/wed_1/OpenAFS_an...

1) AFS, IIRC, required more than one machine in its original configuration. That meant hardware and sysadmins which were expensive--until, suddenly they weren't.

2) Disk, memory and bandwidth were scarce--and then they weren't. AFS made a bunch of solid architectural decisions and then wasted a bunch of time backing some of them down in deference to the hardware of the day and then all that work was wasted when Moore's Law overran everything, anyhow.

3) Everybody was super happy to be running everything locally to escape the tyranny of the "Mainframe Operator" (meaning no NFS or AFS or the like)--until they weren't. Once enough non-technical people appeared who didn't want to do system administration, like, ever, that flipped.

We lost the VMS filesystem in this timeframe, too. Which was also a distributed, remote filesystem.

But those x86 processors sure are cheap ... sigh.

I often wonder how the world would be different if AFS 3.0 could have been freely distributed world wide in 1989 precluding the need for HTTP to be developed at CERN.

For example, one of the big uses of NFS we had was for engineering documents, all of which could be accessed from FrameMaker or Interleaf running on your workstation. Nowaways, all the engineering documentation and more would be accessed through a Web browser from a non-NFS server, and no longer on a shared filesystem.

Another use of NFS we had was for collaborating on shared code by some projects, with SCM storing to NFS servers (other projects used DSEE and ClearCase). But nowaways almost everyone in industry uses distributed Git, syncing to non-NFS servers, with cached copies on their local storage.

I suppose a third thing that changed was CSCW distributed change syncing becoming popular at moving into other tools, such as a live "shared whiteboard" document editing that people can access in their Web browsers. I have mixed feelings about some of the implementations and how they're deployed, but it's pretty wild to have 4 remote people during Covid editing a document in real time at once, and NFS isn't helping with the hard part of that.

Right now, the use case for NFS that first comes to mind is individual humans working with huge files (e.g., for AI training, or other big data), where you want the convenience of being able to access them with any tool from your workstation, and maybe also have big compute servers working with them, without copying things around. You could sorta do these things with big complicated MLops infrastructure, but sometimes that slows you down more than it speeds you up.

Failover, latency, and so on are something you need to think about independently of what transfer protocol you use. NFS may present its own challenges with all the different extensions and flags, but that's true of any mature technology.

That said, live code updates probably aren't a very good idea anyway, for exactly the reasons you mention. Those are the reasons you were right at the time, not any inherent deficiencies on the NFS protocol.