Stress Test for Parallel Disk I/o Using Git and Pnpm

node/npm operations are vastly slower for me on docker with a linux vm, I have tested this before.

APFS does suck though, don't get me wrong

edit: I just tested, 36s for an `npm i` on a decently large project I have locally, 46s with an overlay fs

It's shocking how much performance you gain by temporarily turning off Windows Defender. I had a local disk to disk copy that the Windows File Transfer dialog estimated was going to finish in 10 hours after it settled down. It wasn't even that much data, just a few hundred GB, but that consisted of a great number of small files. On a hunch I tried disabling Windows Defender and the estimate dropped to about 30 minutes, and in the end it was spot on.

More people need to know how this, it is absolutely bonkers how much perf you lose for each file open operation to defender and its kin.

Even excluding directories (like D:\MyProject) from defender isn't a full solution, the fs filter will still use the file handle to do lookup of the file path to see if it is in an excluded directory (at least on win10), but excluding an entire drive (like D:\) does solve this.

I have found that adding git.exe to the process exclusion list makes the biggest difference here, although git is still slow when dealing with lots of files, it goes from unbearable to bearable.

Using a second partition D: is already twice as fast at small-file-writes compared to the system partition C:. This was on Windows 10 with both partitions using NTFS on the same SSD, and everything else on default settings.

This is because C: has additional file system filters for system integrity that are not present on other drives; and also because C: has 8.3 name compatibility enabled by default, but additional NTFS partitions have that disabled. (so if your filenames are longer than the 8.3 format, Windows has to write twice the number of directory entries)

Is it also faster when you use a host OS dir? (WSL can write to those, and it might still bypass a lot of the filter stuff.)

That might not be a filesystem issue. If NPM pulling a complex dependency tree down spawns a lot of sub-processes then it'll hit the same problem many build operations have when transplanted from Unix-a-likes to Windows (via cygwin, for instance): building up and tearing down processes is relatively expensive under Windows. Under many Unix-a-like systems forking a process is generally not much more expensive than starting a new thread, which is why multi-process solutions are more popular away from Windows (if the cost of using processes over threads is low, then it is worth paying it to get the reduced complexity of not needing to worry about thread safety).

Assuming you are using WSL2 (which is essentially Linux in a VM, unlike WSL1 which was a compatibility layer with processes running more directly on Windows) you will see this difference (any performance cost imposed by the VM will be small compared to the difference in process spin-up costs).

I don't think that this will apply to git, though.

I'm curious:

Lets take that given, i.e. massive IO works reliably only when a single process has access.

How will SQLite handle concurrent access by multiple processes when git/npm/whoever switches over to it?

(A: It doesn't, even in a single process, you need to obtain a global lock to write while all readers pause.)

> Why not just have the filesystem be a SQLite DB then?

A lot of people have proposed exactly this, that modern filesystems ought to be relational databases at the directory/metadata level.

It would be amazing to be able to interface with the file system using SQL and have the power of transactions.

> A: because it would suffer from all of the same problems, except worse, because it is not a filesystem

Not really. Database technologies really are far more advanced in many ways. It would not suffer from many filesystem problems, and would actually be better in many ways.

> Why not just have the filesystem be a SQLite DB then?

One major reason not to is that there needs to be a guaranteed way to free up space, e.g. by deleting a file, or doing something similar. Currently SQLite doesn't provide any means to reliably reclaim space without allocating even more first.

Also the API is quite different, e.g. if you can write(2) a single byte at a time if you really want to, and it's relatively fast compared to doing e.g. an INSERT each time, since it goes to the temp VFS buffer first

Not necessarily sqlite, but rocksdb is looking like promising option for foundational storage system. You have things like zenfs which runs rocksdb directly on nvme storage, and then you have lot of higher level systems on top of it like Cephs BlueStore and mysql/maria MyRocks. It is not much of a stretch to imagine whole OS built around it

What is notable here is that it would be both better aligned with how the underlying storage hardware works, and simultaneously also offers more usable interface for applications.

It is pretty apparent that the biggest reason why we commonly use posixy files is inertia rather than them being very good fit for anything these days

https://en.wikipedia.org/wiki/RocksDB#Integration

https://github.com/westerndigitalcorporation/zenfs

> Why not just have the filesystem be a SQLite DB then?

Already exists: https://github.com/narumatt/sqlitefs

No idea what the performance is like, though… it uses FUSE, so that is going to add some overhead compared to an in-kernel filesystem

I had Ubuntu be 6-7x faster than Windows 10, running Ubuntu in VirtualBox.

You're right, in the grand scheme of things, buying a new disk every few years isn't a major hardship but obstacle. My point wasn't really about the cost of hardware or not being able to solve problem, but rather the brittleness of the software, documentation and usabillity.

The difficult backup migration is just one symptom of a larger problem: Time Machine isn't robust. It has sharp edges and can fail in ways that are hard to recover from, like the network backup I lost. That lack of reliability and breaking changes not just in backup software but in whole ecosystem is the real issue for me. Apple "obsoletes" things in its own timeframe regardless of userbase. I've got enormous respect for Apple products and people who are building them and with them. I wish Apple would have "LTS" version of MacOS and better documentation - often tools exist but aren't easy accessible.

I have had good experience with Arq backup 7

Despite its apparent opaqueness, TM is reasonably transparent for the everyday Mac user

Yeah TM has its shortcomings. I went with Arq (non-affiliated) instead.

Genuinely curious... why do you see the need to back up git?

Even if I don't want to push to a server, I use a different disk or USB as an additional remote in git and push to that. Although I do that only for financial data, so only done once a month so it's not too big a hassle.

Go to Time Machine Settings, click the very prominent Options button, and add things to the "Exclude from Backups" list.