Cleaning House in Nx Monorepo, How I Removed Unused Deps Safely

I have witnessed a lot of kinda spurious pinning going on. Or like "ok we need to fix the bounds for this transitive dependency for a bit" and then it just sticks around.

Over a decade that's once a month, which is a lot though!

I think sometimes people will hear advice like "pin your deps" and do a `pip freeze | requirements.lock.txt`, without really absorbing that pinning transitive dependencies like this is generally not what you want.

You want a lock file! But you tend to want transitive dependencies that aren't locked down to get upgraded when you upgrade your direct dependencies. But it's a subtlety that can get lost in the noise.

In my experience it’s usually engineers who run install commands until it builds locally, then check in whatever they’ve got.

There are several methods you can end up with extra/redundant NPM libraries.

At my last company, we had a cleanup project which ended up with a similar number of libraries removed over the epoch which cleaned up+upgraded NPM dependencies.

(1) the number might be multiplied if one library is removed from each package of a monorepo. Eg. If there are 10 packages in the monrepo, then you only need to remove on average 12 NPM libraries.

(2) sometimes updating the upstream libraries to newer versions reduces the size of the upstream dependency tree. Similarly, it was more common for NPM libraries to declare a dependency, whereas more mature libraries might move those to devDependencies or peer dependencies.

(3) removing unnecessary / unused code can unblock you so you can remove a library which is no longer referenced in your repo.

(4) sometimes programmers over engineer early. Sometimes your product requirements change. Reviewing your codebase after each of these conditions allows you to decide if you still need all of the code/dependencies.

(5) as the NodeJS runtime changes, sometimes you can migrate from a 3rd party library to use the built-in tool (we did this with BluebirdJS / Promises).

> Or (too) many cooks?

Just incompetent cooks. In one of my jobs, I had to cleanup a bunch of dependencies like "npm" and "install" because the guy typed "npm install" and then pasted "npm install react" or something like that. And that was not noticed for several months.

in a decade+ old fintech monorepo w/ microfrontends + BFFs + mixed stacks/design systems (react, angular, express, apollo, chakra, tw) the cruft builds up fast.

every time we migrate stuff (like the recent shift to next.js), old apps/libs don’t always get cleaned out. after years and years it’s an npm landfill. knip was basically the bulldozer.

It sounds like they are counting transitive dependencies. If so, that means they deleted far fewer than 120 different lines in their own config to end up with that level of reduction.

We have a vendors package in our monorepo to centralize both tracking and versioning of our production dependencies on the frontend.

We pre build the dependencies and upload them to a CDN as well, which we can then at build replace the import URLs with the stable CDN ones, it works really well. We have thin wrappers around each dependency to give a stable interface as well.

For what backend TypeScript projects we do have we created typed wrappers around the server framework instead of pre-building because those both deploy differently and have different constraints but this still lets us track and upgrade versions centrally while keeping APIs stable.

Using this approach has had a positive impact on performance across the board as we naturally strive to keep our dependency count low because the maintenance burden is more obvious

The ecosystem is setup to encourage this type of pollution.

It happens when devs and businesses reach a point of tolerant negligence w.r.t their codebases. I suspect this is true for the vast majority of codebases, a "labor of love" codebase in the enterprise is a unicorn.

Remember not every project is for some amazing product, with a great agile way of working. Some projects are a withdrawals system internal to an insurance company, for example. No one gives a shit there.

Motivating for any kind of tech debt work on such a project is unlikely to be met with approval, and all the devs who really care about such things will likely have left the project long ago.

I could easily imagine someone introducing a dependency because it's needed for something, then a few months later someone refactoring to remove the code that used it but not being aware that the dependency wasn't used anywhere else. Repeat this pattern a bunch, and the number of unused dependencies will start climbing. Keep in mind that the more dependencies you have, the less obvious one specific unused dependency in the config files will be (in addition to it likely being easier to introduce yet another one; it's a lot harder to object to dependency N + 1 when you didn't object to the N beforehand).

Without changing the policy around dependencies themselves for a project (which presumably would be an uphill battle for anyone feeling strongly enough about this), the only way to avoid this scenario from happening without tooling would be for people to manually verify that a dependency is still used any time they remove a usage of it. Asking the person who introduces the dependency to keep track of every time someone else introduces another usage of it to the codebase would be burdensome even if you could assume that they wouldn't ever switch jobs/stop contributing to the open source project/take vacations when code is merged in their absence, and realistically I doubt that asking people to notice whenever they remove a use of a dependency and manually verify that the dependency is still used elsewhere would be particularly effective. I don't think there's any way to make this obvious in the absence of tooling, which unfortunately isn't super common in my experience even in projects that make ample use of linting other static checks in CI.

I don't totally agree with the takes that this is from devs not caring about their projects or otherwise phoning it in. Often the projects I've seen struggle with things like this (and even things that are a lot more worrisome than this, like missing or flat-out wrong documentation around stuff like testing locally due to the team relying on unwritten shared knowledge of process) have developers who care deeply about what they're doing, but are stressed, burned-out, and desperately trying to tread water to avoid drowning in the sea of things they'd like to do to improve things. That's not to say that every situation like this is blameless, but unless my experience is greatly out of the ordinary, I'd think that quite a lot of us have been in or at least seen situations like this enough that we should have enough empathy not to jump to conclusions about the state of a project being a direct reflection of how much the devs care. (To be clear, I don't think this is what the parent comment I'm replying to is saying, but it does seem like overall there's a bit of a sentiment similar to this in the thread as a whole, so it feels worth calling out).

Because you re-read the same article? It looks like the title here is the same as the URL indicating it hasn't changed.

yeah you can definitely point it at lodash (or any utility lib) and it’ll surface spots where it’s imported but never really used. but for “we could just rewrite this tiny helper ourselves” type stuff, knip won’t judge code quality. it’s more about flagging dead deps/files than saying “ditch lodash.” that call still needs a human.