If All the World Were a Monorepo

That's the objective function of Hastie et al's GLM. I had a good chuckle when I realized the author's last name is Tibshirani. If you know you know.

When you propose a change to something that other things depend on, it makes sense to test those dependents for a regression; this is not earth shattering.

If you want to change something which breaks them, you have to then do it in a different way. First provide a new way of doing something. Then get all the dependencies that use the old way to migrate to the new way. Then when the dependents are no longer relying on the old way, you can push out a change which removes it.

I like the perspective presented in this article, I think CRAN is taking an interesting approach. But this is nuts and bolts. Explicitly saying you're compatible with any future breaking changes!? You can't possibly know that!

I get that a lot of R programmers might be data scientists first and programmers second, so many of them probably don't know semver, but I feel like the language should guide them to a safe choice here. If CRAN is going to email you about reverse dependencies, maybe publishing a package with a crazy semver expression should also trigger an email.

In a true monorepo — the one for the FreeBSD base system, say — if you make a PR that updates some low-level code, then the expectation is that you 1. compile the tree and run all the tests (so far so good), 2. update the high-level code so the tests pass (hmm), and 3. include those updates in your PR. In a true centralized monorepo, a single atomic commit can affect vertical-slice change through a dependency and all of its transitive dependents.

I don’t know what the equivalent would be in distributed “meta-monorepo” development ala CRAN, but it’s not what they’re currently doing.

(One hypothetical approach I could imagine, is that a dependency major-version release of a package can ship with AST-rewriting-algorithm code migrations, which automatically push both “dependency-computed” PRs to the dependents’ repos, while also pushing those same patches as temporary forced overlays onto releases of dependent packages until such time as the related PRs get merged. So your dependents’ tests still have to pass before you can release your package — but you can iteratively update things on your end until those tests do pass, and then trigger a simultaneous release of your package and your dependent packages. It’s then in your dependents’ court to modify + merge your PR to undo the forced overlay, asynchronously, as they wish.)

> CRAN had also rerun the tests for all packages that depend on mine, even if they don’t belong to me!

Another way to frame this is these are the customers of your package's API. If you broke them you are required to ship a fix.

I see why this isn't the default (e.g. on GitHub you have no idea how many people depend on you). But the developer experience is much nicer like this. Google, for example, makes this promise with some of their public tools.

Outside the word of professional software developers, R is used by many academics in statistics, economics, social sciences etc. This rule makes it less likely that their research breaks because of some obscure dependency they don't understand.

> But the migration had a steep cost: over 6 years later, there are thousands of projects still stuck on an older version.

This is a feature, not a bug. The pinning of versions allows systems to independently maintain their own dependency trees. This is how your Linux distribution actually remains stable (or used to, before the onslaught of "rolling release" distributions, and the infection of the "automatically updating application" into product development culture, which constantly leaves me with non-functional Mobile applications whereupon I am forced to update them once a week). You set the versions, and nothing changes, so you can keep using the same software, and it doesn't break. Until you choose to upgrade it and deal with all the breaking shit.

Every decision in life is a tradeoff. Do you go with no version numbers at all, always updating, always fixing things? Or do you always require version numbers, keeping things stable, but having difficulty updating because of a lack of compatible versions? Or do you find some middle ground? There are pros and cons to all these decisions. There is no one best way, only different ways.

This system is unworkable.

The way a software developer thinks about a package is totally different to the way someone trying to perform statistical analysis thinks about packages.

This is the same for CTAN, the name is no coincidence. The packages are for users and not developers.

The downside is that dependees have to manually change their dependency and you get proliferation of packages with informal relationships.

Automated tests, compilation by the package publisher, and enforcement of portability flags and SemVer semantics.

Meanwhile I read the material and t absolutely feels like a cult - "R is fun" is like something they say to persuade themselves they are not in a cult

> In the years since, my discomfort has given away to fascination. I’ve come to respect R’s bold choices, its clarity of focus, and the R community’s continued confidence to ‘do their own thing’.

I would love to see a follow-up article about the key insights that the author took away from diving more deeply into R.

I then discovered that there are often bugs with many of the python stats packages. Many python numerical packages also have the reputation of changing how things work "under the hood" from version to version. This means that you can't trust your output after a version change.

Given all of the above, I can see why "serious" data scientists stick with R and this article is just another reason why.

That essentially makes the high level project a monorepo while giving you the option to work on the submodule on its own.

Wait a second. Another package failed your MAJOR version upgrade because you changed your API? Not following semver is crazy for any package manager to enforce.

I mean, just look at how many projects use “curl and bash” as their distribution method even though the project repositories they could use instead don’t even require anything nearly as onerous as the reverse dependency checks described in this article. If the minimal requirements the current repos have are enough to push projects to alternate distribution, I can’t imagine what would happen if it was added.

Zero wouldn't have been surprising to me, nor would several hundred, but two... what a conveniently actionable number.

It has me wanting to give names to some of my hacks and publish them as packages so that people are more more aware when their changes are breaking changes. On the other hand, if I do something weird, I don't necessity want to burden others with maintaining it. Tradeoffs...