Search code examples
node.jsnpmpackagedependenciesyarnpkg

Best Practices for Maintaining and Updating Large, Private NPM Package Ecosystem

What is the easiest/best way to maintain and keep up-to-date NPM packages in a large, internal ecosystem?

Say we have a number of different private NPM packages, each a project with its own repository, that are consumed in a tree-like fashion. Note that this diagram is a very simplified version of the problem for discussion purposes. The actual dependency tree is easily 10x this complex. If you need to make a change to a low-level package, what is the best/easiest way to update the rest of the tree?

Nested Dependencies Example

  • The top-level green boxes represent our front-end websites.
  • The mid-level blue boxes represent intermediate packages which are consumed by the top-level websites, and also consume other low-level packages.
  • The low-level orange boxes represent packages which get consumed by other packages and websites.

Example: Breaking Change to a Low-Level Package

We'll assume all of these packages are currently on v1.0.0 for the sake of this example. To update @adc/package-1, one must:

  • Make the changes in @adc/package-1 and publish the new version (v2.0.0)
  • Update the mid-level packages to consume @adc/package-1@^2.0.0.
  • Publish new versions of the mid-level packages (i.e. @adc/[email protected], @adc/[email protected], @adc/[email protected])
  • Update the top-level websites to consume all new versions of the packages.

This in total requires six PRs, since each package is its own repository. Most of these updates are just updating the version numbers. It also requires that the PRs are merged in a particular order. This is a lot of overhead for a simple change, and again this is very simplified in comparison to the actual package ecosystem that may exist.

What happens if we don't update the mid-level packages?

The top-level websites will now be using two different versions of @adc/package-1.

  • @adc/package-1@^2.0.0 via its package.json direct dependency
  • @adc/package-1@^1.0.0 via a transient dependency through the mid-level packages.

I believe this can lead to non-deterministic behavior, and it also makes it difficult for the next developer to know what versions they can/should use if they need to make an update. The mid-level packages would become out-of-date and eventually it would require us to update all @adc/ package versions to the latest. In a world where you try to mitigate risk, this would mean regression testing the entire package and all consuming packages/websites.

Solution

  • We have migrated our internal NPM packages into a monorepo and utilized PNPM, Changesets, and NX.

    • PNPM feels like an upgrade over Yarn, and it allows you to use local "workspace" versions of the packages, where all packages use the same version of their dependencies.
    • Changesets allow us to easily identify which packages should be updated for a given change (using a friendly CLI) and automatically updates the CHANGELOG files with corresponding comments.
    • NX allows us to be smart about which test suites to run and which packages to build based on which packages have actual changes.

    I am sure the monorepo will come with its own challenges, but currently this beats having a multitude of individual pull requests, one for each of the old repositories.