How to build and package only the relevant dependencies using nx?

Question

I have a project using nx that has many packages in it. My problem is that I only have a single package.json file so if I want to build only one package I still have to build the root project using npm install.

This is a problem because during the CI/CD step I have to build the whole project which is taking too long, and it also generates a single node_modules folder that becomes huge (3GB) which also makes my bundle size prohibitively big.

How can I build individual packages in a way that the node_modules folder will only contain the dependencies that my package needs instead of having all the dependencies of all packages?

If that's not possible how can I compile a single executable main.js file that bundles all those dependencies?

Edit: I tried splitting the package.json file for all the packages, but whenever I build an individual package I still get all the dependencies loaded into the node_modules folder in the root. Is it possible to have a node_modules folder for each individual package?

Answer 1

I found a solution that is not ideal, but it works at least. What the generatePackageJson does is that it creates a package.json file in the build folder that can be used to run the build again to produce a node_modules folder that only contains the dependencies of the specific target. So in my case what I did is to add "generatePackageJson": true to my build target in my backend package:

{
  "sourceRoot": "apps/backend/src",
  "projectType": "application",
  "targets": {
    "build": {
      // ...
      "options": {
        "outputPath": "dist/apps/backend",
        "main": "apps/backend/src/main.ts",
        "tsConfig": "apps/backend/tsconfig.app.json",
        "generatePackageJson": true, // 👈
        "assets": [
          "apps/backend/src/assets"
        ]
      }
      // ...
    }
  }
}

Now if I run nx build backend I'll have something in dist/apps/backend that can be used as a standalone project. Unfortunately npm checks the parent folders for package.json files and messes up everything so I ended up writing a build script that cleans up the project and enables easy deployment. (This works with Heroku but it is easy to adapt to other PaaS / IaaS solutions. I'm gonna paste it here verbatim:

Note that this script runs from the root folder, so our backend dist folder is at dist/apps/backend relative to where this script runs. (I keep the scripts in the script folder. You are also not supposed to run this locally as it deletes stuff form the project. Only run this in a CI/CD environment.

#!/bin/bash

print_usage() {
    echo "💀 $1"
    echo "Help below 👇"
    echo ""
    echo "Builds the specified project for Heroku."
    echo ""
    echo "Usage: ./heroku-build <nx-project-name>"
    echo ""
    echo "__Note that__ this script is intended to be used from CI/CD, you probably won't need it during development."
    exit 1
}

if [ -z "$1" ]; then
    print_usage "Project name is missing!"
fi

# Heroku build is a little bit different because they have a slug size (deployment size) limit of 500MB.
# If you build the project (not just the package) you'll end up with a `node_modules` folder that's ridiculously big (3GB)
# but it can't be pruned properly. If you think you can prune it without this hacky solution go ahead, but it is unlikely
# that you'll be able to figure it out. **If** you try it please increment the counter below:
# 
# total_hours_wasted_trying_to_prune_node_modules=13
# 
# So how this works is that Heroku will run `npm install --prod` that will delete devDependencies too, so
# 💀 DON'T MOVE nx and @nrwl packages to devDependencies! 💀
# After the project is built you'll have the horrendous `node_modules` folder, but it's not a big deal as we'll delete it.

# Build the project with nx
nx build $1 --prod

# This step is necessary because npm will look for `package.json` files in the parent folder and it will use the `node_modules`
# folder from the parent folder. We don't want that, we want to have only the necessary packages (backend) in the `node_modules` folder.
mv package.json package.json.bak
mv package-lock.json package-lock.json.bak

# We get rid of all the unnecessary packages.
rm -Rf node_modules

# We install the necessary packages.
# In the `nx build` step nx generates a `package.json` that only contains the dependencies of the backend project
# so this will *only* (😒) download 500MB from npm.
npm install dist/apps/backend

# More reading on this problem:
#
# - https://stackoverflow.com/questions/73373307/how-to-build-and-package-only-the-relevant-dependencies-using-nx?noredirect=1#comment129738870_73373307
# - https://github.com/nrwl/nx/issues/1518
# - https://github.com/nestjs/nest/issues/1706#issuecomment-579248915