Intermittent failure (127) to find command in yarn

Question

I'm trying to build something on gcloud, specifically a web site with Yarn and Node.

This is the script I'm trying to run:

"build:langs": "printenv && ls -lah node_modules/.bin && BABEL_ENV=test babel-node scripts/translate.js",

Via

$ yarn run build:langs

Which outputs https://gist.github.com/haf/ebc623bfce5520432c136e44496b58fb — the interesting bits here (formatted for readability):

Step #2 - "js-build-langs": 
PATH=/workspace/src/site.com/node_modules/.bin
  :/usr/local/share/.config/yarn/link/node_modules/.bin
  :/workspace/src/
...
total 40K
Step #2 - "js-build-langs": drwxr-xr-x 2 root root 4.0K Jun 3 19:18 .
Step #2 - "js-build-langs": drwxr-xr-x 1194 root root 36K Jun 3 19:19 ..
Step #2 - "js-build-langs": lrwxrwxrwx 1 root root 20 Jun 3 19:18 JSONStream -> ../JSONStream/bin.js
Step #2 - "js-build-langs": lrwxrwxrwx 1 root root 19 Jun 3 19:18 _mocha -> ../mocha/bin/_mocha
...
Step #2 - "js-build-langs": lrwxrwxrwx 1 root root 30 Jun 3 19:18 babel-node -> ../babel-cli/bin/babel-node.js
...
Step #2 - "js-build-langs": /bin/sh: 1: babel-node: not found
Step #2 - "js-build-langs": error Command failed with exit code 127.
Step #2 - "js-build-langs": info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

An in container builder:

That's not all, because in this freak-show, a few minutes (now: hours) earlier 😨:

The cloudbuild target looks like:

steps:
# JS from https://console.cloud.google.com/gcr/images/cloud-builders/GLOBAL/yarn?gcrImageListsize=50:
- name: gcr.io/cloud-builders/yarn:node-8.11.0
  waitFor: ["-"]
  id: js-install
  dir: src/site.com
  args: ["install"]

- name: gcr.io/cloud-builders/yarn:node-8.11.0
  waitFor: ["js-install"]
  id: js-build-prod
  dir: src/site.com
  args: ["run", "build:prod"]

- name: gcr.io/cloud-builders/yarn:node-8.11.0
  waitFor: ["js-build-prod"]
  id: js-build-langs
  dir: src/site.com
  args: ["run", "build:langs"]

And packages.json snip:

  "devDependencies": {
    "@babel/core": "^7.0.0-beta.49",
    "@babel/node": "^7.0.0-beta.49",

All in all, I'm very confounded. Why does it not work despite PATH containing the binary js file, which is executable non the less?? Why can it find webpack but not babel-node? Why can I run it locally even after a full git clean -fxd? And why would it work sometimes but not other times?

My gut is telling me this is some sort of race condition in GCB

Answer 1

EDIT 3: this comes up once in a while for us, and here's our current thoughts.

When we chain yarn-commands with && in package.json, that causes issues with creating e.g. node_modules/mocha/bin/mocha.js or other executable js files.

We made each command in packages.json non-chained and cleaned out node_modules. At least one build has gone through now.

Googling '127 command not found' and 'yarn' gives no hints whatsoever about this.