We've been using the "new" CICD setup for Azure Data Factory as descibed by the documentation page: https://learn.microsoft.com/en-us/azure/data-factory/continuous-integration-delivery-improvements. We're doing this using Azure DevOps pipelines written in YAML.
This has been working quite well for us, until it suddenly started crashing at the validation step this morning. No code has been changed on our side. I'm wondering if there's some issue with version number or incompatability but I can't find anything.
The Azure YAML code in question is this:
- task: Npm@1
displayName: 'Validate ADF json'
inputs:
command: 'custom'
workingDir: $(packageJsonFolder)
customCommand: 'run build validate $(Build.Repository.LocalPath)/DataFactories/${{ parameters.folderName }} $(adfAzureResourceId)'
The full log of error messages is as following:
node /home/vsts/work/1/s/DataFactories/downloads/main.js validate /home/vsts/work/1/s/DataFactories/internaldataload /subscriptions/xxxxxxxxxxxxxxxx/resourceGroups/xxxxxxxxxxxxxx/providers/Microsoft.DataFactory/factories/xxxxx
/home/vsts/work/1/s/DataFactories/node_modules/@microsoft/azure-data-factory-utilities/lib/bundle.manager.js:53
throw "Execution failed with exit code: " + exitCode;
^
Execution failed with exit code: 1
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! @ build: `node node_modules/@microsoft/azure-data-factory-utilities/lib/index "validate" "/home/vsts/work/1/s/DataFactories/internaldataload" "/subscriptions/xxxxxxxxxxxxxxxx/resourceGroups/xxxxxxxxxxxxxx/providers/Microsoft.DataFactory/factories/xxxxx"`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the @ build script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.
npm ERR! A complete log of this run can be found in:
npm ERR! /home/vsts/.npm/_logs/2022-06-16T09_04_15_237Z-debug.log
=====ERROR=====
Error: Command failed: node /home/vsts/work/1/s/DataFactories/downloads/main.js validate /home/vsts/work/1/s/DataFactories/internaldataload /subscriptions/xxxxxxxxxxxxxxxx/resourceGroups/xxxxxxxxxxxxxx/providers/Microsoft.DataFactory/factories/xxxxx
/home/vsts/work/1/s/DataFactories/downloads/main.js:2
(()=>{var e={82257:(e,t,r)=>{e.exports=r(53605)
at Object.94616 (/home/vsts/work/1/s/DataFactories/downloads/main.js:2:4897020)
at r (/home/vsts/work/1/s/DataFactories/downloads/main.js:2:7274976)
at /home/vsts/work/1/s/DataFactories/downloads/main.js:2:7275513
at /home/vsts/work/1/s/DataFactories/downloads/main.js:2:7953499
at Object.<anonymous> (/home/vsts/work/1/s/DataFactories/downloads/main.js:2:7953503)
at Module._compile (internal/modules/cjs/loader.js:778:30)
at Object.Module._extensions..js (internal/modules/cjs/loader.js:789:10)
at Module.load (internal/modules/cjs/loader.js:653:32)
at tryModuleLoad (internal/modules/cjs/loader.js:593:12)
at Function.Module._load (internal/modules/cjs/loader.js:585:3)
Execution finished....
Found npm debug log, make sure the path matches with the one in npm's output: /home/vsts/.npm/_logs/2022-06-16T09_04_15_237Z-debug.log
0 info it worked if it ends with ok
1 verbose cli [ '/opt/hostedtoolcache/node/10.24.1/x64/bin/node',
1 verbose cli '/opt/hostedtoolcache/node/10.24.1/x64/bin/npm',
1 verbose cli 'run',
1 verbose cli 'build',
1 verbose cli 'validate',
1 verbose cli '/home/vsts/work/1/s/DataFactories/internaldataload',
1 verbose cli '/subscriptions/xxxxxxxxxxxxxxxx/resourceGroups/xxxxxxxxxxxxxx/providers/Microsoft.DataFactory/factories/xxxxx' ]
2 info using [email protected]
3 info using [email protected]
4 verbose run-script [ 'prebuild', 'build', 'postbuild' ]
5 info lifecycle @~prebuild: @
6 info lifecycle @~build: @
7 verbose lifecycle @~build: unsafe-perm in lifecycle true
8 verbose lifecycle @~build: PATH: /opt/hostedtoolcache/node/10.24.1/x64/lib/node_modules/npm/node_modules/npm-lifecycle/node-gyp-bin:/home/vsts/work/1/s/DataFactories/node_modules/.bin:/opt/hostedtoolcache/node/10.24.1/x64/bin:/home/linuxbrew/.linuxbrew/bin:/home/linuxbrew/.linuxbrew/sbin:/home/vsts/.local/bin:/opt/pipx_bin:/home/vsts/.cargo/bin:/home/vsts/.config/composer/vendor/bin:/usr/local/.ghcup/bin:/home/vsts/.dotnet/tools:/snap/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
9 verbose lifecycle @~build: CWD: /home/vsts/work/1/s/DataFactories
10 silly lifecycle @~build: Args: [ '-c',
10 silly lifecycle 'node node_modules/@microsoft/azure-data-factory-utilities/lib/index "validate" "/home/vsts/work/1/s/DataFactories/internaldataload" "/subscriptions/xxxxxxxxxxxxxxxx/resourceGroups/xxxxxxxxxxxxxx/providers/Microsoft.DataFactory/factories/xxxxx"' ]
11 silly lifecycle @~build: Returned: code: 1 signal: null
12 info lifecycle @~build: Failed to exec build script
13 verbose stack Error: @ build: `node node_modules/@microsoft/azure-data-factory-utilities/lib/index "validate" "/home/vsts/work/1/s/DataFactories/internaldataload" "/subscriptions/xxxxxxxxxxxxxxxx/resourceGroups/xxxxxxxxxxxxxx/providers/Microsoft.DataFactory/factories/xxxxx"`
13 verbose stack Exit status 1
13 verbose stack at EventEmitter.<anonymous> (/opt/hostedtoolcache/node/10.24.1/x64/lib/node_modules/npm/node_modules/npm-lifecycle/index.js:332:16)
13 verbose stack at EventEmitter.emit (events.js:198:13)
13 verbose stack at ChildProcess.<anonymous> (/opt/hostedtoolcache/node/10.24.1/x64/lib/node_modules/npm/node_modules/npm-lifecycle/lib/spawn.js:55:14)
13 verbose stack at ChildProcess.emit (events.js:198:13)
13 verbose stack at maybeClose (internal/child_process.js:982:16)
13 verbose stack at Process.ChildProcess._handle.onexit (internal/child_process.js:259:5)
14 verbose pkgid @
15 verbose cwd /home/vsts/work/1/s/DataFactories
16 verbose Linux 5.13.0-1029-azure
17 verbose argv "/opt/hostedtoolcache/node/10.24.1/x64/bin/node" "/opt/hostedtoolcache/node/10.24.1/x64/bin/npm" "run" "build" "validate" "/home/vsts/work/1/s/DataFactories/internaldataload" "/subscriptions/xxxxxxxxxxxxxxxx/resourceGroups/xxxxxxxxxxxxxx/providers/Microsoft.DataFactory/factories/xxxxx"
18 verbose node v10.24.1
19 verbose npm v6.14.12
20 error code ELIFECYCLE
21 error errno 1
22 error @ build: `node node_modules/@microsoft/azure-data-factory-utilities/lib/index "validate" "/home/vsts/work/1/s/DataFactories/internaldataload" "/subscriptions/xxxxxxxxxxxxxxxx/resourceGroups/xxxxxxxxxxxxxx/providers/Microsoft.DataFactory/factories/xxxxx"`
22 error Exit status 1
23 error Failed at the @ build script.
23 error This is probably not a problem with npm. There is likely additional logging output above.
24 verbose exit [ 1, true ]
##[error]Error: Npm failed with return code: 1
Finishing: Validate ADF json
Our package.json file is exactly as from the documentation page.
{
"scripts":{
"build":"node node_modules/@microsoft/azure-data-factory-utilities/lib/index"
},
"dependencies":{
"@microsoft/azure-data-factory-utilities":"^0.1.5"
}
}
Any ideas what we can do?
Fixed after updating the 'Install Node.js' task to match the reference from doc:
# Installs Node and the npm packages saved in your package.json file in the build
- task: NodeTool@0
inputs:
versionSpec: '14.x'
displayName: 'Install Node.js'