Cleanest way to fix Windows 'filename too long' CI tests failure in this code?

Question

I'm trying to solve this Windows filename issue

Basically our CI job fails, with the 'filename too long' error for Windows.

 warning: Could not stat path 'node_modules/data-validator/tests/data/data-examples/ds000247/sub-emptyroom/ses-18910512/meg/sub-emptyroom_ses-18910512_task-noise_run-01_meg.ds/sub-emptyroom_ses-18910512_task-noise_run-01_meg.acq': Filename too long

I've read the docs for Node's path module, which seems like a possible solution. I also read about a Windows prefix (\\?\) to bypass the MAX_PATH...but have no idea how to implement these in a clean way.

This part of the codebase with the tests that are failing. The hardcoded path (testDatasetPath) is likely part of the problem.

function getDirectories(srcpath) {
  return fs.readdirSync(srcpath).filter(function(file) {
    return (
      file !== '.git' && fs.statSync(path.join(srcpath, file)).isDirectory()
    )
  })
}

var missing_session_files = //array of strings here

const dataDirectory = 'data-validator/tests/data/'

function createDatasetFileList(path) {
  const testDatasetPath = `${dataDirectory}${path}`
  if (!isNode) {
    return createFileList(testDatasetPath)
  } else {
    return testDatasetPath
  }
}

createFileList function

function createFileList(dir) {
  const str = dir.substr(dir.lastIndexOf('/') + 1) + '$'
  const rootpath = dir.replace(new RegExp(str), '')
  const paths = getFilepaths(dir, [], rootpath)
  return paths.map(path => {
    return createFile(path, path.replace(rootpath, ''))
  })
}

tl;dr A GitLab CI Job fails on Windows because the node module filenames become too long. How can I make this nodejs code OS agnostic?

Answer 1

This is a known error in the Windows Environment, however, there is a fix..

If you're using an NTFS based filesystem, you should be able to enable long paths in

Local Computer Policy > Computer Configuration > Administrative Templates > System > Filesystem > NTFS

This is also specified in the document you just linked, and theoretically, should work. However, the path shouldn't really be longer than 32 bits,

This will come with some performance hits, but should get the job done.

Also, you should preferably switch to a better package or search for alternatives. If this is your own package, maybe restructure it?

Finally, if none of these work, move your project folder directly to your data drive, (C:\) this will cut down on the other nested and parent folders.

This is inherently bad, and you may run into issues during deployment, if you choose to do it.