I have a batch script that uses xcopy to copy a large number of files. For a while, the following setup worked fine:
set source=<path_to_source_files>
set dest=<path_to_destination>
xcopy /y /s /i /r "%source%" "%dest%"
Today, the same command started producing an error:
File creation error - The system cannot find the path specified.
Furthermore, %errorlevel% was returning 4 (Initialization error occurred. There is not enough memory or disk space, or you entered an invalid drive name or invalid syntax on the command line).
I ruled out space issues, yet the other two possibilities didn't make sense given that many (most?) of the files had already been copied. Just in case, I tried using the /j switch to avoid buffering, but that didn't help.
In an attempt to gather more debugging info, I tried the /l switch to list the files being copied to see if it was any different from what xcopy was already displaying. For some reason, this actually resolved the issue! It even worked when I used the /q switch too (i.e. suppressing the xcopy messages):
xcopy /y /s /i /r /q /l "%source%" "%dest%"
According to this post, xcopy was deprecated back in 2011, so if this is a bug, Microsoft won't fix it. This post doesn't seem to apply either, so I have no idea why this is suddenly an issue (or why the /l switch resolves it, even when the list is then suppressed).
Primarily, this post is just to share the workaround, but hopefully someone can explain the underlying issue and/or workaround.
Updates: the previous workaround doesn't seem to actually work as expected; apparently, with the /l switch, xcopy can finish with %errorlevel%=0 without actually copying anything.
Looking more closely at the list of files, the problematic ones come from Python installation files - specifically, ones from the tensorflow package.
I typically run the script on a VMware VM, but even on my local machine, I consistently hit the same issue on the same file when I omit the /l switch. Interestingly, the problematic files on my local machine are slightly different than the problematic files on a VM (i.e. my local machine progresses further through the tensorflow folder than a VM).
Sometimes, adding the /h switch does the trick, but this doesn't work consistently. The most reliable method seems to be the use of robocopy instead xcopy, but even that command reports a few skipped files.
Originally, the files are zipped to a .tar.gz file, and I sequentially extract the files using 7zip before copying them. I've tried extracting the files directly to the desired directory directly, but the -spe switch doesn't work as expected either.
All things considered, I'm starting to suspect a limitation with tensorflow files.
Glad to know that this issue resolved and thanks for confirming on the actual issue with its resolution.
"The paths didn't seem invalid at first, but then I noticed that the output used relative paths, not full paths. The unzipped folder name was unnecessarily long, and renaming it helped."
Hi @Mofi, Thanks for the helpful comment and posting this answer in this answer section for the community benefit.
Difficult to answer the question without knowing which files with their fully qualified file names could not be copied into which destination directory. It could be that the MAX_PATH limitation (259 characters + a string terminating null character = 260 characters) is responsible for the error message. The listing could work on fully qualified source file name that has less than 260 characters while the fully qualified destination file name has 260 or more characters. There are other file system limitations as well