Snakemake : Failed to set marker file for job started ([WinError 3] The system cannot find the path specified:

I am new to Snakemake, I just started using it for my project. I am trying to run a software package that uses Snakemake, however the workflow fails and the problem seems to be caused by windows not being able to locate some files [please see below for the complete error report]. Could someone please give me some ideas of possible solutions? Thank you

snakemake -s U:\Lab\Bereket_public\custom_merfish_rig\Merlin_test\test_data\results\data\snakemake\SnakeFile.Snakefile --cores 4 U:\Lab\Bereket_public\custom_merfish_rig\Merlin_test\test_data\results\data\GenerateMosaic\tasks\GenerateMosaic.done
Building DAG of jobs...
Provided cores: 4
Rules claiming more threads will be scaled down.
Job stats:
job                            count    min threads    max threads
---------------------------  -------  -------------  -------------
FiducialCorrelationWarp            2              1              1
FiducialCorrelationWarpDone        1              1              1
GenerateMosaic                     1              1              1
SimpleGlobalAlignment              1              1              1
total                              5              1              1

Select jobs to execute...

[Tue Jan 17 10:51:28 2023]
Job 1: Running SimpleGlobalAlignment
Reason: Missing output files: U:/Lab/Bereket_public/custom_merfish_rig/Merlin_test/test_data/results/data/SimpleGlobalAlignment/tasks/SimpleGlobalAlignment.done


[Tue Jan 17 10:51:28 2023]
Job 3: Running FiducialCorrelationWarp 0
Reason: Missing output files: U:/Lab/Bereket_public/custom_merfish_rig/Merlin_test/test_data/results/data/FiducialCorrelationWarp/tasks/FiducialCorrelationWarp_0.done

Failed to set marker file for job started ([WinError 3] The system cannot find the path specified: 'U:\\Lab\\Bereket_public\\custom_merfish_rig\\MERlin_epigen_UCSD\\.snakemake\\incomplete\\tmp72nkj4zo.VTovTGFi' -> 'U:\\Lab\\Bereket_public\\custom_merfish_rig\\MERlin_epigen_UCSD\\.snakemake\\incomplete\\VTovTGFiL0JlcmVrZXRfcHVibGljL2N1c3RvbV9tZXJmaXNoX3JpZy9NZXJsaW5fdGVzdC90ZXN0X2RhdGEvcmVzdWx0cy9kYXRhL0ZpZHVjaWFsQ29ycmVsYXRpb25XYXJwL3Rhc2tzL0ZpZHVjaWFsQ29ycmVsYXRpb25XYXJwXzAuZG9uZQ=='). Snakemake will work, but cannot ensure that output files are complete in case of a kill signal or power loss. Please ensure write permissions for the directory U:\Lab\Bereket_public\custom_merfish_rig\MERlin_epigen_UCSD\.snakemake

[Tue Jan 17 10:51:28 2023]
Job 4: Running FiducialCorrelationWarp 1
Reason: Missing output files: U:/Lab/Bereket_public/custom_merfish_rig/Merlin_test/test_data/results/data/FiducialCorrelationWarp/tasks/FiducialCorrelationWarp_1.done

Failed to set marker file for job started ([WinError 3] The system cannot find the path specified: 'U:\\Lab\\Bereket_public\\custom_merfish_rig\\MERlin_epigen_UCSD\\.snakemake\\incomplete\\tmpo2nxcdd7.VTovTGFi' -> 'U:\\Lab\\Bereket_public\\custom_merfish_rig\\MERlin_epigen_UCSD\\.snakemake\\incomplete\\VTovTGFiL0JlcmVrZXRfcHVibGljL2N1c3RvbV9tZXJmaXNoX3JpZy9NZXJsaW5fdGVzdC90ZXN0X2RhdGEvcmVzdWx0cy9kYXRhL0ZpZHVjaWFsQ29ycmVsYXRpb25XYXJwL3Rhc2tzL0ZpZHVjaWFsQ29ycmVsYXRpb25XYXJwXzEuZG9uZQ=='). Snakemake will work, but cannot ensure that output files are complete in case of a kill signal or power loss. Please ensure write permissions for the directory U:\Lab\Bereket_public\custom_merfish_rig\MERlin_epigen_UCSD\.snakemake
MERlin - the MERFISH decoding pipeline
MERlin - the MERFISH decoding pipeline
MERlin - the MERFISH decoding pipeline
Running SimpleGlobalAlignment
[Tue Jan 17 10:51:34 2023]
Finished job 1.
1 of 5 steps (20%) done
Running FiducialCorrelationWarp
Running FiducialCorrelationWarp
U:\Lab\Bereket_public\custom_merfish_rig\MERlin_epigen_UCSD\merlin\analysis\warp.py:143: FutureWarning: The input object of type 'SimilarityTransform' is an array-like implementing one of the corresponding protocols (`__array__`, `__array_interface__` or `__array_struct__`); but not a sequence (or 0-D). In the future, this object will be coerced as if it was first converted using `np.array(obj)`. To retain the old behaviour, you have to either modify the type 'SimilarityTransform', or assign to an empty array created with `np.empty(correct_shape, dtype=object)`.
  np.array(transformationList,dtype = object), 'offsets',
U:\Lab\Bereket_public\custom_merfish_rig\MERlin_epigen_UCSD\merlin\analysis\warp.py:143: FutureWarning: The input object of type 'SimilarityTransform' is an array-like implementing one of the corresponding protocols (`__array__`, `__array_interface__` or `__array_struct__`); but not a sequence (or 0-D). In the future, this object will be coerced as if it was first converted using `np.array(obj)`. To retain the old behaviour, you have to either modify the type 'SimilarityTransform', or assign to an empty array created with `np.empty(correct_shape, dtype=object)`.
  np.array(transformationList,dtype = object), 'offsets',
WorkflowError:
Error recording metadata for finished job ([WinError 3] The system cannot find the path specified: 'U:\\Lab\\Bereket_public\\custom_merfish_rig\\MERlin_epigen_UCSD\\.snakemake\\metadata\\tmplptve0_x.VTovTGFi' -> 'U:\\Lab\\Bereket_public\\custom_merfish_rig\\MERlin_epigen_UCSD\\.snakemake\\metadata\\VTovTGFiL0JlcmVrZXRfcHVibGljL2N1c3RvbV9tZXJmaXNoX3JpZy9NZXJsaW5fdGVzdC90ZXN0X2RhdGEvcmVzdWx0cy9kYXRhL0ZpZHVjaWFsQ29ycmVsYXRpb25XYXJwL3Rhc2tzL0ZpZHVjaWFsQ29ycmVsYXRpb25XYXJwXzAuZG9uZQ=='). Please ensure write permissions for the directory U:\Lab\Bereket_public\custom_merfish_rig\MERlin_epigen_UCSD\.snakemake
Removing output files of failed job FiducialCorrelationWarp since they might be corrupted:
U:/Lab/Bereket_public/custom_merfish_rig/Merlin_test/test_data/results/data/FiducialCorrelationWarp/tasks/FiducialCorrelationWarp_0.done
WorkflowError:
Error recording metadata for finished job ([WinError 3] The system cannot find the path specified: 'U:\\Lab\\Bereket_public\\custom_merfish_rig\\MERlin_epigen_UCSD\\.snakemake\\metadata\\tmpl6y_8x_q.VTovTGFi' -> 'U:\\Lab\\Bereket_public\\custom_merfish_rig\\MERlin_epigen_UCSD\\.snakemake\\metadata\\VTovTGFiL0JlcmVrZXRfcHVibGljL2N1c3RvbV9tZXJmaXNoX3JpZy9NZXJsaW5fdGVzdC90ZXN0X2RhdGEvcmVzdWx0cy9kYXRhL0ZpZHVjaWFsQ29ycmVsYXRpb25XYXJwL3Rhc2tzL0ZpZHVjaWFsQ29ycmVsYXRpb25XYXJwXzEuZG9uZQ=='). Please ensure write permissions for the directory U:\Lab\Bereket_public\custom_merfish_rig\MERlin_epigen_UCSD\.snakemake
Removing output files of failed job FiducialCorrelationWarp since they might be corrupted:
U:/Lab/Bereket_public/custom_merfish_rig/Merlin_test/test_data/results/data/FiducialCorrelationWarp/tasks/FiducialCorrelationWarp_1.done
Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
Complete log: .snakemake\log\2023-01-17T105128.126829.snakemake.log

Solution

From you question I assume you are not developing the software package which uses snakemake but you are only using it.

You might be running into an issue where the paths snakemake internally creates are to long for windows and exceed the maximum path length of 260 characters which Windows uses by default.

The very long line with the long paths is an indication for this:

Error recording metadata for finished job ([WinError 3] The system cannot find the path specified: 'U:\\Lab\\Bereket_public\\custom_merfish_rig\\MERlin_epigen_UCSD\\.snakemake\\metadata\\tmplptve0_x.VTovTGFi' -> 'U:\\Lab\\Bereket_public\\custom_merfish_rig\\MERlin_epigen_UCSD\\.snakemake\\metadata\\VTovTGFiL0JlcmVrZXRfcHVibGljL2N1c3RvbV9tZXJmaXNoX3JpZy9NZXJsaW5fdGVzdC90ZXN0X2RhdGEvcmVzdWx0cy9kYXRhL0ZpZHVjaWFsQ29ycmVsYXRpb25XYXJwL3Rhc2tzL0ZpZHVjaWFsQ29ycmVsYXRpb25XYXJwXzAuZG9uZQ=='). Please ensure write permissions for the directory U:\Lab\Bereket_public\custom_merfish_rig\MERlin_epigen_UCSD\.snakemake

There are three common solutions to this problem:

Rewrite the Snakemake workflow, i.e. the software you are using, to use shorter paths. Or ask the developers for help. (Slow to implement solution, not necessarily stable and sustainable but permanent and it helps all users).
Move the software and your workflow to a directory with a shorter path. I.e. in your case from U:\Lab\Bereket_public\custom_merfish_rig\MERlin_epigen_UCSD to U:\Merlin_epigen_UCSD. This can but is not guaranteed to help. (Easy and quick to implement, not guaranteed to work)
You can also allow for longer paths in Windows, see this article for details. I've found this solution in the past to work reliably. (Requires some technical knowledge and elevated user privileges, does not transfer to other users or machines).

Unfortunately I'm not aware of any snakemake-based solution to handle this problem.