Search code examples
rbatch-filewindows-task-scheduler

Schedule task to run multiple overlapping instances of the same R script via batch file

I am trying to run an R script every hour that downloads data from an API, manipulate it, and writes out some product. Each run takes 3-5h, but I can't seem to schedule a task that will successfully run multiple instances at the same time.

  • I use the Windows Task Scheduler to plan a task which triggers at a given time and is then repeated every 1 hour.

  • I run it whether logged on or not with the highest privilege (no issue with that)

  • The action involves running a .bat file which contains the following text:

    @echo on start "" "C:\Program Files\R\R-3.4.2\bin\x64\R.exe" CMD BATCH --vanilla F:\storm\src\download_rasters.R

  • If the task is already running, I set it to 'Run a new instance in parallel'

Now, I'm new to batch files and task scheduling, but here are some troubleshooting clues:

  • If I schedule the task to run hourly, the code will run successfully the first time, but it will not run again before the first instance is finished (after about 4h), and so will end up running only every 4h.
  • I tried to launch two separate instances of the R Studio GUI and ran the code in the two separate instances: it worked. This indicates that the code itself doesn't lock files and the API can be called simultaneously with the same key.
  • I scheduled two different tasks, calling two different batch files, each calling a different script file with a different output directory: it worked.
  • I scheduled two different tasks, calling the same batch file at 5-minute interval from each other: the first ran, the second didn't.
  • I also tried Rcmd.exe just in case based on another thread, but it didn't solve the issue.

My guess is that I am missing something in the batch file that would allow it to run separate instances of R from the same batch file, but I can't figure out what it is.

Solution

  • I got it to work by using the batch file to directly run an R command sourcing the script.

    @echo off "C:\Program Files\R\R-3.4.2\bin\x64\R.exe" -e source('F:/storm\src\download_rasters.R')

    It's not great so better solutions are welcome.