aria2c timeout error on large file, but downloads most of it

I posted this on superuser yesterday, but maybe it's better here. I apologize if I am incorrect.

I am trying to trouble shoot the below aria2c command run using ubuntu 14.04. Basically the download starts and gets to about 30 minutes left, then errors with a timeout error. The file being downloaded is 25GB and there is often multiple files that are downloaded using a loop. Any suggestions to make this more efficient and stable? Currently, the each file takes about 4 hours to download, which is ok as long as there are no errors. I do get an aria2c file with the partial dowloaded file as well. Thank you :).

aria2c -x 4 -l log.txt -c -d /home/cmccabe/Desktop/download --http-user "xxxxx"  --http-passwd xxxx xxx://www.example.com/x/x/xxx/"file"

I apologize for the tag as I am not able to create a new one, that was the closest.

Solution

You can write a loop to rerun aria2c until all files are download (see this gist).

Basically, you can put all the links in a file (e.g. list.txt):

http://foo/file1
http://foo/file2
...

Then run loop_aria2.sh:

#!/bin/bash
aria2c -j5 -i list.txt -c --save-session out.txt
has_error=`wc -l < out.txt`

while [ $has_error -gt 0 ]
do
  echo "still has $has_error errors, rerun aria2 to download ..."
  aria2c -j5 -i list.txt -c --save-session out.txt
  has_error=`wc -l < out.txt`
  sleep 10
done

aria2c -j5 -i list.txt -c --save-session out.txt will downloads 5 files in parallel (-j5) and writes the failed files into out.txt. If the out.txt is not empty ($has_error -gt 0), then rerun the same command to continue downloading. The -c option of aria2c will skip completed files.

PS: another simpler solution (without checking error), which just run the aria2 1000 times if you don't mind :)

seq 1000 | parallel -j1 aria2c -i list.txt -c