Search code examples
c#asynchronousasync-awaittask-parallel-libraryparallel.foreach

How to download correctly files simultaneously?

I am trying to download mltiple files simultaneosly. But all files are downloading one by one, sequantilly. So, at first this file downloaded @"http://download.geofabrik.de/europe/cyprus-latest.osm.pbf", and then this file is started to dowload @"http://download.geofabrik.de/europe/finland-latest.osm.pbf",, and the next file to be downloaded is @"http://download.geofabrik.de/europe/great-britain-latest.osm.pbf" and so on.

But I would like to download simultaneously.

So I've the following code based on the code from this answer:

static void Main(string[] args)
{
    Task.Run(async () =>
    {
        await DownloadFiles();
    }).GetAwaiter().GetResult();
}

public static async Task DownloadFiles()
{
    IList<string> urls = new List<string>
    {
        @"http://download.geofabrik.de/europe/cyprus-latest.osm.pbf",
        @"http://download.geofabrik.de/europe/finland-latest.osm.pbf",
        @"http://download.geofabrik.de/europe/great-britain-latest.osm.pbf",
        @"http://download.geofabrik.de/europe/belgium-latest.osm.pbf",
        @"http://download.geofabrik.de/europe/belgium-latest.osm.pbf"
    };

    foreach (var url in urls)
    {
        string fileName = url.Substring(url.LastIndexOf('/'));
        await DownloadFile(url, fileName);
    }            
}

public static async Task DownloadFile(string url, string fileName)
{
    string address = @"D:\Downloads";
    using (var client = new WebClient())
    {
        await client.DownloadFileTaskAsync(url, $"{address}{fileName}");
    }
}

However, when I see in my file system, then I see that files are downloading one by one, sequantially, not simultaneosuly:

enter image description here

In addition, I've tried to use this approach, however there are no simultaneous downloads:

static void Main(string[] args)
{
    IList<string> urls = new List<string>
    {
        @"http://download.geofabrik.de/europe/cyprus-latest.osm.pbf",
        @"http://download.geofabrik.de/europe/finland-latest.osm.pbf",
        @"http://download.geofabrik.de/europe/great-britain-latest.osm.pbf",
        @"http://download.geofabrik.de/europe/belgium-latest.osm.pbf",
        @"http://download.geofabrik.de/europe/belgium-latest.osm.pbf"
    };

    Parallel.ForEach(urls,
        new ParallelOptions { MaxDegreeOfParallelism = 10 },
        DownloadFile);
}

public static void DownloadFile(string url)
{
    string address = @"D:\Downloads";
    using (var sr = new StreamReader(WebRequest.Create(url)
           .GetResponse().GetResponseStream()))
    using (var sw = new StreamWriter(address + url.Substring(url.LastIndexOf('/'))))
    {
        sw.Write(sr.ReadToEnd());
    }
}

Could you tell me how it is possible to download simultaneosly?

Any help would be greatly appreciated.

Solution 
  •     foreach (var url in urls)
    {
        string fileName = url.Substring(url.LastIndexOf('/'));
        await DownloadFile(url, fileName); // you wait to download the item and then move the next
    }

    Instead you should create tasks and wait all of them to complete.

    public static Task DownloadFiles()
{
    IList<string> urls = new List<string>
    {
        @"http://download.geofabrik.de/europe/cyprus-latest.osm.pbf",
        @"http://download.geofabrik.de/europe/finland-latest.osm.pbf",
        @"http://download.geofabrik.de/europe/great-britain-latest.osm.pbf",
        @"http://download.geofabrik.de/europe/belgium-latest.osm.pbf",
        @"http://download.geofabrik.de/europe/belgium-latest.osm.pbf"
    };

       var tasks = urls.Select(url=> {
           var fileName = url.Substring(url.LastIndexOf('/'));
             return DownloadFile(url, fileName);    
         }).ToArray();   
         return Task.WhenAll(tasks);       
}

    Rest of your code can remain same.