Async download function, how to do it? VB.NET

Question

I am working on coronavirus statistics dashboard as university project, and I have some problems with asynchronous source data download from sites with statistics. Well, I failed to understand how to do it myself.

I tried to create my own class with function what will create multiple async web requests and then wait until they all finished, then return results of all these requests.

Imports System.Net.WebClient
Imports System.Net
Public Class AsyncDownload
    Private result As New Collection
    Private Sub DownloadCompletedHander(ByVal sender As Object, ByVal e As System.Net.DownloadStringCompletedEventArgs)
        If e.Cancelled = False AndAlso e.Error Is Nothing Then
            Dim myString As String = CStr(e.Result)
            result.Add(myString, sender.Headers.Item("source"))
        End If
    End Sub

    Public Function Load(sources As Array, keys As Array) As Collection
        Dim i = 0
        Dim WebClients As New Collection
        While (i < sources.Length)
            Dim newClient As New WebClient
            newClient.Headers.Add("source", keys(i))
            newClient.Headers.Add("sourceURL", sources(i))
            AddHandler newClient.DownloadStringCompleted, AddressOf DownloadCompletedHander
            WebClients.Add(newClient)
            i = i + 1
        End While
        i = 1
        For Each client As WebClient In WebClients
            Dim url As String = client.Headers.Item("sourceURL")
            client.DownloadStringAsync(New Uri(url))
        Next
        While (result.Count < WebClients.Count)
        End While
        Return result
    End Function
End Class

And it is used in:

    Dim result As New Collection
    Private Sub test() Handles Me.Load
        Dim downloader As New CoronaStatisticsGetter.AsyncDownload
        result = downloader.Load({"https://opendata.digilugu.ee/covid19/vaccination/v3/opendata_covid19_vaccination_total.json"}, {"Nationalwide Data"})
    End Sub

It should work like:

1. I create a new instance of my class.
2. Calling function Load of this class
3. Funciton Load creates instances of System.Net.WebClient for each url and adds as handler DownloadCompletedHander
4. Function Load goes calls DownloadStringAsync of each client
5. Function Load waits in While loop until result collection items count is not as big as number of url on input
6. If item count in result is same as urls number that means what everything is downloaded, so it breaks loop and returns all requested data

The problem is that it doesn't work, it just endlessly remain in while loop, and as I see using debug collection result is not updated (its size is always 0)

Same time, when I try to asynchronously download it without using my class, everything works fine:

Private Sub Download() 'Handles Me.Load
        Dim wc As New System.Net.WebClient
        wc.Headers.Add("source", "VaccinationByAgeGroup")
        AddHandler wc.DownloadStringCompleted, AddressOf DownloadCompletedHander
        wc.DownloadStringAsync(New Uri("https://opendata.digilugu.ee/covid19/vaccination/v3/opendata_covid19_vaccination_agegroup.json"))
End Sub

Could somebody tell me please why it is not working and where is the problem?

Answer 1

The following shows how one can use System.Net.WebClient with Task to download a string (ie: data) from a URL.

Add a project reference (System.Net)

VS 2019:

• In VS menu, click Project
• Select Add reference...
• Select Assemblies
• Check System.Net
• Click OK

Create a class (name: DownloadedData.vb)

Public Class DownloadedData
    Public Property Data As String
    Public Property Url As String
End Class

Create a class (name: HelperWebClient.vb)

Public Class HelperWebClient

    Public Async Function DownloadDataAsync(urls As List(Of String)) As Task(Of List(Of DownloadedData))
        Dim allTasks As List(Of Task) = New List(Of Task)
        Dim downloadedDataList As List(Of DownloadedData) = New List(Of DownloadedData)

        For i As Integer = 0 To urls.Count - 1
            'set value
            Dim url As String = urls(i)
            Debug.WriteLine(String.Format("[{0}]: Adding {1}", i, url))

            Dim t = Task.Run(Async Function()
                                 'create new instance
                                 Dim wc As WebClient = New WebClient()

                                 'await download
                                 Dim result = Await wc.DownloadStringTaskAsync(url)
                                 Debug.WriteLine(url & " download complete")

                                 'ToDo: add desired code
                                 'add
                                 downloadedDataList.Add(New DownloadedData() With {.Url = url, .Data = result})
                             End Function)
            'add
            allTasks.Add(t)
        Next

        For i As Integer = 0 To allTasks.Count - 1
            'wait for a task to complete
            Dim t = Await Task.WhenAny(allTasks)

            'remove from List
            allTasks.Remove(t)

            'write data to file
            'Note: The following is only for testing.
            'The index in urls won't necessarily correspond to the filename below
            Dim filename As String = System.IO.Path.Combine("C:\Temp", String.Format("CoronavirusData_{0:00}.txt", i))
            System.IO.File.WriteAllText(filename, downloadedDataList(i).Data)

            Debug.WriteLine($"[{i}]: Filename: {filename}")
        Next

        Debug.WriteLine("all tasks complete")

        Return downloadedDataList
    End Function
End Class

Usage:

Private Async Sub btnRun_Click(sender As Object, e As EventArgs) Handles btnRun.Click
    Dim helper As HelperWebClient = New HelperWebClient()

    Dim urls As List(Of String) = New List(Of String)
    urls.Add("https://opendata.digilugu.ee/covid19/vaccination/v3/opendata_covid19_vaccination_total.json")
    urls.Add("https://api.covidtracking.com/v2/states.json")
    urls.Add("https://covidtrackerapi.bsg.ox.ac.uk/api/v2/stringency/date-range/2020-01-01/2022-03-01")
    urls.Add("http://covidsurvey.mit.edu:5000/query?age=20-30&gender=all&country=US&signal=locations_would_attend")

    Dim downloadedDataList = Await helper.DownloadDataAsync(urls)
    Debug.WriteLine("Complete")
End Sub

Resources:

• How do I wait for something to finish in C#?
• How should Task.Run call an async method in VB.NET?
• VB.net ContinueWith