Maintaining N simultaneous HTTP downloads

Question

I have an application that queries a web service by a date range. The web service returns a list of GUID's. I then take each GUID and download a WAV file that it corresponds too. Right now I do it one by one and works fine. What I would like to do is download up to N number of WAV files simultaneously. For some reason I just can not grasp a good way to accomplish this.

I use IP*Works (n/Software) TipwHTTP(async) component.

Anyone have any suggestions to push me in the right direction.

Answer 1

Put each download in a separate thread and manage that list of downloads. You can use OmniThreadLibrary for instance to ease the thread programming. You can also look at my threading unit at Cromis which is a lot simpler but it may be enough for your case. It is very easy to use.

If you dislike threads you can also make an exe that takes input params when started and downloads the content to a specified location.

Just note that downloading many files simultaniously will only probably help if you download from different sources, if not you will still be limited by the bandwith of your only source and also have a threading overhead on you.

Here is a code with my threading unit and Indy for HTTP, just for example because it is easy to understand:

procedure TfMain.FormCreate(Sender: TObject);
var
  Task: ITask;
begin
  FTaskPool.DynamicSize := cbDynamicPoolSize.Checked;
  FTaskPool.MinPoolSize := StrToInt(ePoolSize.Text);
  FTaskPool.OnTaskMessage := OnTaskMessage;
  FTaskPool.Initialize;

  for I := 1 to NumberOfDownloads do
  begin
    Task := FTaskPool.AcquireTask(OnTaskExecute, 'HTTPDownload');
    Task.Values.Ensure('TargeFile').AsString := aFileName;
    Task.Values.Ensure('URL').AsString := aDownloadURL;
    Task.Run;
  end;
end;

procedure TfMain.OnTaskExecute(const Task: ITask);
var
  HTTPClient: TIdHTTP;
  FileStream: TFileStream;
begin
  HTTPClient := TIdHTTP.Create(nil);
  try
    FileStream := TFileStream.Create(Task.Values.Get('TargeFile').AsString, fmCreate);
    try
      HTTPClient.Get(Task.Values.Get('URL').AsInteger, FileStream);
      Task.Message.Ensure('Result').AsString := 'Success';
      Task.SendMessageAsync;
    finally
      FileStream.Free;
    end;   
  finally
    HTTPClient.Free;
  end;
end;

procedure TfMain.OnTaskMessage(const Msg: ITaskValues);
begin
  // do something when a single download has finished
end;