Most efficient way to make a large number of small POSTs to web service

Question

I need to send approximately 10,000 small json strings from a C# client application to a web service which will then be inserted into a database.

My question: Is it better to send a large number of small requests or is there some way to send the whole thing /multiple large chunks?

I'm trying to avoid something like this:

List<Thing> things_to_update = get_things_to_update(); // List now contains 10,000 records
foreach (Thing th in things_to_update)
{
    //  POSTs using HttpWebRequest or similar
    post_to_web_service(th);
}

Answer 1

If you control the server and can change the code, it's definitely better to send them batched.

Just encode your objects into json object and put them in an array. So instead of sending data like

data={ id: 5, name: "John" }

make it an array

data=[{ id: 5, name: "John" }, { id: 6, name: "Jane" }, ...]

then parse it in your Controller action on the server side and insert it. You should create a new Action that handles more than 1 request for sake of cleaner and more maintainable code.

I'd suggest splitting it into smaller batches of 1000 or 2000 rather than sending all 10000 at once. Easy done with Linq:

int batchSize = 1000;
for(int i=0; i<things_to_update.Count; i+= batchSize) {
    List<Thing> batch = things_to_update.Skip(i).Take(batchSize);
    post_to_web_service(batch);
}