Exception with many iterations in webrequest

Question

The application iterates through around 500 webrequests, at random the request returns a 500-error from the server. I belive their is either a problem with the amount of requests made or that at some point the information takes to long to read into the datatable that causes the connection to fail. But these are just guesses on my part. Is there a smarter way to iterate through all of the requests or to use another approach?

I have other iterations made in the same manner for other requests, none of them are as hefty as this one and they don't throw any errors from the server.

foreach (DataRow row in dt.Rows)
{
                            string url = row["href"].ToString();
                            HttpWebRequest productsDetails = (HttpWebRequest)WebRequest.Create(url);
                            productsDetails.Credentials = nc;
                            productsDetails.Method = "GET";
                            productsDetails.Timeout = 5000;
                            productsDetails.ContentType = "application/x-www-form-urlencoded;charset=utf-8";

                            using (HttpWebResponse productsDetailsResponse = (HttpWebResponse)productsDetails.GetResponse())
                            {
                                var detailedRespons = productsDetailsResponse.GetResponseStream();

                                XDocument detailedResponsDoc = XDocument.Load(detailedRespons);

                                //foreach (XElement xe4 in detailedResponsDoc.Descendants("product_option_value"))
                                //{
                                //    DataRow row4 = dt4.NewRow();
                                //    row4["href"] = xe4.Attribute(xlink + "href").Value;
                                //    dt4.Rows.Add(row4);
                                //}

                                string p1 = detailedResponsDoc.Root.Element("combination").Element("id").Value;
                                string p2 = detailedResponsDoc.Root.Element("combination").Element("reference").Value;
                                string p3 = detailedResponsDoc.Root.Element("combination").Element("price").Value;
                                string p4;

                                foreach (XElement xe2 in detailedResponsDoc.Descendants("product_option_value"))
                                {
                                    p4 = (xe2.Value);

                                    DataRow row5 = test.NewRow();
                                    row5["id"] = p1;
                                    row5["referemce"] = p2;
                                    row5["price"] = p3;
                                    row5["POV"] = p4;
                                    test.Rows.Add(row5);

                                    DataRow row4 = dt4.NewRow();
                                    row4["href"] = xe2.Attribute(xlink + "href").Value;
                                    dt4.Rows.Add(row4);
                                }
                                productsDetailsResponse.Close();
                            }
                        }
                    }
                }
            }
            catch (WebException webex)
            {
                WebResponse errResp = webex.Response;
                using (Stream respStream = errResp.GetResponseStream())
                {
                    StreamReader reader = new StreamReader(respStream);
                    string text = reader.ReadToEnd();
                    MessageBox.Show("yttre:" + text);
                }
            }

The error message is a a generic 500 exception from the server, referring me to contact the host. The host don't see anything and the little i have found in some kind of error log on the server don't contain any information.

Answer 1

Make sure the server is not blocking you, some servers have firewalls that block repetitive connections from a single IP address as they believe it will be an attack.

This is a normal and often cannot be disabled by hosts as it is a security feature.

Add a delay to the requests and see if the server responds correctly, if this works, then the server may be blocking you.

Try to make similar requests on a local server like XAMP, if the same errors occur this could be a code fault, like the information being passed to the server (Headers, Post, Get and etc).

Try reusing HttpWebRequest to avoid overhead if you repeatedly create an object, try using asynchronous methods.

There are many variables as to why there might be errors, but the chance of being server-related or HttpWebRequest is the most likely.