cfhttp in cfloop limit? use cfthread

Question

I'm currently trying to develop an API and the stage where I'm at just now is to populate a table with a full data set (ID, first name, last name, dob etc).

The way I've written this is to use a cfloop from 1 to 500,000 (as I don't know what range the IDs range from and to) and within each cfloop I call a function that makes a cfhttp request to the server and retrieve the content.

I then deserialize the returned JSON, call a function to query my table to see if the current item ID already exists and, if not, call a function to insert the record.

However the cfloop seems to stop around the 300 request mark so I was wondering if there was a better way to do what I'm doing? Perhaps by using the CFTHREAD tag which I've never had any experience of using.

The section of code for this is as follows:

<cfset Variables.url = "someurl.html" />
<cfloop from=100000 to=500000 index="itemNo">
    <cfset Variables.itemID = itemNo />
    <cfset getItemData = Application.cfcs.Person.getPersonData(Variables.url,Variables.itemID) />
    <cfif StructKeyExists(Variables,"getPersonData.FileContent")>
        <cfset Variables.getPersonData = DeserializeJSON(getPersonData.FileContent)>
        <cfscript>
            // CHECK IF PERSON ALREADY IN DATABASE
            Variables.getPerson = Application.cfcs.Person.getPersonRecord(Variables.itemID);
            // INSERT ITEM IN TO TABLE
            Variables.DOB = CreateDate(Year(Variables.getPersonData.Item.DateOfBirth.Year),Month(Variables.getPersonData.Item.DateOfBirth.Month),Day(Variables.getPersonData.Item.DateOfBirth.Day));
            Variables.insPerson = Application.cfcs.Person.insPerson(Variables.getPersonData.personID,Variables.getPersonData.Item.FirstName,Variables.getPersonData.Item.LastName,Variables.getPersonData.Item.CommonName,Variables.DOB);   
        </cfscript>
    </cfif>
</cfloop>

Answer 1

Yes it is possible. You need to split up the call. Create a simple htlm page which makes a xmlhttprequest in javascript. I haven't tested the example below but it should work.

<script>
var itemNo= 1;
function download()
{
 var xhr = new XMLHttpRequest();
 xhr.open("GET", "getdata.cfm?itemNo="+itemNo, true);
 xhr.onload = function (e) {
  if (xhr.readyState === 4) {
   if (xhr.status === 200) 
   {
     itemNo++;
     if(itemNo<=500000) download();
   }
   else 
   {
     itemNo++;
    // Error handling
   }
  }
 };
 xhr.onerror = function (e) {
      itemNo++;
 // Error handling
 };
 xhr.send(null);
}
</script>

On the requested page make the the call to the object that makes the cfhttp request.

<!--- getdata.cfm --->
<cfset Variables.url = "someurl.html" />
<cfset Variables.itemID = itemNo />
<cfset getItemData = Application.cfcs.Person.getPersonData(Variables.url,Variables.itemID) />
<cfif StructKeyExists(Variables,"getPersonData.FileContent")>
    <cfset Variables.getPersonData = DeserializeJSON(getPersonData.FileContent)>
    <cfscript>
        // CHECK IF PERSON ALREADY IN DATABASE
        Variables.getPerson = Application.cfcs.Person.getPersonRecord(Variables.itemID);
        // INSERT ITEM IN TO TABLE
        Variables.DOB = CreateDate(Year(Variables.getPersonData.Item.DateOfBirth.Year),Month(Variables.getPersonData.Item.DateOfBirth.Month),Day(Variables.getPersonData.Item.DateOfBirth.Day));
        Variables.insPerson = Application.cfcs.Person.insPerson(Variables.getPersonData.personID,Variables.getPersonData.Item.FirstName,Variables.getPersonData.Item.LastName,Variables.getPersonData.Item.CommonName,Variables.DOB);   
    </cfscript>
</cfif>

On the requested page you could use cfhtread to make multiple http request simultaneously. You can look here for more information about using cfthread together with cfhttp http://www.bennadel.com/blog/749-Learning-ColdFusion-8-CFThread-Part-II-Parallel-Threads.htm