Search code examples
multithreadingcoldfusioncfhttp

Coldfusion limit to total number of threads

I've got some code that is trying to create 100 threaded http calls. It seems to be getting capped at about 40.

When I do threadJoin I'm only getting 38 - 40 sets of results from my http calls, despite the loop being from 1 - 100.

// thread http calls
pages = 100;

for (page="1";page <= pages; page++) {

    thread name="req#page#" {

        grabber.setURL('http://site.com/search.htm');
        // request headers
        grabber.addParam(type="url",name="page",value="#page#");
        results = grabber.send().getPrefix();

        arrayAppend(VARIABLES.arrResults,results.fileContent);
    }
}

// rejoin threads
for (page="2";page <= pages; page++) {
    threadJoin('req#page#',10000);
}

Is there a limit to the number of threads that CF can create? Is it to do with Java running in the background? Or can it not handle that many http requests?

Is there a just a much better way for me to do this than threaded HTTP calls?

Solution

  • The result you're seeing is likely because your variables aren't thread safe.

    grabber.addParam(type="url",name="page",value="#page#");

    That line is accessing Variables.Page which is shared by all of the spawned threads. Because threads start at different times, the value of page is often different from the value you think it is. This will lead to multiple threads having the same value for page.

    Instead, if you pass page as an attribute to the thread, then each thread will have its own version of the variable, and you will end up with 100 unique values. (1-100).

    Additionally you're writing to a shared variable as well.

    arrayAppend(VARIABLES.arrResults,results.fileContent);

    ArrayAppend is not thread safe and you will be overwriting versions of VARIABLES.arrResults with other versions of itself, instead of appending each bit.

    You want to set the result to a thread variable, and then access that once the joins are complete.

    thread name="req#page#" page=Variables.page {

    grabber.setURL('http://site.com/search.htm');
    // request headers
    grabber.addParam(type="url",name="page",value="#Attributes.page#");
    results = grabber.send().getPrefix();

    thread.Result = results.fileContent;
}

    And the join:

    // rejoin threads
for (page="2";page <= pages; page++) {
    threadJoin('req#page#',10000);
    arrayAppend(VARIABLES.arrResults, CFThread['req#page#'].Result);
}