Why is 6-7 threads faster than 20?

Question

In school we were introduced to C++11 threads. The teacher gave us a simple assessment to complete which was to make a basic web crawler using 20 threads. To me threading is pretty new, although I do understand the basics.

I would like to mention that I am not looking for someone to complete my assessment as it is already done. I only want to understand the reason why using 6 threads is always faster than using 20.

Please see code sample below.

main.cpp:

do
{
    for (size_t i = 0; i < THREAD_COUNT; i++)
    {
        threads[i] = std::thread(SweepUrlList);
    }

    for (size_t i = 0; i < THREAD_COUNT; i++)
    {
        threads[i].join();
    }

    std::cout << std::endl;
    WriteToConsole();
    listUrl = listNewUrl;
    listNewUrl.clear();
} while (listUrl.size() != 0);

Basically this assigns to each worker thread the job to complete which is the method SweepUrlList that can be found below and then join all thread.

while (1)
{
    mutextGetNextUrl.lock();
    std::set<std::string>::iterator it = listUrl.begin();
    if (it == listUrl.end())
    {
        mutextGetNextUrl.unlock();
        break;
    }
    std::string url(*it);
    listUrl.erase(*it);
    mutextGetNextUrl.unlock();
    ExtractEmail(url, listEmail);
    std::cout << ".";
}

So each worker thread loop until ListUrl is empty. ExtractEmail is a method that downloads the webpage (using curl) and parse it to extract emails from mailto links. The only blocking call in ExtractEmail can be found below:

if(email.length() != 0)
{
    mutextInsertNewEmail.lock();
    ListEmail.insert(email);
    mutextInsertNewEmail.unlock();
}

All answers are welcome and if possible links to any documentation you found to answer this question.

Answer 1

This is a fairly universal problem with threading, and at its core:

What you are demonstrating is thread Scheduling. The operating system is going to work with the various threads, and schedule work where there is currently not work.

Assuming you have 4 cores and hyper threading you have 8 processors that can carry the load, but also that of other applications (Operating System, C++ debugger, and your application to start).

In theory, you would probably be OK on performance up until about 8 intensive threads. After you reach the most threads your processor can effectively use, then threads begin to compete against each other for resources. This can be seen (especially with intensive applications and tight loops) by poor performance.

Finally, this is a simplified answer but I suspect what you are seeing.