APR joing thread pool threads when using apr_thread_pool_schedule

gcc 4.7.2
c89
apr utility 1.4

Hello,

I am using a thread pool to start threads. However, I can't see any apr function that allows me to wait for the threads to join.

The code sippet, removed all error checking and non-essential parts:

int main(void)
{
    /* Initialize apr internal structures */
    apr_initialize();

    /* Create memory pool */
    rv = apr_pool_create(&mem_pool, NULL);

    /* Create thread pool */
    memset(&buf, 0, sizeof buf);
    rv = apr_thread_pool_create(&thd_pool,
                                init_threads,
                                max_threads,
                                mem_pool);

    /* Process the number of jobs */
#define NUMBER_JOBS 1
    for(i = 0; i < NUMBER_JOBS; i++) {
        rv = apr_thread_pool_schedule(thd_pool,
                                      timeout_duration,
                                      (void*)channel,
                                      (apr_interval_time_t)flash_timeout,
                                      NULL);

    }

    /* 
     * Join all threads here 
     */

    /* Destroy resources */
    apr_thread_pool_destroy(thd_pool);
    apr_pool_destroy(mem_pool);
    apr_terminate();

    return 0;
error:
    apr_thread_pool_destroy(thd_pool);
    apr_pool_destroy(mem_pool);
    apr_terminate();

    return 1;
}

void* timeout_duration(apr_thread_t *thd, void *data)
{
    channel_t *channel = (channel_t*)data;

    LOG_DEBUG("Channel timeout notification [ %zu ]", channel->id);
}

I couldn't see any apr utity functions that join threads.

However, I did find this function apr_thread_join(apr_status_t *retval, apr_thread_t *thd) However, it takes a apr_thread_t as an argument.

The function timeout_duration takes a apr_thread_t but how can I manage to pass it back, if I need to use it for joining?

Just a side note question. Is there any sample projects that use the apr and I can reference. The documentation is very limited.

Many thanks for any suggestions,

Solution

Short anser

You don't need to join the threads in the thread-pool. When you call apr_thread_pool_destroy the function will block, until all threads have finished their current task.

To answer your last question first: I didn't find an example but the libapr and libapr-util are open-source, you can read the source and this is what I did: (I checked the SVN-trunk here rev 1441871)

Long answer

The interesting files:

First check in apr_thread_pool.c:394. Here we find the implementation of apr_thread_pool_destroy. We can see the it calls a function called apr_pool_cleanup_run with three arguments, one is the pool-storage, one is the thread-pool-context and the last one is a function-pointer to the function thread_pool_cleanup.

If we follow apr_pool_cleanup_run we will get to apr_pools.c:2453 and see that apr_pool_cleanup_kill is called. Reading this last function shows us, that here in several loops over the elements (the threads) are cleaned by (what we will see later) calling the cleanup_fn-function-argument.

Now back in function apr_pool_cleanup_run there is a final call to cleanup_fn.

The real action is going on the function-pointer passed to apr_pool_cleanup_run. So, if we go back to apr_thread_pool.c:329 we find the function thread_pool_cleanup.

In it the context-variable terminated is set to 1 and then the function is "sleeping" until _myself->thd_cnt has become 0.

Searching for the usage of terminated we find that thread_pool_func is exiting its loop when terminated is not 0. It turns out that thread_pool_func is the function each thread in the threadpool is using. In the loop a task is fetched and executed. When the loop has terminated (because terminated has become 1) the following code is executed:

 /* idle thread been asked to stop, will be joined */
 --me->thd_cnt;

This will ultimately lead to the thd_cnt==0 which is the terminate-condition for the loop in thread_pool_cleanup.

When you call apr_thread_pool_destroy all threads are stopped cleanly before the function returns.