I'm new to using the pthread library in C and I have an assignment for my class to write a simple program using them. The basic description of the program is it takes 1 or more input files containing website names and 1 output file name. I then need to create 1 thread per input file to read in the website names and push them onto a queue. Then I need to create a couple of threads to pull those names off of the queue, find their IP Address, and then write that information out to the output file. The command line arguments are expected as follows:
./multi-lookup [one or more input files] [single output file name]
My issue is this. Whenever I run the program with only 1 thread to push information to the output file then everything works properly. When I make it two threads then the program hangs and none of my testing "printf" statements are even printed. My best guess is that deadlock is occurring somehow and that I'm not using my mutexes properly but I can't figure out how to fix it. Please help!
If you need any information that I'm not providing then just let me know. Sorry for the lack of comments in the code.
#include <stdlib.h>
#include <stdio.h>
#include <string.h>
#include <errno.h>
#include <pthread.h>
#include "util.h"
#include "queue.h"
#define STRING_SIZE 1025
#define INPUTFS "%1024s"
#define USAGE "<inputFilePath> <outputFilePath>"
#define NUM_RESOLVERS 2
queue q;
pthread_mutex_t locks[2];
int requestors_finished;
void* requestors(void* input_file);
void* resolvers(void* output_file);
int main(int argc, char* argv[])
{
FILE* inputfp = NULL;
FILE* outputfp = NULL;
char errorstr[STRING_SIZE];
pthread_t requestor_threads[argc - 2];
pthread_t resolver_threads[NUM_RESOLVERS];
int return_code;
requestors_finished = 0;
if(queue_init(&q, 10) == QUEUE_FAILURE)
fprintf(stderr, "Error: queue_init failed!\n");
if(argc < 3)
{
fprintf(stderr, "Not enough arguments: %d\n", (argc - 1));
fprintf(stderr, "Usage:\n %s %s\n", argv[0], USAGE);
return 1;
}
pthread_mutex_init(&locks[0], NULL);
pthread_mutex_init(&locks[1], NULL);
int i;
for(i = 0; i < (argc - 2); i++)
{
inputfp = fopen(argv[i+1], "r");
if(!inputfp)
{
sprintf(errorstr, "Error Opening Input File: %s", argv[i]);
perror(errorstr);
break;
}
return_code = pthread_create(&(requestor_threads[i]), NULL, requestors, inputfp);
if(return_code)
{
printf("ERROR: return code from pthread_create() is %d\n", return_code);
exit(1);
}
}
outputfp = fopen(argv[i+1], "w");
if(!outputfp)
{
sprintf(errorstr, "Errord opening Output File: %s", argv[i+1]);
perror(errorstr);
exit(1);
}
for(i = 0; i < NUM_RESOLVERS; i++)
{
return_code = pthread_create(&(resolver_threads[i]), NULL, resolvers, outputfp);
if(return_code)
{
printf("ERROR: return code from pthread_create() is %d\n", return_code);
exit(1);
}
}
for(i = 0; i < (argc - 2); i++)
pthread_join(requestor_threads[i], NULL);
requestors_finished = 1;
for(i = 0; i < NUM_RESOLVERS; i++)
pthread_join(resolver_threads[i], NULL);
pthread_mutex_destroy(&locks[0]);
pthread_mutex_destroy(&locks[1]);
return 0;
}
void* requestors(void* input_file)
{
char* hostname = (char*) malloc(STRING_SIZE);
FILE* input = input_file;
while(fscanf(input, INPUTFS, hostname) > 0)
{
while(queue_is_full(&q))
usleep((rand()%100));
if(!queue_is_full(&q))
{
pthread_mutex_lock(&locks[0]);
if(queue_push(&q, (void*)hostname) == QUEUE_FAILURE)
fprintf(stderr, "Error: queue_push failed on %s\n", hostname);
pthread_mutex_unlock(&locks[0]);
}
hostname = (char*) malloc(STRING_SIZE);
}
printf("%d\n", queue_is_full(&q));
free(hostname);
fclose(input);
pthread_exit(NULL);
}
void* resolvers(void* output_file)
{
char* hostname;
char ipstr[INET6_ADDRSTRLEN];
FILE* output = output_file;
int is_empty = queue_is_empty(&q);
//while(!queue_is_empty(&q) && !requestors_finished)
while((!requestors_finished) || (!is_empty))
{
while(is_empty)
usleep((rand()%100));
pthread_mutex_lock(&locks[0]);
hostname = (char*) queue_pop(&q);
pthread_mutex_unlock(&locks[0]);
if(dnslookup(hostname, ipstr, sizeof(ipstr)) == UTIL_FAILURE)
{
fprintf(stderr, "DNSlookup error: %s\n", hostname);
strncpy(ipstr, "", sizeof(ipstr));
}
pthread_mutex_lock(&locks[1]);
fprintf(output, "%s,%s\n", hostname, ipstr);
pthread_mutex_unlock(&locks[1]);
free(hostname);
is_empty = queue_is_empty(&q);
}
pthread_exit(NULL);
}
Although I'm not familiar with your "queue.h" library, you need to pay attention to the following:
When you check whether your queue is empty you are not acquiring the mutex, meaning that the following scenario might happen:
requestors thread checks for emptiness (let's call it thread1) and just before it executes pthread_mutex_lock(&locks[0]); (and after if(!queue_is_full(&q)) ) thread1 gets contex switched requestors threads fill the queue up and when out thread1 finally gets hold of the mutex if will try to insert to the full queue. Now if your queue implementation crashes when one tries to insert more elements into an already full queue thread1 will never unlock the mutex and you'll have a deadlock.Another scenario:
resolver thread runs first requestors_finished is initially 0 so (!requestors_finished) || (!is_empty) is initially true. is_empty is true.while(is_empty) usleep((rand()%100)); and sleep forever, because you pthread_join this thread your program will never terminate because this value is never updated in the loop.The general idea to remember is that when you access some resource that is not atomic and might be accessed by other threads you need to make sure you're the only one performing actions on this resource.
Using a mutex is OK but you should consider that you cannot anticipate when will a context switch occur, so if you want to chech e.g whether the queue is empty you should do this while having the mutex locked and not unlock it until you're finished with it otherwise there's no guarantee that it'll stay empty when the next line executes.
You might also want to consider reading more about the consumer producer problem.
To help you know (and control) when the consumers (resolver) threads should run and when the producer threads produce you should consider using conditional variables.
Some misc. stuff:
pthread_t requestor_threads[argc - 2]; is using VLA and not in a good way - think what will happen if I give no parameters to your program. Either decide on some maximum and define it or create it dynamically after having checked the validity of the input.requestors threads should open the file themselves There might be some more problems but start by fixing those.