strtok() C-Strings to Array

Question

Currently learning C, Having some trouble with passing c-string tokens into array. Lines come in by standard input, strtok is used to split the line up, and I want to put each into an array properly. an EOF check is required for exiting the input stream. Here's what I have, set up so that it will print the tokens back to me (these tokens will be converted to ASCII in a different code segment, just trying to get this part to work first).

#include <stdio.h>
#include <stdlib.h>
#include <string.h>

int main()
{
  char string[1024]; //Initialize a char array of 1024 (input limit)

  char *token;
  char *token_arr[1024]; //array to store tokens.
  char *out; //used
  int count = 0;

  while(fgets(string, 1023, stdin) != NULL) //Read lines from standard input until EOF is detected.
  {
    if (count == 0)
      token = strtok(string, " \n"); //If first loop, Get the first token of current input

    while (token != NULL) //read tokens into the array and increment the counter until all tokens are stored
    {
      token_arr[count] = token;
      count++;
      token = strtok(NULL, " \n");
    }
  }

  for (int i = 0; i < count; i++)
    printf("%s\n", token_arr[i]);
  return 0;
}

this seems like proper logic to me, but then i'm still learning. The issue seems to be with streaming in multiple lines before sending the EOF signal with ctrl-D.

For example, given an input of:

this line will be fine

the program returns:

this line will be fine

But if given:

none of this

is going to work

It returns:

is going to work

ing to work

to work

any help is greatly appreciated. I'll keep working at it in the meantime.

Answer 1

There are a couple of issues here:

1. You never call token = strtok(string, " \n"); again once the string is "reset" to a new value, so strtok() still thinks it is tokenizing your original string.

2. strtok is returning pointers to "substrings" inside string. You are changing the contents of what is in string and so your second line effectively corrupts your first (since the original contents of string are overwritten).

To do what you want you need to either read each line into a different buffer or duplicate the strings returned by strtok (strdup() is one way - just remember to free() each copy...)