Search code examples
cstringpointerstokenize

strtok showing undesirable behaviour


Following is my code fragment:

int main()
{
   char *str[4];
   char *ptr;

   char Str[25];
   char Str1[25];
   memset(Str,32,25);
   char temp[25];
   sprintf(Str,"123;;False;CXL;;true");
   printf("Old String [%s]",Str);
   ptr = &Str ;
   str[0]=strtok(ptr,";");
   str[1]=strtok(NULL,";");
   str[2]=strtok(NULL,";");
   str[3]=strtok(NULL,";");

   printf("str[0] =[%s]",str[0]);
   printf("str[1] =[%s]",str[1]);
   printf("str[2] =[%s]",str[2]);
   printf("str[3] =[%s]",str[3]);

   //strncpy(temp,str,25);
   memcpy(str[0],"345",3);
   sprintf(Str1,"%s;%s;%s;%s",str[0],str[1],str[2],str[3]);
   printf("New String [%s]",Str1);
   return 0;
}

My concern is why is it ignoring the 'null' values in my original string? As per the output I get 4 tokens but in fact I have 6 tokens with delimiter ';', it's ignoring the 'null' values in b/w and not considering them as separate tokens.

Output is:

Old String [123;;False;CXL;;true]
str[0] =[123]str[1] =[False]str[2] =[CXL]str[3] =[true]
New String [345;False;CXL;true]

Is there a workaround for this that I'm not aware of?

Thanks!


Solution

  • strsep MAN says

    The strsep() function is intended as a replacement for the strtok() function. While the strtok() function should be preferred for portability reasons (it conforms to ISO/IEC 9899:1990 ("ISO C90")) it is unable to handle empty fields, i.e., detect fields delimited by two adjacent delimiter characters, or to be used for more than a single string at a time. The strsep() function first appeared in 4.4BSD.

    Example

    char *t, *str, *save;
    
    save = str = strdup("123;;False;CXL;;true");
    while ((t = strsep(&str, ";")) != NULL)
        printf("Token=%s\n", t);
    
    free(save);
    

    Output

    Token=123
    Token=
    Token=False
    Token=CXL
    Token=
    Token=true