c++ strtok skips second token or consecutive delimiter

Question

I am trying to read CSV comma delimited file, content of file are

      one,,three

And code to read file is this…

      inFile.getline(line, 500);                            
      token1 = strtok(line, ",");
      token2 = strtok(NULL, ",");
      token3 = strtok(NULL, ",");

      if(token1 != NULL){
             cout << "token1 = " << token1 << "
";      
      }else{ 
             cout << "token1 = null
" ;
      }
      if(token2 != NULL){ 
             cout << "token2 = " << token2 << "
";      
      }else{ 
             cout << "token2 = null
" ;
      } 
      if(token3 != NULL){ 
             cout << "token3 = " << token3 << "
";      
      }else{ 
             cout << "token3 = null
";
      }

Output is this

token1 = one
token2 = three
token3 = null

Whereas my expectation are that output should be like this…

token1 = one
token2 = null
token3 = three

I did change if statements from

     if(token1 != NULL) 

To

     if(token1)

But it as well doesn’t works.

After checking this example http://www.cplusplus.com/reference/cstring/strtok/, I have updated

   token2 = strtok(NULL, ",");

To

   token2 = strtok(NULL, ",,");

As well it does not works

Answer 1

Once I did face this problem while reading CSV comma delimited file. But we can't use strtok() as our solution in such problems where the delimiter character appears consecutively. Because according to the standard

The first call in the sequence searches the string pointed to by s1 for the first character that is not contained in the current separator string pointed to by s2. If no such character is found, then there are no tokens in the string pointed to by s1 and the strtok function returns a null pointer. If such a character is found, it is the start of the first token. C11 §7.24.5.8 3

So, for my case I defined another solution using strpbrk() function which will also be useful for you.

#include<iostream.h>

char *strtok_new(char * string, char const * delimiter){
   static char *source = NULL;
   char *p, *riturn = 0;
   if(string != NULL)         source = string;
   if(source == NULL)         return NULL;

   if((p = strpbrk (source, delimiter)) != NULL) {
      *p  = 0;
      riturn = source;
      source = ++p;
   }
return riturn;
}

int main(){
   char string[] = "one,,three,";
   char delimiter[] = ",";
   char * p    = strtok_new(string, delimiter);

   while(p){
            if(*p)  cout << p << endl;
            else    cout << "No data" << endl;                
            p = strtok_new(NULL, delimiter);
   }
   system("pause");
   return 0;
}

Output

one
No data
three

Hope this is your desired output.