Search code examples
c++boostboost-tokenizer

boost::tokenizer to consider absence of tokens between separators


I am using boost::tokenizer to get ';' separated fields from a string. I am able to retrieve the fields as shown in the code below but i have 2 questions:

  1. Is there any function which tokenizer provides to know the count of tokens in a string based on the separator provided?
  2. Supposing the test string has 3 fields a;b;c . The following piece of code will print all of them. But i need to print empty fields too. E.g. incase of a string a;;;b;c the token should also contain nothing as 2nd and 3rd element. Or in other words the 2nd and 3rd token should be empty.
#include <boost/tokenizer.hpp>
namespace std;
namespace boost;
int main()
{
    string data="a;;;;b;c";
    boost::char_separator<char> obj(";");
    boost::tokenizer<boost::char_separator<char> > tokens(data,obj);
    cout<<endl<<tokens.countTokens();
    for(boost::tokenizer<boost::char_separator<char> >::iterator it=tokens.begin();
    it!=tokens.end();
    ++it)
    {
        std::cout<<*it<<endl;
    }
}

Solution

  • 1) You can just count difference between end and begin.

    const size_t count = std::distance(tokens.begin(), tokens.end());
    

    2) You should just construct separator right.

    boost::char_separator<char> obj(";", "", boost::keep_empty_tokens);
    

    Live example