Search code examples
c++boostboost-tokenizer

What are the default character separators for boost::char_separator (for use with boost::tokenizer)?

The answer to this question seems like it should be easily obtainable by looking at the Boost documentation for char_separator, or by googling.

However, nowhere can I find the answer to this question: What are the default separators for boost::char_separator (for use with boost::tokenizer)?

Thanks!

Solution

  • http://www.boost.org/doc/libs/1_55_0/libs/tokenizer/introduc.htm

    ... If you do not specify anything, the default TokenizerFunction is char_delimiters_separator which defaults to breaking up a string based on space and punctuation. ...

    According to the source code, "punctuation" is defined as std::ispunct(E) != 0, while "space" is defined as std::isspace(E) != 0.