Regex (JS Notation): Select spaces not in [ [], {}, "" ] to tokenize string

Question

So I need to tokenize a string by all spaces not between quotes, I am using regex in Javascript notation.

For example:

" Test Test " ab c " Test" "Test " "Test" "T e s t"

becomes

[" Test Test ",ab,c," Test","Test ","Test","T e s t"]

For my use case however, the solution should work in the following test setting: https://www.regextester.com/

All Spaces not within quotes should be highlighted in the above setting. If they are highlighted in the above setting they would be parsed correctly in my program.

For more specificity, I am using Boost::Regex C++ to do the parsing as follows:

...
std::string test_string("\" Test Test \" ab c \" Test\" \"Test \" \"Test\" \"T e s t\"");
// (,|;)?\\s+     : Split on ,\s or ;\s
// (?![^\\[]*\\]) : Ignore spaces inside []
// (?![^\\{]*\\}) : Ignore spaces inside {}
// (?![^\"].*\")  : Ignore spaces inside "" !!! MY ATTEMPT DOESN'T WORK !!!

//Note the below regex delimiter declaration does not include the erroneous regex.
boost::regex delimiter("(,|;\\s|\\s)+(?![^\\[]*\\])(?![^\\(]*\\))(?![^\\{]*\\})");
std::vector<std::string> string_vector;
boost::split_regex(string_vector, test_string, delimiter);

For those of you who do not use Boost::regex or C++ the above link should enable testing of viable regex for the above use case.

Thank you all for you assistance I hope you can help me with the above problem.

Answer 1

I would 100% not use regular expressions for this. First off, because it's way easier to express as a PEG grammar instead. E.g.:

std::vector<std::string> tokens(std::string_view input) {
    namespace x3 = boost::spirit::x3;
    std::vector<std::string> r;

    auto atom                            //
        = '[' >> *~x3::char_(']') >> ']' //
        | '{' >> *~x3::char_('}') >> '}' //
        | '"' >> *~x3::char_('"') >> '"' //
        | x3::graph;

    auto token = x3::raw[*atom];

    parse(input.begin(), input.end(), token % +x3::space, r);
    return r;
}

This, off the bat, already performs as you intend:

Live On Coliru

int main() {
    for (std::string const input : {R"(" Test Test " ab c " Test" "Test " "Test" "T e s t")"}) {
        std::cout << input << "\n";
        for (auto& tok : tokens(input))
            std::cout << " - " << quoted(tok, '\'') << "\n";
    }
}

Output:

" Test Test " ab c " Test" "Test " "Test" "T e s t"
 - '" Test Test "'
 - 'ab'
 - 'c'
 - '" Test"'
 - '"Test "'
 - '"Test"'
 - '"T e s t"'

BONUS

Where this really makes the difference, is when you realize that you wanted to be able to handle nested constructs (e.g. "string" [ {1,2,"3,4", [true,"more [string]"], 9 }, "bye ]).

Regular expressions are notoriously bad at this. Spirit grammar rules can be recursive though. If you make your grammar description more explicit I could show you examples.