yaml-0.2.7 GetNextDocument() hit assertion fail in Scanner::peek

Question

yaml-cpp team and everyone,

Our product receives an unfixed size of json response from a cloud service provider. We currently used a buffer with 16KB initial size to receive it, then pass it to yaml parser(we are using yaml-0.2.7). We expect yaml parser to throw an exception if the json document is incomplete during parsing and we will double the size of the buffer.

Today, we hit an assertion which shows "Assertion failed: (!m_tokens.empty()), function peek, file .../yaml-cpp-0.2.7/src/scanner.cpp, line 41." when doing parser.GetNextDocument(doc). The json document was incomplete due to small receiving buffer.

After examining the buffer and doing some experiment, I found out if some characters are missing following a certain pattern, the assertion in scanner will be hit during GetNextDocument. Such as for '{"access": "abc"}', if the last '}' is missing, then the assertion will be hit. Same thing happens if it is '{"access":' or '{"access"'. It will not hit the assertion if it is '"{"access":"abc' (note abc does not have the trailing double quote).

Will it help if I upgrade to the latest 0.5.3 version? I looked at the source code and saw the same assertion in Scanner::peek function is still there.

Here is the source code of the function where assertion is hit:

Token& Scanner::peek() {
{
    EnsureTokensInQueue();
    /** THIS ONE GOT HIT **/
    assert(!m_tokens.empty());  // should we be asserting here? I mean, we really just be checking
                                // if it's empty before peeking.
#if 0
    static Token *pLast = 0;
    if(pLast != &m_tokens.front())
        std::cerr << "peek: " << m_tokens.front() << "\n";
    pLast = &m_tokens.front();
#endif

    return m_tokens.front();
}

Much appreciate for the help! :)

Answer 1

This is fixed in 0.5.3, although I'm not sure when precisely it was fixed. I added tests for your examples and they pass (by throwing exceptions) as expected.