How to parse a list of tokens with FParsec

Question

I am trying to parse a list of tokens with FParsec, where each token is either a block of text or a tag - for example:

This is a {type of test} test, and it {succeeds or fails}

Here is the parser:

type Parser<'t> = Parser<'t, unit>

type Token =
| Text of string
| Tag of string

let escape fromString toString : Parser<_> =
    pstring fromString |>> (fun c -> toString)

let content : Parser<_> =
    let contentNormal = many1Satisfy (fun c -> c <> '{' && c <> '}')
    let openBraceEscaped = escape "{{" "{"
    let closeBraceEscaped = escape "}}" "}"
    let contentEscaped = openBraceEscaped <|> closeBraceEscaped
    stringsSepBy contentNormal contentEscaped

let ident : Parser<_> =
    let isIdentifierFirstChar c = isLetter c || c = '_'
    let isIdentifierChar c = isLetter c || isDigit c || c = '_'
    spaces >>. many1Satisfy2L isIdentifierFirstChar isIdentifierChar "identifier" .>> spaces

let text = content |>> Text

let tag = 
    ident |> between (skipString "{") (skipString "}")
    |>> Tag

let token = text <|> tag
let tokens = many token .>>. eof   

the following tests work:

> run token "abc def" ;;
val it : ParserResult<Token,unit> = Success: Text "abc def"

> run token "{abc def}" ;;
val it : ParserResult<Token,unit> = Success: Tag "abc def"

but trying to run tokens results in an exception:

> run tokens "{abc} def" ;;
System.InvalidOperationException: (Ln: 1, Col: 10): The combinator 'many' was 
    applied to a parser that succeeds without consuming input and without
    changing the parser state in any other way. (If no exception had been raised,
    the combinator likely would have entered an infinite loop.)

I've gone over this stackoverflow question but nothing I've tried works. I even added the following, but I get the same exception:

let tokenFwd, tokenRef = createParserForwardedToRef<Token, unit>()
do tokenRef := choice [tag; text]
let readEndOfInput : Parser<unit, unit> = spaces >>. eof
let readExprs = many tokenFwd
let readExprsTillEnd = readExprs .>> readEndOfInput

run readExprsTillEnd "{abc} def"  // System.InvalidOperationException ... The combinator 'many' was applied  ...

I believe the problem is stringsSepBy in content, but I can't figure out any other way to get a string with the escaped items

Any help would be much appreciated - I have been going through this for a couple days now and can't figure it out.

Answer 1

stringsSepBy accepts zero strings, causing token to accept an empty string, causing many to complain.

I changed it to the following to verify that that was the line you need to work on.

many1 (contentNormal <|> contentEscaped) |>> fun l -> String.concat "" l

Also I got away from stringsSepBy contentNormal contentEscaped, because that says you need to match contentNormals with contentEscapeds in between them. So a{{b}}c is ok, but {{b}}, {{b}}c and a{{b}} will fail.