Search code examples
c#pascalstring-literals

String literal recognition problem


I'm trying to recognize string literal by reading string per symbol. Example of my scanner skeleton:

public sealed class Scanner
{        
    // some class inner implementations        
    /// <summary>
    /// 
    /// </summary>
    /// <param name="Line"></param>
    /// <param name="LineNumber"></param>
    public void Run(String Line, Int32 LineNumber)
    {
        var ChPosition = default(Int32);
        var ChCurrent  = default(Char);
        var Value      = new StringBuilder();

        while (default(Char) != Line.ElementAtOrDefault<Char>(ChPosition))
        {
            ChCurrent = Line.ElementAtOrDefault<Char>(ChPosition);

            #region [Whitespace]
            if (Char.IsWhiteSpace(ChCurrent))
            {
                ChPosition++;
            }
            #endregion
            else
            {
                switch (ChCurrent)
                {                        
                    #region [String Literal (")]
                    case '"':
                        {
                            // skipping " sign, include only string inner value
                            ChCurrent = Line.ElementAtOrDefault<Char>(++ChPosition);

                            // ...? Problematic place!!!

                            this.Tokens.Enqueue(new SharedEntities.Token
                            {
                                Class = SharedEntities.Token.TokenClass.StringLiteral,
                                Value = Value.ToString()
                            }
                            );
                            Value.Clear();
                            ChPosition++;
                            break;
                        }
                    #endregion                        
                        {
                            throw new ScanningException(
                            "<syntax_error#" + ChCurrent.ToString() + ">\n"
                            + "Unsupported character appeared at: {ln: "
                            + LineNumber.ToString()
                            + "; pos: "
                            + (ChPosition + 1).ToString()
                            + "}"
                            );
                        }
                } // [switch(ChCurrent)]                   
            } // [if(Char.IsWhiteSpace(ChCurrent))...else]
        } // [while(default(Char) != Line.ElementAtOrDefault<Char>(ChPosition))]
    } // [public void Run(String Line, Int32 LineNumber)]
} // [public sealed class Scanner]

My target is to parse pascal-like string: "{everything enclosed, but ", only "" pair is allowed}".


Solution

  • First, you are obviously using some kind of parsing library, you would have better chance if you had modified your code, e.g. to something like I did, so that anybody can copy, paste, run your code.

    Answer is simple, your (string literal)-parsing region does not parse all input. Here is your code modified to be used without any additional library:

    public class Test
    {
        static char ElementAtOrDefault(string value, int position)
        {
            return position >= value.Length ? default(char) : value[position];
        }
        static string parseStringLiteral(string value, ref int ChPosition)
        {
            StringBuilder Value = new StringBuilder();
            char ChCurrent = ElementAtOrDefault(value, ++ChPosition);
            while (ChCurrent != '"')
            {
                Value.Append(ChCurrent);
                ChCurrent = ElementAtOrDefault(value, ++ChPosition);
                if (ChCurrent == '"')
                {
                    // "" sequence only acceptable
                    if (ElementAtOrDefault(value, ChPosition + 1) == '"')
                    {
                        Value.Append(ChCurrent);
                        // skip 2nd double quote
                        ChPosition++;
                        // move position next
                        ChCurrent = ElementAtOrDefault(value, ++ChPosition);
                    }
                }
                else if (default(Char) == ChCurrent)
                {
                    // message: unterminated string
                    throw new Exception("ScanningException");
                }
            }
            ChPosition++;
            return Value.ToString();
        }
    
        public static void test(string literal)
        {
            Console.WriteLine("testing literal with " + literal.Length + 
                " chars:\n" + literal);
            try
            {
                int pos = 0;
                string res = parseStringLiteral(literal, ref pos);
                Console.WriteLine("Parsed " + res.Length + " chars:\n" + res);
            }
            catch (Exception ex)
            {
                Console.WriteLine("Error: " + ex.Message);
            }
            Console.WriteLine();
        }
    
        public static int Main(string[] args)
        {
            test(@"""Hello Language Design""");
            test(@"""Is there any problems with the """"strings""""?""");
            test(@"""v#:';?325;.<>,|+_)""(*&^%$#@![]{}\|-_=""");
            return 0;
        }
    }
    

    Running this program produces output:

    testing literal with 23 chars:
    "Hello Language Design"
    Parsed 21 chars:
    Hello Language Design
    
    testing literal with 45 chars:
    "Is there any problems with the ""strings""?"
    Parsed 41 chars:
    Is there any problems with the "strings"?
    
    testing literal with 39 chars:
    "v#:';?325;.,|+_)"(*&^%$#@![]{}\|-_="
    Parsed 18 chars:
    v#:';?325;.,|+_)

    So it works for your testing, but algorithm is not correct, try running:

    //literal with "", should produce ", but it does not
    test(@"""""""""");
    

    And you will incorrectly get:

    testing literal with 4 chars:
    """"
    Parsed 0 chars:

    Problem is, if you encounter character " in your while condition, you do not check next character, if it is " or not:

    while (ChCurrent != '"') //bug
    

    Of course, I created correct version for you :-) Here it is (it uses your style, just edited version of yours):

    static string parseStringLiteral(string value, ref int ChPosition)
    {
        StringBuilder Value = new StringBuilder();
        char ChCurrent = ElementAtOrDefault(value, ++ChPosition);
        bool goon = true;
        while (goon)
        {
            if (ChCurrent == '"')
            {
                // "" sequence only acceptable
                if (ElementAtOrDefault(value, ChPosition + 1) == '"')
                {
                    Value.Append(ChCurrent);
                    // skip 2nd double quote
                    ChPosition++;
                    // move position next
                    ChCurrent = ElementAtOrDefault(value, ++ChPosition);
                }
                else goon = false; //break;
            }
            else if (default(Char) == ChCurrent)
            {
                // message: unterminated string
                throw new Exception("ScanningException");
            }
            else
            {
                Value.Append(ChCurrent);
                ChCurrent = ElementAtOrDefault(value, ++ChPosition);
            }
        }
        ChPosition++;
        return Value.ToString();
    }
    

    Happy coding :-)