Search code examples
objective-cmacoscocoaparsekit

Using ParseKit Tokenization and DelimitedString

I'm trying to use ParseKit's Tokenization with SQL Server syntax. If you take a look at the following example:

CREATE PROCEDURE [dbo].[test]
  @MyId UNIQUEIDENTIFIER
AS
BEGIN
  -- something something
END
GO

I'm using the following to setup comments (and multiline comments):

PKTokenizer *t = [PKTokenizer tokenizerWithString: [inputString substringWithRange: range]];

// SQL does not support // comments.
[t.commentState removeSingleLineStartMarker: @"//"];

// But it does support -- comments.
[t setTokenizerState:t.commentState from: '-' to:'-'];
[t.commentState addSingleLineStartMarker: @"--"];

[t setTokenizerState: t.commentState from: '/' to: '/'];
[t.commentState addMultiLineStartMarker: @"/*" endMarker: @"*/"];

Which sets up support for handling sql comments. Now I'm trying to add support for detecting parameters such as @MyId. Current the tokenizer picks these up as words, but does not include the @ in the string value. My thought was to attempt using a delimitState by doing:

[t.delimitState addStartMarker: @"@" endMarker: @" " allowedCharacterSet: [NSCharacterSet alphanumericCharacterSet]];

Then check for token.isDelimitedString, but it does not seem to pick any up.

Am I going about this wrong? Any suggestions on how I could include the @ in my wordstate?

Solution

  • Developer of ParseKit here.

    To match @MyId as a single delimited string token, try:

    [t setTokenizerState:t.delimitState from:'@' to:'@'];
[t.delimitState addStartMarker:@"@" endMarker:nil allowedCharacterSet:[NSCharacterSet alphanumericCharacterSet]];