Search code examples
pythonregexstringtags

Word tokenization using python regular expressions

I am trying to split strings into lists of "tags" in python. The splitting should handle strings such as "HappyBirthday" and remove most punctuation but preserve hyphens, and apostrophes. My starting point is:

tags = re.findall("([A-Z]{2,}(?=[A-Z]|$)|[A-Z][a-z]*)|\w+-\w+|[\w']+"

I would want to turn this sample data:

Jeff's dog is un-American SomeTimes! BUT NOTAlways

Into:

['Jeff's', 'dog', 'is', 'un-American', 'Some', 'Times', 'BUT', 'NOT', 'Always']

P.S. I am sorry my description isn't very good. I am not sure how to explain it, and have been mostly unsuccessful with google. I hope the example illustrates it properly.

Edit: i think i needed to be more precise, so also,

  1. if the word is hypenated and capital, like 'UN-American' will it keep it as one word so output would be 'UN-American'
  2. if the hyphen has a space on either or both sides, a la 'THIS- is' or 'This - is' it should ignore the hypen and produce ["THIS", "is"] and ["This", "is"] respecticly,
  3. and simmilarly for an apostrophe if its in the middle of a word like "What'sItCalled" it should produce ["What's","It", "Called"]
Solution

  • I suggest the following:

    re.findall("[A-Z]{2,}(?![a-z])|[A-Z][a-z]+(?=[A-Z])|[\'\w\-]+",s)

    This yields for your example:

    ["Jeff's", 'dog', 'is', 'un-American', 'Some', 'Times', 'BUT', 'NOT', 'Always']

    Explanation: The RegExp is made up of 3 alternatives:

    1. [A-Z]{2,}(?![a-z]) matches words with all letters capital
    2. [A-Z][a-z]+(?=[A-Z]) matches words with a first capital letter. The lookahead (?=[A-Z]) stops the match before the next capital letter
    3. [\'\w\-]+ matches all the rest, i.e. words which may contain ' and -.