Split and count emojis and words in a given string in Python

Question

For a given string, I'm trying to count the number of appearances of each word and emoji. I did it already here for emojis that consists only from 1 emoji. The problem is that a lot of the current emojis are composed from a few emojis.

Like the emoji 👨‍👩‍👦‍👦 consists of four emojis - 👨‍ 👩‍ 👦‍ 👦, and emojis with human skin color, for example 🙅🏽 is 🙅 🏽 etc.

The problem boils down to how to split the string in the right order, and then counting them is easy.

There are some good questions that addressed the same thing, like link1 and link2 , but none of them applies to the general solution (or the solution is outdated or I just can't figure it out).

For example, if the string would be hello 👩🏾‍🎓 emoji hello 👨‍👩‍👦‍👦, then I'll have {'hello':2, 'emoji':1, '👨‍👩‍👦‍👦':1, '👩🏾‍🎓':1} My strings are from Whatsapp, and all were encoded in utf8.

I had many bad attempts. Help would be appreciated.

Answer 1

Use the 3rd party regex module, which supports recognizing grapheme clusters (sequences of Unicode codepoints rendered as a single character):

>>> import regex
>>> s='👨‍👩‍👦‍👦🙅🏽'
>>> regex.findall(r'\X',s)
['👨\u200d👩\u200d👦\u200d👦', '🙅🏽']
>>> for c in regex.findall('\X',s):
...     print(c)
... 
👨‍👩‍👦‍👦
🙅🏽

To count them:

>>> data = regex.findall(r'\X',s)
>>> from collections import Counter
>>> Counter(data)
Counter({'👨\u200d👩\u200d👦\u200d👦': 1, '🙅🏽': 1})