Using selected keywords from a string column to form one hot encoding type columns of pandas DataFrame

Question

To demonstrate my question, consider the following example. Lets assume I have a following dataframe

index	ignore_x	ignore_y	phrases
0	43	23	cat eats mice
1	1.3	33	water is pure
2	13	63	machine learning
3	15	35	where there is a will, there is a way

Now consider that I have certain words I want to form dummy variables only for those.

keywords = [cat, is]

To do that, separate columns are populated for each of the keyword

index	x_ignore	y_ignore	phrases	kw_cat	kw_is
0	43	23	cat eats mice	0	0
1	1.3	33	water is pure	0	0
2	13	63	machine learning	0	0
3	15	35	where there is a will, there is a way	0	0

Each phrase is scanned for the words and if there is a presence, the column returns True or get 1. (An alternative could be to count the occurrence as well, but lets keep it simple for now)

index	x_ignore	y_ignore	phrases	kw_cat	kw_is
0	43	23	cat eats mice	1	0
1	1.3	33	water is pure	0	1
2	13	63	machine learning	0	0
3	15	35	where there is a will, there is a way	0	1

---

What I've been trying? Loosely, I've been trying to do something like this

for row, element in enumerate(df):
    for item in keywords:
        if item in df['phrases'].str.split(' '):
            df.loc[row, element] = 1

But this is not helping me out. It rather throws me a diagonal of 1s on those dummy variables.

Thanks :)

Edit: Just boldened the keywords to help you guys go through quickly :)

Answer 1

You can use nltk.tokenizer.word_tokenize() to split the sentences into word list

import nltk

keywords = ['cat', 'is']

tokenize = df['phrases'].apply(nltk.tokenize.word_tokenize)

print(tokenize)

0                                    [cat, eats, mice]
1                                    [water, is, pure]
2                                  [machine, learning]
3    [where, there, is, a, will, ,, there, is, a, way]

Then loop through keywords and check if keyword in generated word list.

for keyword in keywords:
    df[f'kw_{keyword}'] = tokenize.apply(lambda lst: int(keyword in lst))

print(df)

   index  ignore_x  ignore_y                                phrases  kw_cat  kw_is
0      0      43.0        23                          cat eats mice       1      0
1      1       1.3        33                          water is pure       0      1
2      2      13.0        63                       machine learning       0      0
3      3      15.0        35  where there is a will, there is a way       0      1