Bag of Words with json array

Question

I'm trying to follow this tutorial in order to make a custom bag of words.

from sklearn.feature_extraction.text import CountVectorizer

corpus = [
'All my cats in a row',
    'When my cat sits down, she looks like a Furby toy!',
    'The cat from outer space',
    'Sunshine loves to sit like this for some reason.'
]
vectorizer = CountVectorizer()
print( vectorizer.fit_transform(corpus).todense() )
print( vectorizer.vocabulary_ )

This script print that:

[[1 0 1 0 0 0 0 1 0 0 0 1 0 0 1 0 0 0 0 0 0 0 0 0 0 0]
 [0 1 0 1 0 0 1 0 1 1 0 1 0 0 0 1 0 1 0 0 0 0 0 0 1 1]
 [0 1 0 0 0 1 0 0 0 0 0 0 1 0 0 0 0 0 0 1 0 1 0 0 0 0]
 [0 0 0 0 1 0 0 0 1 0 1 0 0 1 0 0 1 0 1 0 1 0 1 1 0 0]]
{u'all': 0, u'sunshine': 20, u'some': 18, u'down': 3, u'reason': 13, u'looks': 9, u'in': 7, u'outer': 12, u'sits': 17, u'row': 14, u'toy': 24, u'from': 5, u'like': 8, u'for': 4, u'space': 19, u'this': 22, u'sit': 16, u'when': 25, u'cat': 1, u'to': 23, u'cats': 2, u'she': 15, u'loves': 10, u'furby': 6, u'the': 21, u'my': 11}

So here's my problem: I have a json file with this data structure:

[
    {
        "id": "1",
        "class": "positive",
        "tags": [
            "tag1",
            "tag2"
        ]
    },
    {
        "id": "2",
        "class": "negative",
        "tags": [
            "tag1",
            "tag3"
        ]
    }
]

So I'm trying to vectorize the tags array without any success.

I've tried something like this:

data = json.load(open('data.json'));
print( vectorizer.fit_transform(data).todense() )

also:

for element in data:
print( vectorizer.fit_transform(element).todense() ) 
#or 
print( vectorizer.fit_transform(element['tags']).todense() ) 

nobody works. Any ideas?

Answer 1

1. Use pandas to read the json file into a DataFrame

import pandas as pd
from sklearn.feature_extraction.text import CountVectorizer

df = pd.read_json('data.json', orient='values')
print(df)

This is what your DataFrame should look like:

Out[]:       
      class  id          tags
0  positive   1  [tag1, tag2]
1  negative   2  [tag1, tag3]

2. Convert the tags column from list to str

df['tags'] = df['tags'].apply(lambda x: ' '.join(x))
print(df)

This will result in converting tags to space separated strings:

Out[]:       
class  id       tags
0  positive   1  tag1 tag2
1  negative   2  tag1 tag3

3. Plug the tags column / pandas Series into CountVectorizer

vectorizer = CountVectorizer()
print(vectorizer.fit_transform(df['tags']).todense())
print(vectorizer.vocabulary_)

This will result in the output that you want:

Out[]:       
[[1 1 0]
 [1 0 1]]
{'tag1': 0, 'tag2': 1, 'tag3': 2}