How to use spacy to do Name Entity recognition on CSV file

Question

I have tried so many things to do name entity recognition on a column in my csv file, i tried ne_chunk but i am unable to get the result of my ne_chunk in columns like so

ID  STORY                                       PERSON  NE   NP  NN VB  GE
1   Washington, a police officer James...        1      0    0   0   0   1

Instead after using this code,

news=pd.read_csv("news.csv")

news['tokenize'] = news.apply(lambda row: nltk.word_tokenize(row['STORY']), axis=1)


news['pos_tags'] = news.apply(lambda row: nltk.pos_tag(row['tokenize']), axis=1)

news['entityrecog']=news.apply(lambda row: nltk.ne_chunk(row['pos_tags']), axis=1)

tag_count_df = pd.DataFrame(news['entityrecognition'].map(lambda x: Counter(tag[1] for tag in x)).to_list())

news=pd.concat([news, tag_count_df], axis=1).fillna(0).drop(['entityrecognition'], axis=1)

news.to_csv("news.csv")

i got this error

IndexError : list index out of range

So, i am wondering if i could do this using spaCy which is another thing that i have no clue about. Can anyone help?

Answer 1

It seems that you are checking the chunks incorrectly, that's why you get a key error. I'm guessing a little about what you want to do, but this creates new columns for each NER type returned by NLTK. It would be a little cleaner to predefined and zero each NER type column (as this gives you NaN if NERs don't exist).

def extract_ner_count(tagged):
    entities = {}
    chunks = nltk.ne_chunk(tagged)
    for chunk in chunks:
        if type(chunk) is nltk.Tree:
          #if you don't need the entities, just add the label directly rather than this.
          t = ''.join(c[0] for c in chunk.leaves())
          entities[t] = chunk.label()
    return Counter(entities.values())

news=pd.read_csv("news.csv")
news['tokenize'] = news.apply(lambda row: nltk.word_tokenize(row['STORY']), axis=1)
news['pos_tags'] = news.apply(lambda row: nltk.pos_tag(row['tokenize']), axis=1)
news['entityrecognition']=news.apply(lambda row: extract_ner_count(row['pos_tags']), axis=1)
news = pd.concat([news, pd.DataFrame(list(news["entityrecognition"]))], axis=1)

print(news.head())

If all you want is the counts the following is more performant and doesn't have NaNs:

tagger = nltk.PerceptronTagger()
chunker = nltk.data.load(nltk.chunk._MULTICLASS_NE_CHUNKER)
NE_Types = {'GPE', 'ORGANIZATION', 'LOCATION', 'GSP', 'O', 'FACILITY', 'PERSON'}

def extract_ner_count(text):
    c = Counter()
    chunks = chunker.parse(tagger.tag(nltk.word_tokenize(text,preserve_line=True)))
    for chunk in chunks:
        if type(chunk) is nltk.Tree:
            c.update([chunk.label()])
    return c

news=pd.read_csv("news.csv")
for NE_Type in NE_Types:
    news[NE_Type] = 0
news.update(list(news["STORY"].apply(extract_ner_count)))

print(news.head())