Spacy - Chunk NE tokens

Question

Let's say that I have a document, like so:

import spacy

nlp = spacy.load('en')

doc = nlp('My name is John Smith')

[t for t in doc]
> [My, name, is, John, Smith]

Spacy is intelligent enough to realize that 'John Smith' is a multi-token named entity:

[e for e in doc.ents]
> [John Smith]

How can I make it chunk named entities into discrete tokens, like so:

> [My, name, is, John Smith]

Answer 1

Spacy documentation on NER says that you can access token entity annotations using the token.ent_iob_ and token.ent_type_ attributes.

https://spacy.io/usage/linguistic-features#accessing

Example:

import spacy

nlp = spacy.load('en')
doc = nlp('My name is John Smith')


ne = []
merged = []
for t in doc:
    # "O" -> current token is not part of the NE
    if t.ent_iob_ == "O":
        if len(ne) > 0:
            merged.append(" ".join(ne))
            ne = []
        merged.append(t.text)
    else:
        ne.append(t.text)

if len(ne) > 0:
    merged.append(" ".join(ne))

print(merged)

This will print:

['My', 'name', 'is', 'John Smith']