Search code examples
pythonnlpspacysentence

Using POS and PUNCT tokens in custom sentence boundaries in spaCy

I am trying to split sentences into clauses using spaCy for classification with a MLLib. I have searched for one of two solutions that I consider the best way to approach but haven't quite had much luck.

  1. Option: Would be to use the tokens in the doc i.e. token.pos_ that match to SCONJ and split as a sentence.

  2. Option: Would be to create a list using whatever spaCy has as a dictionary of values it identifies as SCONJ

The issue with 1 is that I only have .text, .i, and no .pos_ as the custom boundaries (as far as I am aware needs to be run before the parser.

The issue with 2 is that I can't seem to find the dictionary. It is also a really hacky approach.

import deplacy
from spacy.language import Language

# Uncomment to visualise how the tokens are labelled
# deplacy.render(doc)

custom_EOS = ['.', ',', '!', '!']
custom_conj = ['then', 'so']


@Language.component("set_custom_boundaries")
def set_custom_boundaries(doc):
    for token in doc[:-1]:
        if token.text in custom_EOS:
            doc[token.i + 1].is_sent_start = True
        if token.text in custom_conj:
            doc[token.i].is_sent_start = True
    return doc


def set_sentence_breaks(doc):
    for token in doc:
        if token == "SCONJ":
            doc[token.i].is_sent_start = True


def main():
    text = "In the add user use case, we need to consider speed and reliability " \
           "so use of a relational DB would be better than using SQLite. Though " \
           "it may take extra effort to convert @Bot"

    nlp = spacy.load("en_core_web_sm")
    nlp.add_pipe("set_custom_boundaries", before="parser")
    doc = nlp(text)
    # for token in doc:
    #     print(token.pos_)

    print("Sentences:", [sent.text for sent in doc.sents])


if __name__ == "__main__":
    main()

Current Output
Sentences: ['In the add user use case,',
'we need to consider speed and reliability,
'so the use of a relational DB would be better than using SQLite.',
'Though it may take extra effort to convert @Bot']

Solution

  • I would recommend not trying to do anything clever with is_sent_starts - while it is user-accessible, it's really not intended to be used in that way, and there is at least one unresolved issue related to it.

    Since you just need these divisions for some other classifier, it's enough for you to just get the string, right? In that case I recommend you run the spaCy pipeline as usual and then split sentences on SCONJ tokens (if just using SCONJ is working for your use case). Something like:

    out = []
for sent in doc.sents:
    last = sent[0].i
    for tok in sent:
        if tok.pos_ == "SCONJ":
            out.append(doc[last:tok.i])
            last = tok.i + 1
    out.append(doc[last:sent[-1].i])

    Alternately, if that's not good enough, you can identify subsentences using the dependency parse to find verbs in subsentences (by their relation to SCONJ, for example), saving the subsentences, and then adding another sentence based on the root.