NotImplementedError: 'split_respect_sentence_boundary=True' is only compatible with split_by='word'

Question

I have the following lines of code

from haystack.document_stores import InMemoryDocumentStore, SQLDocumentStore
from haystack.nodes import TextConverter, PDFToTextConverter,PreProcessor
from haystack.utils import clean_wiki_text, convert_files_to_docs, fetch_archive_from_http, print_answers

doc_dir = "C:\\Users\\abcd\\Downloads\\PDF Files\\"

docs = convert_files_to_docs(dir_path=doc_dir, clean_func=None, split_paragraphs=True


preprocessor = PreProcessor(
    clean_empty_lines=True,
    clean_whitespace=True,
    clean_header_footer=True,
    split_by="passage",
    split_length=2)
doc = preprocessor.process(docs)

When i try to run it, i get the following error message

NotImplementedError                       Traceback (most recent call last)
c:\Users\abcd\Downloads\solr9.ipynb Cell 27 in <cell line: 23>()
     16 print(type(docs))
     17 preprocessor = PreProcessor(
     18     clean_empty_lines=True,
     19     clean_whitespace=True,
     20     clean_header_footer=True,
     21     split_by="passage",
     22     split_length=2)
---> 23 doc = preprocessor.process(docs)

File ~\AppData\Roaming\Python\Python39\site-packages\haystack\nodes\preprocessor\preprocessor.py:167, in PreProcessor.process(self, documents, clean_whitespace, clean_header_footer, clean_empty_lines, remove_substrings, split_by, split_length, split_overlap, split_respect_sentence_boundary, id_hash_keys)
    165     ret = self._process_single(document=documents, id_hash_keys=id_hash_keys, **kwargs)  # type: ignore
    166 elif isinstance(documents, list):
--> 167     ret = self._process_batch(documents=list(documents), id_hash_keys=id_hash_keys, **kwargs)
    168 else:
    169     raise Exception("documents provided to PreProcessor.prepreprocess() is not of type list nor Document")


File ~\AppData\Roaming\Python\Python39\site-packages\haystack\nodes\preprocessor\preprocessor.py:225, in PreProcessor._process_batch(self, documents, id_hash_keys, **kwargs)
    222 def _process_batch(
    223     self, documents: List[Union[dict, Document]], id_hash_keys: Optional[List[str]] = None, **kwargs
    224 ) -> List[Document]:
--> 225     nested_docs = [
    226         self._process_single(d, id_hash_keys=id_hash_keys, **kwargs)
...
--> 324     raise NotImplementedError("'split_respect_sentence_boundary=True' is only compatible with split_by='word'.")
    326 if type(document.content) is not str:
    327     logger.error("Document content is not of type str. Nothing to split.")

NotImplementedError: 'split_respect_sentence_boundary=True' is only compatible with split_by='word'.

I don't even have split_respect_sentence_boundary=True as my argument and also i don't have split_by='word' rather i have it set as split_by="passage".

This is the same error if i try changing it to split_by="sentence".

Do let me know if i am missing out anything here.

Tried using split_by="sentence" but getting same error.

Answer 1

As you can see in the PreProcessor API docs, the default value for split_respect_sentence_boundary is True.

In order to make your code work, you should specify split_respect_sentence_boundary=False:

preprocessor = PreProcessor(
    clean_empty_lines=True,
    clean_whitespace=True,
    clean_header_footer=True,
    split_by="passage",
    split_length=2,
    split_respect_sentence_boundary=False)

I agree that this behavior is not intuitive. Currently, this node is undergoing a major refactoring.