Search code examples
postgresqlrustfull-text-searchstemmingsnowball

Does Rust's tool Tantivy support Snowball stemmers like in the Postgres full-text search?

I've found that it supports 17 languages, but how does Tantivy solve this problem? What stemmers does it use under the hood?

PostgreSQL uses Snowball to implement full-text search https://snowballstem.org/demo.html that supports 29 languages.

How to apply Snowball stemmers with Tantivy? Does it support Snowball?

Solution

  • What stemmers does it use under the hood?

    Lift the hood to find that Tantivy currently depends on the stemmers in rust-stemmers.

    [dependencies]

# ...

rust-stemmers = "1.2.0"

    In turn, rust-stemmers is well documented as providing Snowball stemmers for multiple languages:

    This crate implements some stemmer algorithms found in the snowball project which are compiled to rust using the rust-backend of the snowball compiler.