Google Search API Wildcard

Question

I have a Python project running on Google App Engine. I have a set of data currently placed at datastore. On user side, I fetch them from my API and show them to the user on a Google Visualization table with client side search. Because the limitations I can only fetch 1000 record at one query. I want my users search from all records that I have. I can fetch them with multiple queries before showing them but fetching 1000 records already taking 5-6 second so this process can exceed 30 seconds timeout and I don't think putting around 20.000 records on a table is good idea.

So I decided to put my records on Google Search API. Wrote a script to sync important data between datastore and Search API Index. When perform a search, couldn't find anything like wildcard character. For example let's say I have user field stores a string which contains "Ilhan" value. When user search for "Ilha" that record not show up. I want to show record includes "Ilhan" value even if it partially typed. So basically SQL equivalent of my search should be something like "select * from users where user like '%ilh%'".

I wonder if there is a way to that or is this not how Search API works?

Answer 1

I setup similar functionality purely within datastore. I have a repeated computed property that contains all the search substrings that can be formed for a given object.

class User(ndb.Model):
    # ... other fields
    search_strings = ndb.ComputedProperty(
        lambda self: [i.lower() for i in all_substrings(strings=[
            self.email,
            self.first_name,
            self.last_name,], repeated=True)

Your search query would then look like this:

User.query(User.search_strings == search_text.strip().lower()).fetch_page(20)

If you don't need the other features of Google Search API and if the number of substrings per entity won't put you at risk of hitting the 900 properties limit, then I'd recommend doing this instead as it's pretty simple and straight forward.

As for taking 5-6 seconds to fetch 1000 records, do you need to fetch that many? why not fetch only 100 or even 20 and use the query cursor for the user to pull the next page only if they need it.