My team is using SearchKick (Rails) in a project to interact with ElasticSearch.
An index (Contacts) consists of ~10 universal, searchable fields such as name, email, birthday, age, etc.
Additionally, we need to allow the program's users to create their own custom fields among a set of predefined types (String, Integer, Boolean, etc).
The plan was to either create a unique index per user (realistically that could scale up to low 4 digits) or use the single Contacts index and update the mapping to add a new custom field each time. I would assume that a realistic upper bound on custom fields would be 5k across the entire program.
The unique index approach seems like it will definitely encounter scaling problems at some point. Is that standard in a case like this?
Is there a huge performance cost to adding that many fields to an index's mapping if each document only contains 10 - 20 fields?
Are there any other standard approaches here?
Here you can find some details about why you cannot have too many indices.
Here you can find why the limit of fields is 1000
Having one index per user will be an overkill IMHO. Be careful of having basically the same field but with slightly different name. Use some autocomplete on the fronted in order to guide your users to reuse fields and not add new ones that basically do the same thing. This will allow you to use the Reindex API in the future in order to do some curation and create new indices (and delete the old ones) when you've reached a certain point. Basically a manual db vacuum of some sort.
P.S. There are use cases of indices having as many as 7000 fields or more and they seem to work fine. You can always try an MVP with one index and many fields and just try to group those fields when you've hit a milestone, by manual curation.