Failed to parse mapping: Field [similarity] requires field [index] to be configured

Question

when I use dense_vecto and set cosine to similarity, start application will get a mapper_parsing_exception : Field [similarity] requires field [index] to be configured But the @Field annotation has already configured the index to be true, So I don't know why I reported this error. Did I miss any other configurations or annotations?

dependencies version: org.springframework.boot:spring-boot-starter-data-elasticsearch:3.1.2

my entity:

@Data
@Document(indexName = "spare_parts")
public class SparePartsDoc {

    @Id
    @Field(name = "id", type = FieldType.Keyword)
    String id;

    @Field(name = "data_id", type = FieldType.Keyword)
    String dataId;

    @Field(name = "type", type = FieldType.Keyword)
    String type;

    @Field(name = "second_type", type = FieldType.Keyword)
    String secondType;

    @Field(name = "brand", type = FieldType.Text)
    String picture;

    @Field(name = "brand", type = FieldType.Keyword)
    String brand;

    @Field(name = "color", type = FieldType.Keyword)
    String color;

    @Field(name = "quality", type = FieldType.Keyword)
    String quality;

    @Field(name = "level", type = FieldType.Keyword)
    String level;

    @Field(name = "specs", type = FieldType.Object)
    Map<String, String> specs;

    @Field(name = "remark", type = FieldType.Object)
    Map<String, String> remark;

    @Field(name = "code", type = FieldType.Keyword)
    String code;

    @Field(name = "product_name", type = FieldType.Text)
    String productName;

    @Field(name = "embedding", type = FieldType.Dense_Vector, dims= 768, similarity = "cosine")
    double[] embedding;

    @Field(name = "product_code", type = FieldType.Keyword)
    String productCode;

    @Field(name = "binding_complete_machin_id", type = FieldType.Keyword)
    String bindingCompleteMachinId;

    @Field(name = "increment_code", type = FieldType.Keyword)
    Integer incrementCode;

    @Field(name = "is_valid", type = FieldType.Keyword)
    String isValid;

    @Field(name = "create_time", type = FieldType.Date)
    LocalDateTime createTime;

    @Field(name = "last_update_time", type = FieldType.Date)
    LocalDateTime lastUpdateTime;

    @Transient
    float score;

}

part of exception:

Caused by: org.springframework.data.elasticsearch.UncategorizedElasticsearchException: [es/indices.create] failed: [mapper_parsing_exception] Failed to parse mapping: Field [similarity] requires field [index] to be configured
    at org.springframework.data.elasticsearch.client.elc.ElasticsearchExceptionTranslator.translateExceptionIfPossible(ElasticsearchExceptionTranslator.java:102)
    at org.springframework.data.elasticsearch.client.elc.ElasticsearchExceptionTranslator.translateException(ElasticsearchExceptionTranslator.java:63)
    at org.springframework.data.elasticsearch.client.elc.ChildTemplate.execute(ChildTemplate.java:73)
    at org.springframework.data.elasticsearch.client.elc.IndicesTemplate.doCreate(IndicesTemplate.java:145)
    at org.springframework.data.elasticsearch.client.elc.IndicesTemplate.createWithMapping(IndicesTemplate.java:135)
    at org.springframework.data.elasticsearch.repository.support.SimpleElasticsearchRepository.<init>(SimpleElasticsearchRepository.java:84)
    at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77)
    at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499)
    at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:480)
    at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:211)
    ... 106 more
Caused by: co.elastic.clients.elasticsearch._types.ElasticsearchException: [es/indices.create] failed: [mapper_parsing_exception] Failed to parse mapping: Field [similarity] requires field [index] to be configured
    at co.elastic.clients.transport.rest_client.RestClientTransport.getHighLevelResponse(RestClientTransport.java:334)
    at co.elastic.clients.transport.rest_client.RestClientTransport.performRequest(RestClientTransport.java:154)
    at co.elastic.clients.elasticsearch.indices.ElasticsearchIndicesClient.create(ElasticsearchIndicesClient.java:266)
    at org.springframework.data.elasticsearch.client.elc.IndicesTemplate.lambda$doCreate$0(IndicesTemplate.java:145)
    at org.springframework.data.elasticsearch.client.elc.ChildTemplate.execute(ChildTemplate.java:71)
    ... 115 more

I attempted to create an index using IndexOperations and manually configured the index and similarity without any issues.

@Test
void test(){
    IndexOperations indexOperations = elasticsearchOperations.indexOps(RetrieverDoc.class);
    indexOperations.delete();
    indexOperations.create();
    Document mapping = indexOperations.createMapping(RetrieverDoc.class);
    if (mapping.get("properties") instanceof LinkedHashMap properties) {
        if (properties.get("embedding") instanceof LinkedHashMap embedding) {
            embedding.put("index", true);
            embedding.pu("similarity", "dot_product");
            indexOperations.putMapping(mapping);
        }
    }
}

Answer 1

On the @Field annotation the index parameter defaults to true. When the mapping is created, the value is only written in the mapping when it is false. This is because according to the Elasticsearch documentation true is the default value if it is not set. So it astonishes me that you need to set it explicitly in the mapping.

This could be changed in Spring Data Elasticsearch by always setting the value for index when the type is similarity, I created an issue for that.