Elasticsearch - Reindex documents with stored / excluded fields

Question

Im having an index mapping with the following configuration:

"mappings" : {
  "_source" : {
    "excludes" : [
      "special_field"
    ]
  },
  "properties" : {
    "special_field" : {
      "type" : "text",
      "store" : true
    },
  }
}

So, when A new document is indexed using this mapping a got de following result:

{
  "_index": "********-2021",
  "_id": "************",
  "_source": {
    ...
  },
  "fields": {
    "special_field": [
      "my special text"
    ]
  }
}

If a _search query is perfomed, special_field is not returned inside _source as its excluded.

With the following _search query, special_field data is returned perfectly:

GET ********-2021/_search
{
  "stored_fields": [ "special_field" ],
  "_source": true
}

Right now im trying to reindex all documents inside that index, but im loosing the info stored in special_field and only _source field is getting reindexed.

Is there a way to put that special_field back inside _source field?

Is there a way to reindex that documents without loosing special_field data?

How could these documents be migrated to another cluster without loosing special_field data?

Thank you all.

Answer 1

Thx Hamid Bayat, I finally got it using a small logstash pipeline.

I will share it:

input {
  elasticsearch {
    hosts => "my-first-cluster:9200"
    index => "my-index-pattern-*"
    user => "****"
    password => "****"
    query => '{ "stored_fields": [ "special_field" ], "_source": true }'
    size => 500
    scroll => "5m"
    docinfo => true
    docinfo_fields => ["_index", "_type", "_id", "fields"]
  }
}

filter {
  if [@metadata][fields][special_field]{
    mutate {
      add_field => { "special_field" => "%{[@metadata][fields][special_field]}" }
    }
  }
}

output {
  elasticsearch {
    hosts => ["http://my-second-cluster:9200"]
    password => "****"
    user => "****"
    index => "%{[@metadata][_index]}"
    document_id => "%{[@metadata][_id]}"
    template => "/usr/share/logstash/config/index_template.json"
    template_name => "template-name"
    template_overwrite => true 
  }
}

I had to add fields into docinfo_fields => ["_index", "_type", "_id", "fields"] elasticsearch input plugin and all my stored_fields were on [@metadata][fields] event field.

As the @metadata field is not indexed i had to add a new field at root level with [@metadata][fields][special_field] value.

Its working like a charm.