How to use ingest pipelines on already indexed documents?

Question

I've been using FOSElasticaBundle to index my documents (which are entities from a Symfony project kept in a db through Doctrine) into Elastic Search. FOSElastica does an automatic mapping and index after that all the documents.

The problem is that there's some actions I would want to apply on every documents (on those already indexed and those which will be after), so pipelines and painless seems to be a good solution.

But, I can't get to understand how to apply a pipeline to documents that are already indexed, do you have an idea how ?

I've seen that you can add a "pipeline=my_pipeline_name" after an ES request but you can do it for a single document while I'd want it to affect all the documents.

Answer 1

You can use Pipeline while you move your data from one index to another index.

You would need to make use of Reindex API in order for it to be executed on the data during its movement/ingestion_process from one index to another.

Note: This is index level operation meaning it would affect all the documents.

Below is summary of steps:

• Create a temporary_index,
• Reindex from source_index to temporary_index make use of Reindex API. Also including the pipeline (sample query provided below)
• Delete and re-create the source_index. Ensure that the mappings are also included while creating the index.
• Execute the same query this with source_index as destination name and temporary_index as source name without the pipeline

Below is how you make use of Reindex API with pipeline

POST _reindex
{
  "source": {
    "index": "source_index_name"
  },
  "dest": {
    "index": "temporary_index",
    "pipeline": "some_ingest_pipeline"
  }
}

Let me know if this helps!