Elastic Search: Adding an element on an array

Question

I am trying to batch update documents on the elastic search index. I want to know how can I achieve this scenario.

1. I have to create document if no document of that primaryKey exist.
2. I have to add the data in the array of the document if the primary key exist.

For example -

For initial write / if primary key not present.

Document written =

{
  PrimaryKey,
  DataList: [
    {
      DataField1: fieldValue1,
      DataField2: fieldValue2,
    }
  ]
}

if the document was present, the entry would have been appended to the list

{
  PrimaryKey,
  DataList: [
    {
      DataField1: fieldValue1,
      DataField2: fieldValue2,
    },
    {
      DataField1: fieldValue3,
      DataField2: fieldValue4
    }
    ....
  ]
}

In a batch update both types of primaryKeys may be present one which have document already present in the index, some document which was never added to the index.

Answer 1

I think this example can serve as a basis for your bulk. What I did was to consider that the _id and PrimaryKey are the same because the way to know if the docmentos exists is through the _id, if it doesn't exist a new document is created. I used the script to add items to the list if it already exists.

Read more about Update API upsert parameter.

Mapping

PUT my-index-000001
{
  "mappings": {
    "properties": {
      "PrimaryKey": {
        "type": "keyword"
      },
      "DataField1": {
        "type": "nested"
      }
    }
  }
}

POST my-index-000001/_doc/1
{
  "PrimaryKeyame": 1,
  "DataList": [
    {
      "DataField1": "fieldValue1",
      "DataField2": "fieldValue2"
    }
  ]
}

Bulk will add items to doc 1 and create the new document 2 (this does not exist in the index).

POST _bulk
{ "update" : { "_id" : "1", "_index" : "my-index-000001", "retry_on_conflict" : 3} }
{ "script" : { "source": "if (ctx._source.PrimaryKeyame != null) { ctx._source.DataList.addAll(params.DataList); }", "lang" : "painless", "params": { "PrimaryKeyame": "1", "DataList": [{"DataField1": "fieldValue3","DataField2": "fieldValue4"}]}}, "upsert" : {"PrimaryKeyame": "1", "DataList": [{"DataField1": "fieldValue3","DataField2": "fieldValue4"}]}}
{ "update" : { "_id" : "2", "_index" : "my-index-000001", "retry_on_conflict" : 3} }
{ "script" : { "source": "if (ctx._source.PrimaryKeyame != null) { ctx._source.DataList.addAll(params.DataList); }", "lang" : "painless", "params": { "PrimaryKeyame": "2", "DataList": [{"DataField1": "fieldValue3","DataField2": "fieldValue4"}]}}, "upsert" : {"PrimaryKeyame": "2", "DataList": [{"DataField1": "fieldValue3","DataField2": "fieldValue4"}]}}

Get Documents:

"hits": [
      {
        "_index": "my-index-000001",
        "_id": "1",
        "_score": 1,
        "_source": {
          "PrimaryKeyame": 1,
          "DataList": [
            {
              "DataField1": "fieldValue1",
              "DataField2": "fieldValue2"
            },
            {
              "DataField2": "fieldValue4",
              "DataField1": "fieldValue3"
            }
          ]
        }
      },
      {
        "_index": "my-index-000001",
        "_id": "2",
        "_score": 1,
        "_source": {
          "PrimaryKeyame": "2",
          "DataList": [
            {
              "DataField1": "fieldValue3",
              "DataField2": "fieldValue4"
            }
          ]
        }
      }
    ]