Search code examples
mongodbelasticsearchmongo-connector

Missing documents while connecting Mongodb and elasticsearch via mongo-connector

I am trying to transfer documents from MongoDB 4.4.5 to Elasticsearch7.12 via mongo-connector 3.1.1

I start by using:

sudo mongo-connector -m 127.0.0.1:27017 -t 127.0.0.1:9200 -d elastic2_doc_manager -n Data.myData --continue-on-error --batch-size=1000 --only-dump --batch-size=1000 --no-dump --tz-aware

at the beginning of the log, I can see the total number of the documents in MongoDB which is 454

enter image description here

when the process is done I get

enter image description here

and in Elasticsearch index I can only see 268 as in the images shows

enter image description here

any idea what could be the issue here?

Solution

  • This issue is Releated to reaching the max number for fields's mapping on Elastivsearch to Solve this issue increase this limit on Elasticsearch's index via

    PUT my_data_index/_settings
{
  "index.mapping.total_fields.limit": 20000

}