Is it okay to set field mapping limit in elastic to 50k?

Question

My application is a survey creation app where user can create survey with many questions of different types. Each survey can then be shared to any number of people whose responses are recorded as below... 1 response looks something like this:

{
          "id" : 256,                                        // submission id
          "timeTaken" : "39.00",
          "startTime" : "2020-07-19T05:37:38.873Z",
          "state" : "COMPLETED",
          "completedTime" : "2020-07-19T05:38:17.873Z",
          "deviceType" : "COMPUTER",
          "ip" : null,
          "account_id" : 2,
          "channel_id" : 48,
          "contact_id" : null,
          "survey_id" : 10,
          "trigger_id" : 93,
          "trigger_contact_id" : null,
          "locked" : false,
          "location" : null,
          "language" : null,
          "submission_id" : 256,
          "question_90" : {
            "skipped" : false,
            "answer_choices" : [ 79 ]
          },
          "question_122" : {
            "skipped" : false,
            "otherChoice" : null,
            "answer_choices" : [ 115, 113, 111, 110, 114 ]
          },
          "question_106" : {
            "skipped" : false,
            "answer_choices" : [ 85 ]
          },
          "question_120" : {
            "answer_txt": "Great service",
            "skipped" : false
          },
          "question_118" : {
            "answer_txt": "Hello people",
            "skipped" : false
          },
          "question_121" : {
            "skipped" : false,
            "answer_date" : "2020-06-04T20:01:49.783Z",
            "answer_timezone" : 330
          },
          "question_108" : {
            "skipped" : false,
            "answer_int" : "93"
          },
          "question_105" : {
            "skipped" : false,
            "answer_string" : "+1 202 9932219"
          },
          "question_93" : {
            "skipped" : false,
            "answer_string" : "Kyra60@yahoo.com"
          },
          "question_117" : {
            "skipped" : false
          },
          "question_92" : {
            "skipped" : false,
            "answer_txt" : "composite"
          },
          "question_107" : {
            "skipped" : false,
            "answer_bool" : true
          },
      }

I need all the field here for analysis and querying. But most of the time the doc field limit exceeds the elasticsearch default limit of 1000. I am pretty sure that its 1000 for performance, but how can I honor this and insert a document which may have like 20k + fields.

Any insight on this?

Answer 1

You should definitely not go in a mapping explosion direction. keys are stored in memory so you'll have an over consumption of RAM and poor performances. You have it documented here : elastic.co/blog/found-crash-elasticsearch#mapping-explosion

You could easily map your answers with a very few quantity of keys by using a nested datatype.

"answers": [
          {
           "question_id": 122,
            "skipped" : false,
            "otherChoice" : null,
            "answer_choices" : [ 115, 113, 111, 110, 114 ]
          }
]