How to fix: http.client.HTTPException: got more than 100 headers when using bulk from python elasticsearch API

Question

I'm using elasticsearch-oss:6.8.3 with python 3.7

I'm using the bulk function to update value in my ES likes this :

for hit in hits:
        # hit = {
        #   '_index': 'my-index',
        #   '_score': '1.0',
        #   '_type': '_doc',
        #   '_id': 'YNi6920BHiHVzMIEjF0_', 
        #   '_source': {}
        # }
        del hit["_score"]

        hit["_source"].update({something_to_update})
        hit["_op_type"] = "update"

        # Need to deepcopy otherwise as we are in a generator, this will create an id loop in pyhton and raise an ES error
        _source = {"doc": deepcopy(hit["_source"])}

        # yield result
        yield hit

For now it throughs the error : http.client.HTTPException: got more than 100 headers when using bulk from python elasticsearch API

I think this comes from the python limitation size for http headers. So, I'd like to pass all ids in form each hit in the body of the request but I don't know how to do that...

Answer 1

For now I have a dirty fix which is :

import http.client

http.client._MAXHEADERS = 1000