Using Elasticsearch to collect information from Yahoo-Finance query server

Question

I have been trying to query yahoo finance using the following url:

https://query1.finance.yahoo.com/v1/finance/lookup?formatted=true&lang=en-US&region=US&query=A*&type=equity&count=10001&start=0&scroll=true

I receive this message:

{"finance":{"result":null,"error":{"code":"internal-error","description":"Elasticsearch exception [type=illegal_argument_exception, reason=Result window is too large, from + size must be less than or equal to: [10000] but was [10001]. See the scroll api for a more efficient way to request large data sets. This limit can be set by changing the [index.max_result_window] index level setting.]"}}}

I am not familiar with Elasticsearch so for the last few days I have been trying to learn about it and spent almost all my time trying to figure this out but I can't find much information about this. I'd rather completely avoid using Elasticsearch and use python to write a request to that link and somehow set the "index.max_result_window" so the website would return more than 10,000 rows of data. I am trying to save the data I receive from the above url as a json file.

I have tried to register for the free Elasticsearch service online but I could not figure out how to provide the above url as an input source.

any help is highly appreciated.

------ UPDATE 12 Jan 2023 ------
I could not find a way to resolve this specific problem but I managed to obtain the data by breaking down the query into smaller pieces. For example, Query A would return more than 10,000 rows of data so I broke down A into a list of all possible alphabet combinations and conducted the query. This way, the results are less than 10,000 and then I concatenated all the results.

Answer 1

I could not find a solution to retrieve more than 10,000 rows from Yahoo Finance but I found a workout around it.

I managed to obtain the data by breaking down the query into smaller pieces. For example, Query "A" would return more than 10,000 rows of data so I broke down "A" into a list of all possible alphabet combinations and conducted the query. This way, the results are less than 10,000 and then I concatenated all the results. Hope this can help others trying to retrieve a large set of data from Yahoo Finance