Search code examples
socratasoda

Socrata API not returning most recent data (only up to 2 year old)

I am using the Socrata API to pull the Commitment of Traders Report data. But instead of returning data all the way to 2024 (as you can see from here, there is data up to 2024) the data stops at June 2022.

I am using this request https://publicreporting.cftc.gov/resource/jun7-fc8e.json?cftc_commodity_code=088&$where=report_date_as_yyyy_mm_dd%20%3E=%20%272020-01-01T00:00:00.000%27 to pull data from 2020 onward (I started with 2014 and I though it stopped me at 2022 because I hit some throttling or some kind of limit, but moving it to 2020 yields the same result)

Any idea of what I am doing wrong? I tried to submit the request with an App Token as well (it gives unlimited requests), but the results are the same.

Thanks!

Solution 
  •   {
    "id": "240709088691C",
    "market_and_exchange_names": "GOLD - COMMODITY EXCHANGE INC.",  
    "report_date_as_yyyy_mm_dd": "2024-07-09T00:00:00.000",
    "yyyy_report_week_ww": "2024 Report Week 28",
    "contract_market_name": "GOLD",
    "cftc_commodity_code": "088 ",  // <-- trailing space
    "commodity_name": "GOLD",
    "commodity": "GOLD", 
   ...
  }

    A typo is made around Sept 2022 that adds a trailing space at the end of cftc_commodity_code:

    cftc_commodity_code='088' returns items before Oct 2022
    cftc_commodity_code='088%20' returns items after Sept 2022

    SOLUTION:

    Get data description:
    https://publicreporting.cftc.gov/api/views/jun7-fc8e