Search code examples
javascriptd3.jsobservablehq

Item limit on d3.json observablehq

I'm working on an Observable notebook, I'm trying to load json data from a url, but when I do:

dataSet = d3.json("https://www.datos.gov.co/resource/cdhn-7vn8.json")

I get an array of just 1000 items, although the original source has more than 44000 rows. I have searched for examples on how to load the full json but I haven't found anything.

Solution

  • Very often with APIs there is a limit on the number of records you are allowed to get at any one time, and in your case that is 1000. I had a quick look for some info and found a video. It is in Spanish, which I don't understand, but it looks like it will give you the answer you need. Specifically, at 4mins 30 in the video it looks like you can specify an offset.

    https://herramientas.datos.gov.co/es/content/desarrollar-usando-los-datos

    Video screenshot at 4min 30secs

    So, in your code you need to loop your way to 44000 rows, 1000 at at time, accessing the API with an ever increasing offset 1000, 2000, 3000 ... until you have all the data you need.

    [Update] I have had a chance to test this now and can confirm the use of the offset parameter.

    https://www.datos.gov.co/resource/cdhn-7vn8.json

    or

    https://www.datos.gov.co/resource/cdhn-7vn8.json?$offset=0

    will return the first 1000 records.

    https://www.datos.gov.co/resource/cdhn-7vn8.json?$offset=1000

    will return the next 1000 records, and so on.

    I recommend the use of Postman for testing APIs, it's a great tool. https://www.getpostman.com/apps

    Bryan