Search code examples
apipaginationintegrationwebhookslarge-data

how would I expose 200k+ records via an API?

what would be the best option for exposing 220k records to third party applications?

  • SF style 'bulk API' - independent of the standard API to maintain availability
  • server-side pagination
  • call back to a ftp generated file?
  • webhooks?

This bulk will have to happen once a day or so. ANY OTHER SUGGESTIONS WELCOME!

Solution

  • I would use a streaming API. This is an API that does a "select * from table" and then streams the results to the consumer. You do this using a for loop to fetch and output the records. This way you never use much memory and as long as you frequently flush the output the webserver will not close the connection and you will support any size of result set.

    I know this works as I (shameless plug) wrote the mysql-crud-api that actually does this.