Search code examples
google-cloud-platformgoogle-cloud-dataprep

Upload data from API into GCP dataprep


Is it possible to import data from a Restful API directly into data prep?

I think there might be a couple of work arounds... 1: Save the results to a JSON file in a GCS bucket and import from there. 2: Import the results into a Big Query table and then import into data prep from there.

It would be much smoother to just call an API and get a result set, as opposed to having to take an extra step. I just can't find anywhere that explains how to do this.

TIA!


Solution

  • Long story short: there's no real way to directly stream data into Data Prep. Even the new Dataprep Premium Edition expects that you'll have the data in some form of a database--though this does expand your options to Google Sheets, Salesforce, Oracle, Microsoft SQLServer, MySQL and PostgreSQL.

    Personally, I've just gotten in the habit of writing directly into BigQuery and/or Firestore-to-BigQuery to get around this sort of thing. It also has the nice side effect of being another type of logging from applications.