singer tap-zendesk - how to extract catalog.json with selected:True from discovery mode

Question

I am using singer's tap-zendesk library and want to extract data from specific schemas.

I am running the following command in sync mode:

tap-zendesk --config config.json --catalog catalog.json.

Currently my config.json file has the following parameters:

{
    "email": "<email>",
    "api_token": "<token>",
    "subdomain": "<domain>",
    "start_date": "<start_date>"
  }

I've managed to extract data by putting 'selected':true under schema, properties and metadata in the catalog.json file. But I was wondering if there was an easier way to do this? There are around 15 streams I need to go through.

I manage to get the catalog.json file through the discovery mode command:

tap-zendesk --config config.json --discover > catalog.json

The output looks something like the following, but that means that I have to go and add selected:True under every field.

{
  "streams": [
    {
      "stream": "tickets",
      "tap_stream_id": "tickets",
      "schema": {
        **"selected": "true"**,
        "properties": {
          "organization_id": {
            **"selected": "true"**,},
      "metadata": [
        {
          "breadcrumb": [],
          "metadata": {
            **"selected": "true"**

}

Answer 1

The selected=true needs to be applied only once per stream. This needs to be added to the metadata section under the stream where breadcrumbs = []. This is very poorly documented.

Please see this blog post for some helpful details: https://medium.com/getting-started-guides/extracting-ticket-data-from-zendesk-using-singer-io-tap-zendesk-57a8da8c3477