How to override a stream's primary key properties or incremental replication key in Meltano?

Question

There are times when we want to override a stream's key properties (primary key) or it's incremental replication key. What's the best way to do this in Meltano?

Answer 1

In your Meltano yaml file, you can add a table-key-properties and/or replication-key override using the metadata extra config.

Docs reference: https://docs.meltano.com/concepts/plugins#metadata-extra

Example yaml code:

extractors:
- name: tap-postgres
  metadata:
    some_stream_id:
      table-key-properties: [id]
      replication-key: created_at
      replication-method: INCREMENTAL

This example sets the primary key to id and the replication key to created_at.

Update regarding wildcards (2022-03-30)

Per @visch's comment, wildcards can also be used in stream names to match multiple streams at once. Such as in these example:

extractors:
- name: tap-postgres
  metadata:
    "*":
      # set all streams to "full table" mode
      replication-method: FULL_TABLE

And:

extractors:
- name: tap-postgres
  metadata:
    "*":
      # ignore primary keys for all streams
      table-key-properties: []