Search code examples
apache-kafkadebezium

debezium could not access file "decoderbufs" using postgres 11 with default plugin pgoutput

I'm new to kafka, I'm trying to use the debezium postgres connector. but even using postgres version 11 with the standard plugin I get this error: org.apache.kafka.connect.errors.ConnectException: org.postgresql.util.PSQLException: ERROR: could not access file "decoderbufs": No such file or directory

To run kafka / debezium I'm using the image of the fast-data-dev docker as you can see below

  # this is our kafka cluster.
  kafka-cluster:
    image: landoop/fast-data-dev:latest
    environment:
      ADV_HOST: 127.0.0.1         # Change to 192.168.99.100 if using Docker Toolbox
      RUNTESTS: 0                 # Disable Running tests so the cluster starts faster
    ports:
      - 2181:2181                 # Zookeeper
      - 3030:3030                 # Landoop UI
      - 8081-8083:8081-8083       # REST Proxy, Schema Registry, Kafka Connect ports
      - 9581-9585:9581-9585       # JMX Ports
      - 9092:9092                 # Kafka Broker

after running i can open my localhost: 3030 to choose the debezium connector, i configured it this way:

enter image description here

and I'm using aws postgres rds in version 11.5 I saw several tutorials using wal2json, but I didn't find it in rds.extensions and didn't see a way to add it. Anyway, as of version 10, debezium can use pgoutput and apparently no configuration is necessary.

the rds.logical_replication property is set to 1 when executing SHOW wal_level; in the terminal I see that it returns logical in the documentation says that you have to set max_wal_senders = 1 and max_replication_slots = 1 put in the rds the minimum is 5, so I left the default that is 10

I did not define the role REPLICATION because from what I understand in the rds there is no way

in this image you can see the version used is 11.5

enter image description here

but I get the error as you can see below

enter image description here

Solution

  • You haven't set "plugin.name" property to "pgoutput" in your Debezium connector properties which you've already figured out. But this answer is for others who don't know where to set this option and for more clarity.

    In the following connector properties:

    enter image description here

    As you haven't set plugin.name option it is taking the default value which is decoderbufs and that's why you were getting the following error:

    enter image description here

    Setting the "plugin.name" explicitly to "pgoutput" should solve the issue.

    { 
  "connector.class": "io.debezium.connector.postgresql.PostgresConnector", 
  "database.user": "postgres",
  "database.dbname": "xxxxx",
  "tasks.max": "1", 
  "database.hostname": "xxxx.rds.amazonaws.com", 
  "database.password": "xxxx", 
  "database.server.name": "database-1",
  "database.port": "5432",
  "plugin.name": "pgoutput" --> this property
}