Search code examples
apache-kafkaapache-kafka-connectksqldbdebezium

is there a way to limit kafka connect heap space when debezium connector is fetching data from your sql server

i am trying to set up a connector that fetches data from an SQL server to use with apache kafka. I've set up all of the kafka services with a docker-compose file, however the SQL server is on another server. This is the configuration of my debezium connector in ksqldb:

create source connector sql_connector with 
('connector.class'='io.debezium.connector.sqlserver.SqlServerConnector',
'database.server.name'='sqlserver',
'database.hostname'= 'xxxxx',
'database.port'='1433',
'database.user'= 'xxxx',
'database.password'= 'xxxxxx',
'database.dbname'='xxxxxxxxx',
'database.history.kafka.bootstrap.servers'='broker:29092',
'database.history.kafka.topic'='dbz_dbhistory.sqlserver.asgard-01');

When i do this, i get a response that the connector is succesfully created however when i query ksqldb by using 'show connectors' for my connectors i get the following error message:

io.confluent.ksql.util.KsqlServerException: org.apache.http.conn.HttpHostConnectException: Connect to connect:8083 [connect/172.18.0.6] failed: Connection refused (Connection refused)
Caused by: org.apache.http.conn.HttpHostConnectException: Connect to
        connect:8083 [connect/172.18.0.6] failed: Connection refused (Connection
        refused)
Caused by: Could not connect to the server.
Caused by: Could not connect to the server.

When i inspect my kafka connect logs i can see that its issueing select statements to the server but after a while i get the following error and my kafka connect shuts down:

OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x00000000b5400000, 118489088, 0) failed; error='Not enough space' (errno=12)
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 118489088 bytes for committing reserved memory.
# An error report file with more information is saved as:

any ideas on how to fix this? Other then just giving my server more ram.

Solution

  • Your machine has less than ~118MB of free memory:

    Native memory allocation (mmap) failed to map 118489088 bytes for committing reserved memory

    You will need to increase or free up memory on the machine to get the JVM to start. If it's running, you can change the heap memory settings of the JVM using the following environment variable:

    KAFKA_HEAP_OPTS="-Xms256M -Xmx2G"