Search code examples
apache-kafkaoracle-golden-gate

OracleGoldengate BigData for Kafka does not produce UPDATE messages

I'm experimenting with OGG BD (v12.3.2.1.1) to propagate all changes from an Oracle DB (11.2.0.4) to Kafka topics.

I got the basic setup all working, and INSERT and DELETE to the schemas/tables I'm monitoring are captured and produce Kafka messages onto the configured topics (1 topic per table). So far everything as expected.

But UPDATE operations on the database do not work. They are not producing any error, but simply no corresponding messages appears on the topic. I've verified with the logdump utility that the UPDATE is actually propagated into the trail file, but the kafkahandler does not pick it up.

Any ideas what might be wrong? Configurations are below

OGG PRM file

SETENV (TZ = "UTC")
REPLICAT kafkajsn
TARGETDB LIBFILE libggjava.so SET property=dirprm/kafkajsn.props

REPORTCOUNT EVERY 1 MINUTES, RATE
REPERROR DEFAULT DISCARD

APPLYNOOPUPDATES
ALLOWNOOPUPDATES
GETTRUNCATES
GETUPDATEBEFORES

DDL INCLUDE MAPPED

GROUPTRANSOPS 100

-- all tables have an 'ID' column
MAP *.*, TARGET *.*, KEYCOLS(ID);

Kafkahandler parameter file

gg.handlerlist = kafkahandler
gg.handler.kafkahandler.type=kafka
gg.handler.kafkahandler.KafkaProducerConfigFile=kafka_producer.properties

gg.handler.kafkahandler.topicMappingTemplate=${schemaName}_${tableName}
gg.handler.kafkahandler.keyMappingTemplate=${primaryKeys}

gg.handler.kafkahandler.SchemaTopicName=SCHEMA_CHANGES

gg.handler.kafkahandler.mode=op
gg.handler.kafkahandler.format=json_row
gg.handler.kafkahandler.format.includePrimaryKeys=true

gg.classpath=dirprm/:/etc/kafka/conf:/usr/hdp/current/kafka-broker/libs/*

javawriter.bootoptions=-Xmx1g -Xms1g -Djava.class.path=ggjava/ggjava.jar

The logdump of a trail file for the REPLICAT (this was an UPDATE ... SET comment = 'foo122' WHERE id=...)

2019/05/04 21:10:02.000.601 GGSUnifiedUpdate     Len    30 RBA 64926561
Name: ORDER  (TDR Index: 19)
After  Image:                                             Partition 12   G  s  
0c00 0000 0400 0800 0000 0400 666f 6f31 0400 0a00 | ............foo1.... 
0000 0600 666f 6f31 3232                          | ....foo122 
Before Image          Len    16 (x00000010)
BeforeColumnLen     12 (x0000000c)
Column     4 (x0004), Len     8 (x0008) 

After Image           Len    14 (x0000000e)
Column     4 (x0004), Len    10 (x000a)

EDIT: error message

I was wrong when I said it didn't produce an error message, I just didn't see it. In the discard file (.dsc) I get

Operation failed at seqno 45 rba 108411
Discarding record on action DISCARD on error 0
Problem replicating ORDER to ORDER
Mapping problem with unified update record (target format)...
*
TRANSACTION_COMMENT = folkjhlkjhlkjhkljl jk lj5345u
000000: 66 6f 6c 6b 6a 68 6c 6b 6a 68 6c 6b 6a 68 6b 6c |folkjhlkjhlkjhkl|
000010: 6a 6c 20 6a 6b 20 6c 6a 35 33 34 35 75          |jl jk lj5345u   |

TRANSACTION_COMMENT = folkjhlkjhlkjhkljl jk ljjhlkjhlkh5345u
000000: 66 6f 6c 6b 6a 68 6c 6b 6a 68 6c 6b 6a 68 6b 6c |folkjhlkjhlkjhkl|
000010: 6a 6c 20 6a 6b 20 6c 6a 6a 68 6c 6b 6a 68 6c 6b |jl jk ljjhlkjhlk|
000020: 68 35 33 34 35 75                               |h5345u          |
Solution

  • I found out what the problem was. I was missing the step to ADD TRANDATA to all the tables in the source database. I had done this before, but during testing I had re-setup the whole schema and of course those setting were missing.

    After applying it to all tables, I now start seeing the updates coming in as Kafka messages.

    I still need to find out how to produce full before/after images in the Kafka message, currently it only logs the ID columns and the modified ones.