Search code examples
pythonapache-kafkanewrelic

Error in NewRelic on publishing to kafka in Python

I am publishing to kafka using kafka-python and have newrelic integrated in my application. the publish method of kafka-python fails due to an error Error 'MessageTransaction' object has no attribute 'destination_name' in newrelic

File "/Users/mishika/supply-metrics/app/kafka_manager/base_consumer_v2.py", line 55, in produce
    self.producer.send(topic, value=event_data, key=key).get()
  File "/Users/mishika/miniconda3/envs/myenv/lib/python3.10/site-packages/newrelic/hooks/messagebroker_kafkapython.py", line 68, in wrap_KafkaProducer_send
    return wrapped(
  File "/Users/mishika/miniconda3/envs/myenv/lib/python3.10/site-packages/kafka/producer/kafka.py", line 581, in send
    value_bytes = self._serialize(
  File "/Users/mishika/miniconda3/envs/myenv/lib/python3.10/site-packages/kafka/producer/kafka.py", line 714, in _serialize
    return f(data)
  File "/Users/mishika/miniconda3/envs/myenv/lib/python3.10/site-packages/newrelic/hooks/messagebroker_kafkapython.py", line 203, in _wrap_serializer
    topic = transaction.destination_name
AttributeError: 'MessageTransaction' object has no attribute 'destination_name'

my newrelic is initialised like this in main.py

import newrelic.agent
newrelic.agent.initialize()


[newrelic]
license_key = 
app_name = 
monitor_mode = true
log_level = debug
high_security = false
transaction_tracer.enabled = true
transaction_tracer.transaction_threshold = apdex_f
transaction_tracer.record_sql = obfuscated
transaction_tracer.stack_trace_threshold = 0.5
transaction_tracer.explain_enabled = true
transaction_tracer.explain_threshold = 0.5
transaction_tracer.function_trace =
error_collector.enabled = true
error_collector.ignore_errors = pycommon.exceptions.common_exceptions:DuplicateError rest_framework.exceptions:ValidationError rest_framework.exceptions:NotFound rest_framework.exceptions:ParseError
browser_monitoring.auto_instrument = true
thread_profiler.enabled = true
distributed_tracing.enabled = false

my metrics are being published to newrelic on logging operation along with transaction_tracer defined in my newrelic.ini file

I have removed the initialization part from main.py and removed metrics produce from logging operation and only kept what was defined in newrelic.ini file, But still facing the same issue

Solution

  • Disabling kafka producer transaction on newrelic by keeping below in newrelic.ini file you would be able to send the message to kafka

    [import-hook:kafka.producer.kafka]
enabled = false

    I have also posted this issue with newrelic-python-agent Error in NewRelic on publishing to kafka in Python for any further solve on enabling this kafka producer transaction on newrelic back