Search code examples
eclipse-hono

How to know the connections number of MQTT adapter?


We want to do a performance test on Hono to verify that it supports 12000 device connections (MQTT), but I can't find the number of device connections in grafana's dashboard. I also checked the prometheus data endpoint of the MQTT adapter and found no metrics related to the number of connections. The metrics of the mqtt adapter are as follows

# HELP jvm_threads_live_threads The current number of live threads including both daemon and non-daemon threads
# TYPE jvm_threads_live_threads gauge
jvm_threads_live_threads{component_name="hono-mqtt",component_type="adapter",host="hono-poc-adapter-mqtt-vertx-6db45b86f6-wqpbm",} 30.0
# HELP hono_connections_authenticated
# TYPE hono_connections_authenticated gauge
hono_connections_authenticated{component_name="hono-mqtt",component_type="adapter",host="hono-poc-adapter-mqtt-vertx-6db45b86f6-wqpbm",tenant="loadtest",} 0.0
# HELP logback_events_total Number of error level events that made it to the logs
# TYPE logback_events_total counter
logback_events_total{component_name="hono-mqtt",component_type="adapter",host="hono-poc-adapter-mqtt-vertx-6db45b86f6-wqpbm",level="debug",} 813043.0
logback_events_total{component_name="hono-mqtt",component_type="adapter",host="hono-poc-adapter-mqtt-vertx-6db45b86f6-wqpbm",level="info",} 1276.0
logback_events_total{component_name="hono-mqtt",component_type="adapter",host="hono-poc-adapter-mqtt-vertx-6db45b86f6-wqpbm",level="trace",} 0.0
logback_events_total{component_name="hono-mqtt",component_type="adapter",host="hono-poc-adapter-mqtt-vertx-6db45b86f6-wqpbm",level="warn",} 1.0
logback_events_total{component_name="hono-mqtt",component_type="adapter",host="hono-poc-adapter-mqtt-vertx-6db45b86f6-wqpbm",level="error",} 8.0
# HELP jvm_memory_used_bytes The amount of used memory
# TYPE jvm_memory_used_bytes gauge
jvm_memory_used_bytes{area="heap",component_name="hono-mqtt",component_type="adapter",host="hono-poc-adapter-mqtt-vertx-6db45b86f6-wqpbm",id="Tenured Gen",} 2.987608E7
jvm_memory_used_bytes{area="nonheap",component_name="hono-mqtt",component_type="adapter",host="hono-poc-adapter-mqtt-vertx-6db45b86f6-wqpbm",id="CodeHeap 'profiled nmethods'",} 1.6964352E7
jvm_memory_used_bytes{area="heap",component_name="hono-mqtt",component_type="adapter",host="hono-poc-adapter-mqtt-vertx-6db45b86f6-wqpbm",id="Survivor Space",} 315472.0
jvm_memory_used_bytes{area="nonheap",component_name="hono-mqtt",component_type="adapter",host="hono-poc-adapter-mqtt-vertx-6db45b86f6-wqpbm",id="Compressed Class Space",} 6350352.0
jvm_memory_used_bytes{area="nonheap",component_name="hono-mqtt",component_type="adapter",host="hono-poc-adapter-mqtt-vertx-6db45b86f6-wqpbm",id="CodeHeap 'non-profiled nmethods'",} 1.046592E7
jvm_memory_used_bytes{area="nonheap",component_name="hono-mqtt",component_type="adapter",host="hono-poc-adapter-mqtt-vertx-6db45b86f6-wqpbm",id="CodeHeap 'non-nmethods'",} 1260928.0
jvm_memory_used_bytes{area="nonheap",component_name="hono-mqtt",component_type="adapter",host="hono-poc-adapter-mqtt-vertx-6db45b86f6-wqpbm",id="Metaspace",} 5.6632848E7
jvm_memory_used_bytes{area="heap",component_name="hono-mqtt",component_type="adapter",host="hono-poc-adapter-mqtt-vertx-6db45b86f6-wqpbm",id="Eden Space",} 7935840.0
# HELP process_start_time_seconds Start time of the process since unix epoch.
# TYPE process_start_time_seconds gauge
process_start_time_seconds{component_name="hono-mqtt",component_type="adapter",host="hono-poc-adapter-mqtt-vertx-6db45b86f6-wqpbm",} 1.591340143952E9
# HELP jvm_classes_loaded_classes The number of classes that are currently loaded in the Java virtual machine
# TYPE jvm_classes_loaded_classes gauge
jvm_classes_loaded_classes{component_name="hono-mqtt",component_type="adapter",host="hono-poc-adapter-mqtt-vertx-6db45b86f6-wqpbm",} 9043.0
# HELP jvm_threads_states_threads The current number of threads having NEW state
# TYPE jvm_threads_states_threads gauge
jvm_threads_states_threads{component_name="hono-mqtt",component_type="adapter",host="hono-poc-adapter-mqtt-vertx-6db45b86f6-wqpbm",state="timed-waiting",} 2.0
jvm_threads_states_threads{component_name="hono-mqtt",component_type="adapter",host="hono-poc-adapter-mqtt-vertx-6db45b86f6-wqpbm",state="terminated",} 0.0
jvm_threads_states_threads{component_name="hono-mqtt",component_type="adapter",host="hono-poc-adapter-mqtt-vertx-6db45b86f6-wqpbm",state="waiting",} 22.0
jvm_threads_states_threads{component_name="hono-mqtt",component_type="adapter",host="hono-poc-adapter-mqtt-vertx-6db45b86f6-wqpbm",state="blocked",} 0.0
jvm_threads_states_threads{component_name="hono-mqtt",component_type="adapter",host="hono-poc-adapter-mqtt-vertx-6db45b86f6-wqpbm",state="new",} 0.0
jvm_threads_states_threads{component_name="hono-mqtt",component_type="adapter",host="hono-poc-adapter-mqtt-vertx-6db45b86f6-wqpbm",state="runnable",} 6.0
# HELP jvm_buffer_memory_used_bytes An estimate of the memory that the Java virtual machine is using for this buffer pool
# TYPE jvm_buffer_memory_used_bytes gauge
jvm_buffer_memory_used_bytes{component_name="hono-mqtt",component_type="adapter",host="hono-poc-adapter-mqtt-vertx-6db45b86f6-wqpbm",id="mapped",} 0.0
jvm_buffer_memory_used_bytes{component_name="hono-mqtt",component_type="adapter",host="hono-poc-adapter-mqtt-vertx-6db45b86f6-wqpbm",id="direct",} 5.034343E7
# HELP process_files_open_files The open file descriptor count
# TYPE process_files_open_files gauge
process_files_open_files{component_name="hono-mqtt",component_type="adapter",host="hono-poc-adapter-mqtt-vertx-6db45b86f6-wqpbm",} 32.0
# HELP process_files_max_files The maximum file descriptor count
# TYPE process_files_max_files gauge
process_files_max_files{component_name="hono-mqtt",component_type="adapter",host="hono-poc-adapter-mqtt-vertx-6db45b86f6-wqpbm",} 1048576.0
# HELP jvm_gc_memory_allocated_bytes_total Incremented for an increase in the size of the young generation memory pool after one GC to before the next
# TYPE jvm_gc_memory_allocated_bytes_total counter
jvm_gc_memory_allocated_bytes_total{component_name="hono-mqtt",component_type="adapter",host="hono-poc-adapter-mqtt-vertx-6db45b86f6-wqpbm",} 7.3979744768E10
# HELP jvm_threads_peak_threads The peak live thread count since the Java virtual machine started or peak was reset
# TYPE jvm_threads_peak_threads gauge
jvm_threads_peak_threads{component_name="hono-mqtt",component_type="adapter",host="hono-poc-adapter-mqtt-vertx-6db45b86f6-wqpbm",} 30.0
# HELP hono_connections_unauthenticated
# TYPE hono_connections_unauthenticated gauge
hono_connections_unauthenticated{component_name="hono-mqtt",component_type="adapter",host="hono-poc-adapter-mqtt-vertx-6db45b86f6-wqpbm",} 0.0
# HELP jvm_gc_memory_promoted_bytes_total Count of positive increases in the size of the old generation memory pool before GC to after GC
# TYPE jvm_gc_memory_promoted_bytes_total counter
jvm_gc_memory_promoted_bytes_total{component_name="hono-mqtt",component_type="adapter",host="hono-poc-adapter-mqtt-vertx-6db45b86f6-wqpbm",} 2.1365048E8
# HELP jvm_buffer_count_buffers An estimate of the number of buffers in the pool
# TYPE jvm_buffer_count_buffers gauge
jvm_buffer_count_buffers{component_name="hono-mqtt",component_type="adapter",host="hono-poc-adapter-mqtt-vertx-6db45b86f6-wqpbm",id="mapped",} 0.0
jvm_buffer_count_buffers{component_name="hono-mqtt",component_type="adapter",host="hono-poc-adapter-mqtt-vertx-6db45b86f6-wqpbm",id="direct",} 9.0
# HELP hono_connections_authenticated_duration_seconds_max
# TYPE hono_connections_authenticated_duration_seconds_max gauge
hono_connections_authenticated_duration_seconds_max{component_name="hono-mqtt",component_type="adapter",host="hono-poc-adapter-mqtt-vertx-6db45b86f6-wqpbm",tenant="loadtest",} 20.0
# HELP hono_connections_authenticated_duration_seconds
# TYPE hono_connections_authenticated_duration_seconds summary
hono_connections_authenticated_duration_seconds_count{component_name="hono-mqtt",component_type="adapter",host="hono-poc-adapter-mqtt-vertx-6db45b86f6-wqpbm",tenant="loadtest",} 39.0
hono_connections_authenticated_duration_seconds_sum{component_name="hono-mqtt",component_type="adapter",host="hono-poc-adapter-mqtt-vertx-6db45b86f6-wqpbm",tenant="loadtest",} 554.666
# HELP jvm_gc_pause_seconds Time spent in GC pause
# TYPE jvm_gc_pause_seconds summary
jvm_gc_pause_seconds_count{action="end of minor GC",cause="GCLocker Initiated GC",component_name="hono-mqtt",component_type="adapter",host="hono-poc-adapter-mqtt-vertx-6db45b86f6-wqpbm",} 1.0
jvm_gc_pause_seconds_sum{action="end of minor GC",cause="GCLocker Initiated GC",component_name="hono-mqtt",component_type="adapter",host="hono-poc-adapter-mqtt-vertx-6db45b86f6-wqpbm",} 0.077
jvm_gc_pause_seconds_count{action="end of major GC",cause="Allocation Failure",component_name="hono-mqtt",component_type="adapter",host="hono-poc-adapter-mqtt-vertx-6db45b86f6-wqpbm",} 12.0
jvm_gc_pause_seconds_sum{action="end of major GC",cause="Allocation Failure",component_name="hono-mqtt",component_type="adapter",host="hono-poc-adapter-mqtt-vertx-6db45b86f6-wqpbm",} 2.14
jvm_gc_pause_seconds_count{action="end of minor GC",cause="Allocation Failure",component_name="hono-mqtt",component_type="adapter",host="hono-poc-adapter-mqtt-vertx-6db45b86f6-wqpbm",} 4483.0
jvm_gc_pause_seconds_sum{action="end of minor GC",cause="Allocation Failure",component_name="hono-mqtt",component_type="adapter",host="hono-poc-adapter-mqtt-vertx-6db45b86f6-wqpbm",} 6.652
jvm_gc_pause_seconds_count{action="end of major GC",cause="Metadata GC Threshold",component_name="hono-mqtt",component_type="adapter",host="hono-poc-adapter-mqtt-vertx-6db45b86f6-wqpbm",} 1.0
jvm_gc_pause_seconds_sum{action="end of major GC",cause="Metadata GC Threshold",component_name="hono-mqtt",component_type="adapter",host="hono-poc-adapter-mqtt-vertx-6db45b86f6-wqpbm",} 0.12
# HELP jvm_gc_pause_seconds_max Time spent in GC pause
# TYPE jvm_gc_pause_seconds_max gauge
jvm_gc_pause_seconds_max{action="end of minor GC",cause="GCLocker Initiated GC",component_name="hono-mqtt",component_type="adapter",host="hono-poc-adapter-mqtt-vertx-6db45b86f6-wqpbm",} 0.0
jvm_gc_pause_seconds_max{action="end of major GC",cause="Allocation Failure",component_name="hono-mqtt",component_type="adapter",host="hono-poc-adapter-mqtt-vertx-6db45b86f6-wqpbm",} 0.0
jvm_gc_pause_seconds_max{action="end of minor GC",cause="Allocation Failure",component_name="hono-mqtt",component_type="adapter",host="hono-poc-adapter-mqtt-vertx-6db45b86f6-wqpbm",} 0.002
jvm_gc_pause_seconds_max{action="end of major GC",cause="Metadata GC Threshold",component_name="hono-mqtt",component_type="adapter",host="hono-poc-adapter-mqtt-vertx-6db45b86f6-wqpbm",} 0.0
# HELP process_uptime_seconds The uptime of the Java virtual machine
# TYPE process_uptime_seconds gauge
process_uptime_seconds{component_name="hono-mqtt",component_type="adapter",host="hono-poc-adapter-mqtt-vertx-6db45b86f6-wqpbm",} 345543.191
# HELP process_cpu_usage The "recent cpu usage" for the Java Virtual Machine process
# TYPE process_cpu_usage gauge
process_cpu_usage{component_name="hono-mqtt",component_type="adapter",host="hono-poc-adapter-mqtt-vertx-6db45b86f6-wqpbm",} 6.546644844517185E-4
# HELP jvm_memory_committed_bytes The amount of memory in bytes that is committed for the Java virtual machine to use
# TYPE jvm_memory_committed_bytes gauge
jvm_memory_committed_bytes{area="heap",component_name="hono-mqtt",component_type="adapter",host="hono-poc-adapter-mqtt-vertx-6db45b86f6-wqpbm",id="Tenured Gen",} 4.8328704E7
jvm_memory_committed_bytes{area="nonheap",component_name="hono-mqtt",component_type="adapter",host="hono-poc-adapter-mqtt-vertx-6db45b86f6-wqpbm",id="CodeHeap 'profiled nmethods'",} 1.7891328E7
jvm_memory_committed_bytes{area="heap",component_name="hono-mqtt",component_type="adapter",host="hono-poc-adapter-mqtt-vertx-6db45b86f6-wqpbm",id="Survivor Space",} 2424832.0
jvm_memory_committed_bytes{area="nonheap",component_name="hono-mqtt",component_type="adapter",host="hono-poc-adapter-mqtt-vertx-6db45b86f6-wqpbm",id="Compressed Class Space",} 7077888.0
jvm_memory_committed_bytes{area="nonheap",component_name="hono-mqtt",component_type="adapter",host="hono-poc-adapter-mqtt-vertx-6db45b86f6-wqpbm",id="CodeHeap 'non-profiled nmethods'",} 1.0747904E7
jvm_memory_committed_bytes{area="nonheap",component_name="hono-mqtt",component_type="adapter",host="hono-poc-adapter-mqtt-vertx-6db45b86f6-wqpbm",id="CodeHeap 'non-nmethods'",} 2555904.0
jvm_memory_committed_bytes{area="nonheap",component_name="hono-mqtt",component_type="adapter",host="hono-poc-adapter-mqtt-vertx-6db45b86f6-wqpbm",id="Metaspace",} 5.8368E7
jvm_memory_committed_bytes{area="heap",component_name="hono-mqtt",component_type="adapter",host="hono-poc-adapter-mqtt-vertx-6db45b86f6-wqpbm",id="Eden Space",} 1.9464192E7
# HELP system_cpu_usage The "recent cpu usage" for the whole system
# TYPE system_cpu_usage gauge
system_cpu_usage{component_name="hono-mqtt",component_type="adapter",host="hono-poc-adapter-mqtt-vertx-6db45b86f6-wqpbm",} 0.036333878887070375
# HELP jvm_gc_live_data_size_bytes Size of old generation memory pool after a full GC
# TYPE jvm_gc_live_data_size_bytes gauge
jvm_gc_live_data_size_bytes{component_name="hono-mqtt",component_type="adapter",host="hono-poc-adapter-mqtt-vertx-6db45b86f6-wqpbm",} 2.7826992E7
# HELP system_load_average_1m The sum of the number of runnable entities queued to available processors and the number of runnable entities running on the available processors averaged over a period of time
# TYPE system_load_average_1m gauge
system_load_average_1m{component_name="hono-mqtt",component_type="adapter",host="hono-poc-adapter-mqtt-vertx-6db45b86f6-wqpbm",} 0.93
# HELP jvm_classes_unloaded_classes_total The total number of classes unloaded since the Java virtual machine has started execution
# TYPE jvm_classes_unloaded_classes_total counter
jvm_classes_unloaded_classes_total{component_name="hono-mqtt",component_type="adapter",host="hono-poc-adapter-mqtt-vertx-6db45b86f6-wqpbm",} 197.0
# HELP jvm_gc_max_data_size_bytes Max size of old generation memory pool
# TYPE jvm_gc_max_data_size_bytes gauge
jvm_gc_max_data_size_bytes{component_name="hono-mqtt",component_type="adapter",host="hono-poc-adapter-mqtt-vertx-6db45b86f6-wqpbm",} 1.146486784E9
# HELP jvm_memory_max_bytes The maximum amount of memory in bytes that can be used for memory management
# TYPE jvm_memory_max_bytes gauge
jvm_memory_max_bytes{area="heap",component_name="hono-mqtt",component_type="adapter",host="hono-poc-adapter-mqtt-vertx-6db45b86f6-wqpbm",id="Tenured Gen",} 1.146486784E9
jvm_memory_max_bytes{area="nonheap",component_name="hono-mqtt",component_type="adapter",host="hono-poc-adapter-mqtt-vertx-6db45b86f6-wqpbm",id="CodeHeap 'profiled nmethods'",} 1.22912768E8
jvm_memory_max_bytes{area="heap",component_name="hono-mqtt",component_type="adapter",host="hono-poc-adapter-mqtt-vertx-6db45b86f6-wqpbm",id="Survivor Space",} 5.7278464E7
jvm_memory_max_bytes{area="nonheap",component_name="hono-mqtt",component_type="adapter",host="hono-poc-adapter-mqtt-vertx-6db45b86f6-wqpbm",id="Compressed Class Space",} 1.073741824E9
jvm_memory_max_bytes{area="nonheap",component_name="hono-mqtt",component_type="adapter",host="hono-poc-adapter-mqtt-vertx-6db45b86f6-wqpbm",id="CodeHeap 'non-profiled nmethods'",} 1.22916864E8
jvm_memory_max_bytes{area="nonheap",component_name="hono-mqtt",component_type="adapter",host="hono-poc-adapter-mqtt-vertx-6db45b86f6-wqpbm",id="CodeHeap 'non-nmethods'",} 5828608.0
jvm_memory_max_bytes{area="nonheap",component_name="hono-mqtt",component_type="adapter",host="hono-poc-adapter-mqtt-vertx-6db45b86f6-wqpbm",id="Metaspace",} -1.0
jvm_memory_max_bytes{area="heap",component_name="hono-mqtt",component_type="adapter",host="hono-poc-adapter-mqtt-vertx-6db45b86f6-wqpbm",id="Eden Space",} 4.58620928E8
# HELP jvm_threads_daemon_threads The current number of live daemon threads
# TYPE jvm_threads_daemon_threads gauge
jvm_threads_daemon_threads{component_name="hono-mqtt",component_type="adapter",host="hono-poc-adapter-mqtt-vertx-6db45b86f6-wqpbm",} 6.0
# HELP system_cpu_count The number of processors available to the Java virtual machine
# TYPE system_cpu_count gauge
system_cpu_count{component_name="hono-mqtt",component_type="adapter",host="hono-poc-adapter-mqtt-vertx-6db45b86f6-wqpbm",} 1.0
# HELP jvm_buffer_total_capacity_bytes An estimate of the total capacity of the buffers in this pool
# TYPE jvm_buffer_total_capacity_bytes gauge
jvm_buffer_total_capacity_bytes{component_name="hono-mqtt",component_type="adapter",host="hono-poc-adapter-mqtt-vertx-6db45b86f6-wqpbm",id="mapped",} 0.0
jvm_buffer_total_capacity_bytes{component_name="hono-mqtt",component_type="adapter",host="hono-poc-adapter-mqtt-vertx-6db45b86f6-wqpbm",id="direct",} 5.0343429E7
  • How should I know how many devices the MQTT adapter has connected?
  • In general, how many connections can a 1c2G MQTT POD support?

Solution

  • Hono's components report several metrics like number of connected devices, messages transferred etc. These metrics are reported to a monitoring backend which usually implements a time series data store. Hono by default supports Prometheus for that purpose. You should be able to retrieve the relevant information from the Prometheus server using its query API.

    For a pod with 1 CPU and 2GB of memory available, I would think that you should be able to connect at least 25000 devices. However, note that the number of connections is not only limited by CPU and RAM but also by the number of file descriptors available to the OS and maybe other factors.