Search code examples
javatcpazure-load-balancerjfr

What exactly does read/write count represent in Java Fligh Record?

So, my situation on a high level is investigation of a source of SNAT port exhaustion. With no indication of any issues in our logs the only clue we managed to find using JFR profiling is this Read Count (screenshot from JDK Mission Control). This was recorded in about 20 minute timeframe.

enter image description here

According to our logs we did only about 70 HTTP requests; SNAT Connection Count metric in Azure shows about 1,4k per minute during the same period.

enter image description here

How do I interpret this Count in JFR? What exactly does it represent as far as TCP goes? I was not able to find any good explanation.

We are using Jersey HTTP Client, with ApacheConnector and PoolingHttpClientConnectionManagerBuilder with maximum 10 connections per route.

Solution

  • The count is the number of events emitted, i.e. Socket Read or Socket Write. By default, only events that exceed 20 ms are emitted. You can set the threshold 0 ms to get all the events, but be aware that overhead can be significant and the file can become huge.

    There are two ways you can change the settings:

    1. Use JMC

    Set the I/O threshold in the recording wizard, or create a custom .jfc using Window -> Template Manager and start it on command line:

    $ java -XX:StartFlightRecording:settings=my.jfc ...

    or

    $ jcmd <pid> JFR.start settings=my.jfc ...

    2. Edit the .jfc file manually.

    If you don't start the recording using JMC, copy the default.jfc file in JAVA_HOME/lib/jfr to a custom.jfc file and set the threshold to 0 ms for the jdk.SocketRead and jdk.SocketWrite event.

    <event name="jdk.SocketRead">
  <setting name="enabled">true</setting>
  <setting name="stackTrace">true</setting>
  <setting name="threshold" control="socket-threshold">0 ms</setting>
</event>

<event name="jdk.SocketWrite">
  <setting name="enabled">true</setting>
  <setting name="stackTrace">true</setting>
  <setting name="threshold" control="socket-threshold">0 ms</setting>
</event>

    JDK 17

    If you are using JDK 17, or later, you can also use the JAVA_HOME/bin/jfr tool.

    $ jfr configure --interactive

    Hit Enter until the wizard asks for the Socket I/O Threshold, type 0 ms and hit Enter until a custom.jfc file is written. Then use it like this:

    $ java -XX:StartFlightRecording:settings=custom.jfc ..

    or specify I/O threshold directly on command line:

    $ java -XX:StartFlightRecording:socket-threshold=0ms ...

$ jcmd <pid> JFR.start socket-threshold=0m