Tomcat JDBC connection amount vs netstat ESTABLISHED connections

Question

On one of our environments there is tomcat 8 with following JDBC datasource configuration:

<Resource   name="jdbc/mydatasource"
            auth="Container"
            type="javax.sql.DataSource"
            factory="org.apache.tomcat.jdbc.pool.DataSourceFactory"
            driverClassName="oracle.jdbc.OracleDriver"
            url="jdbc:oracle:thin:@myhostname:1521:dbname"
            username="username"
            password="password"
            removeAbandoned="false"
            removeAbandonedTimeout="300"
            logAbandoned="true"
            initialSize="5"
            maxActive="100"
            maxIdle="100"
            minIdle="5"
            maxWait="120000"
/>

Netstat command returns following results:

netstat -anu | grep ${pid}| grep ESTABLISHED | grep ${myhostname}:1521 | wc -l

55

This result remains stable during several days.

At the same time I monitor tomcat datasource attributes via JMX:

numActive = 0

numIdle = 5

These results also remain stable during several days.

It looks like tomcat creates JDBC connection pool with initial size of 5 connections, keeps amount of idle connections at 5, but for some reasons amount of established connections showed by netstat is 11 times bigger.

When minIdle="5" in context.xml and JMX shows numActive+numIdle=5 netstat should also show 5 established connections, shouldn't it?

I suppose that while sustaining JDBC pool tomcat continuously opens and closes connections to database, which remain in ESTABLISHED state from netstat point of view until being dropped by database.

My question is how to change tomcat datasource configuration to make netstat output consistent with this configuration?

Thank you very much in advance.

Answer 1

There are some things to look at here:
1) Your netstat command is looking at UDP connections (-u) only which is not what you want. Use instead:

netstat -ant | grep -c "${myhostname}:1521.*ESTABLISHED"

# -ant : all, numerical outptut, TCP connections
# grep -c : returns count of matches
# no need to grep PID unless you have more than one tomcat instance,
#  just matching host:1521 is enough.

2) Are local ports on established connections changing? If yes then the connections are actively opened and closed by the pool maintenance stuff.
3) Is DBCP JMX correctly configured? Perhaps you are monitoring this and is not what you want.

To watch changes continuously every 1 second use:

watch -n1 "netstat -ant | grep ':1521.*ESTABLISHED' | nl | tail -n 5"
    11  tcp        0      0 192.168.1.7:50890       172.17.0.23:1521     ESTABLISHED
    12  tcp        0      0 192.168.1.7:58192       172.17.0.23:1521       ESTABLISHED
    13  tcp        0      0 192.168.1.7:54224       172.17.0.23:1521        ESTABLISHED
    14  tcp        0      0 192.168.1.7:34500       172.17.0.23:1521     ESTABLISHED
    15  tcp        0      0 192.168.1.7:54888       172.17.0.23:1521      ESTABLISHED