It appears that Neo4j 3.4.1 (Windows) is using more ports than what they state in their documentation (see https://neo4j.com/docs/operations-manual/current/configuration/ports/). In addition to the standard ports for connections it is also opening large blocks of ports. We've seen from about 20 to 250 ports being opened in relatively contiguous blocks anywhere in the 5000s to 35000s. These are all ports opened by the same Java process (Neo4j server run as a console application) or the Commons Daemon Server Runner process (Windows service). The ports appeared to be paired off where every two ports are established in a TCP connection with each other.
I cannot figure out what these ports or for or any documentation that mentions the use of these ports. It seems that the number of ports is relative to the size of the database.
I've also looked at Neo4j 2.3.2 and it seems to be using some mystery ports as well, though the block size was only six in that case.
Since we are using the APOC libraries, I removed it from the plugins directory and performed a restart of the server to see if that had any impact. That does not seem to be an issue.
It certainly would be nice to understand why the process is using these ports; however, the fundamental problem is that it is acting like a bad citizen, hogging large blocks of ports where better behaved applications tend to exist. Is there a way to control in what range they will be opened? It seems that if these port blocks are necessary, then they should be opened in the Ephemeral port range and not trounce upon other applications.
Edit: Per suggestion by @InverseFalcon, here is an example of occupied ports by the Java process using Neo4j 3.5.3:
H:\>netstat -a -o | find /I "17048"
TCP 0.0.0.0:7473 mysvr:0 LISTENING 17048
TCP 0.0.0.0:7474 mysvr:0 LISTENING 17048
TCP 0.0.0.0:7687 mysvr:0 LISTENING 17048
TCP 127.0.0.1:36418 mysvr:36419 ESTABLISHED 17048
TCP 127.0.0.1:36419 mysvr:36418 ESTABLISHED 17048
TCP 127.0.0.1:36420 mysvr:36421 ESTABLISHED 17048
TCP 127.0.0.1:36421 mysvr:36420 ESTABLISHED 17048
TCP 127.0.0.1:36422 mysvr:36423 ESTABLISHED 17048
TCP 127.0.0.1:36423 mysvr:36422 ESTABLISHED 17048
TCP 127.0.0.1:36424 mysvr:36425 ESTABLISHED 17048
TCP 127.0.0.1:36425 mysvr:36424 ESTABLISHED 17048
TCP 127.0.0.1:36426 mysvr:36427 ESTABLISHED 17048
TCP 127.0.0.1:36427 mysvr:36426 ESTABLISHED 17048
TCP 127.0.0.1:36428 mysvr:36429 ESTABLISHED 17048
TCP 127.0.0.1:36429 mysvr:36428 ESTABLISHED 17048
TCP 127.0.0.1:36430 mysvr:36431 ESTABLISHED 17048
TCP 127.0.0.1:36431 mysvr:36430 ESTABLISHED 17048
TCP 127.0.0.1:36432 mysvr:36433 ESTABLISHED 17048
TCP 127.0.0.1:36433 mysvr:36432 ESTABLISHED 17048
TCP 127.0.0.1:36434 mysvr:36435 ESTABLISHED 17048
TCP 127.0.0.1:36435 mysvr:36434 ESTABLISHED 17048
TCP 127.0.0.1:36436 mysvr:36437 ESTABLISHED 17048
TCP 127.0.0.1:36437 mysvr:36436 ESTABLISHED 17048
TCP 127.0.0.1:36438 mysvr:36439 ESTABLISHED 17048
TCP 127.0.0.1:36439 mysvr:36438 ESTABLISHED 17048
TCP 127.0.0.1:36440 mysvr:36441 ESTABLISHED 17048
TCP 127.0.0.1:36441 mysvr:36440 ESTABLISHED 17048
TCP 127.0.0.1:36442 mysvr:36443 ESTABLISHED 17048
TCP 127.0.0.1:36443 mysvr:36442 ESTABLISHED 17048
TCP 127.0.0.1:36444 mysvr:36445 ESTABLISHED 17048
TCP 127.0.0.1:36445 mysvr:36444 ESTABLISHED 17048
TCP 127.0.0.1:36446 mysvr:36447 ESTABLISHED 17048
TCP 127.0.0.1:36447 mysvr:36446 ESTABLISHED 17048
TCP 127.0.0.1:36448 mysvr:36449 ESTABLISHED 17048
TCP 127.0.0.1:36449 mysvr:36448 ESTABLISHED 17048
TCP 127.0.0.1:36450 mysvr:36451 ESTABLISHED 17048
TCP 127.0.0.1:36451 mysvr:36450 ESTABLISHED 17048
TCP 127.0.0.1:36452 mysvr:36453 ESTABLISHED 17048
TCP 127.0.0.1:36453 mysvr:36452 ESTABLISHED 17048
TCP 127.0.0.1:36454 mysvr:36455 ESTABLISHED 17048
TCP 127.0.0.1:36455 mysvr:36454 ESTABLISHED 17048
TCP 127.0.0.1:36456 mysvr:36457 ESTABLISHED 17048
TCP 127.0.0.1:36457 mysvr:36456 ESTABLISHED 17048
TCP 127.0.0.1:36458 mysvr:36459 ESTABLISHED 17048
TCP 127.0.0.1:36459 mysvr:36458 ESTABLISHED 17048
TCP 127.0.0.1:36460 mysvr:36461 ESTABLISHED 17048
TCP 127.0.0.1:36461 mysvr:36460 ESTABLISHED 17048
TCP 127.0.0.1:36462 mysvr:36463 ESTABLISHED 17048
TCP 127.0.0.1:36463 mysvr:36462 ESTABLISHED 17048
TCP 127.0.0.1:36464 mysvr:36465 ESTABLISHED 17048
TCP 127.0.0.1:36465 mysvr:36464 ESTABLISHED 17048
TCP 127.0.0.1:36466 mysvr:36467 ESTABLISHED 17048
TCP 127.0.0.1:36467 mysvr:36466 ESTABLISHED 17048
TCP 127.0.0.1:36468 mysvr:36469 ESTABLISHED 17048
TCP 127.0.0.1:36469 mysvr:36468 ESTABLISHED 17048
TCP 127.0.0.1:36470 mysvr:36471 ESTABLISHED 17048
TCP 127.0.0.1:36471 mysvr:36470 ESTABLISHED 17048
TCP 127.0.0.1:36472 mysvr:36473 ESTABLISHED 17048
TCP 127.0.0.1:36473 mysvr:36472 ESTABLISHED 17048
TCP 127.0.0.1:36474 mysvr:36475 ESTABLISHED 17048
TCP 127.0.0.1:36475 mysvr:36474 ESTABLISHED 17048
TCP 127.0.0.1:36476 mysvr:36477 ESTABLISHED 17048
TCP 127.0.0.1:36477 mysvr:36476 ESTABLISHED 17048
TCP 127.0.0.1:36478 mysvr:36479 ESTABLISHED 17048
TCP 127.0.0.1:36479 mysvr:36478 ESTABLISHED 17048
TCP [::]:7473 mysvr:0 LISTENING 17048
TCP [::]:7474 mysvr:0 LISTENING 17048
TCP [::]:7687 mysvr:0 LISTENING 17048
TCP [::1]:7687 mysvr:36483 ESTABLISHED 17048
TCP [::1]:7687 mysvr:36484 ESTABLISHED 17048
TCP [::1]:7687 mysvr:36485 ESTABLISHED 17048
TCP [::1]:7687 mysvr:36486 ESTABLISHED 17048
TCP [::1]:7687 mysvr:36487 ESTABLISHED 17048
If bolt is turned off, the block is smaller:
H:\>netstat -a -o | find /I "20520"
TCP 0.0.0.0:7473 mysvr:0 LISTENING 20520
TCP 0.0.0.0:7474 mysvr:0 LISTENING 20520
TCP 127.0.0.1:36902 mysvr:36903 ESTABLISHED 20520
TCP 127.0.0.1:36903 mysvr:36902 ESTABLISHED 20520
TCP 127.0.0.1:36904 mysvr:36905 ESTABLISHED 20520
TCP 127.0.0.1:36905 mysvr:36904 ESTABLISHED 20520
TCP 127.0.0.1:36906 mysvr:36907 ESTABLISHED 20520
TCP 127.0.0.1:36907 mysvr:36906 ESTABLISHED 20520
TCP 127.0.0.1:36908 mysvr:36909 ESTABLISHED 20520
TCP 127.0.0.1:36909 mysvr:36908 ESTABLISHED 20520
TCP 127.0.0.1:36910 mysvr:36911 ESTABLISHED 20520
TCP 127.0.0.1:36911 mysvr:36910 ESTABLISHED 20520
TCP 127.0.0.1:36912 mysvr:36913 ESTABLISHED 20520
TCP 127.0.0.1:36913 mysvr:36912 ESTABLISHED 20520
TCP [::]:7473 mysvr:0 LISTENING 20520
TCP [::]:7474 mysvr:0 LISTENING 20520
And if https is turned off, it is even smaller:
H:\>netstat -a -o | find /I "20556"
TCP 0.0.0.0:7474 mysvr:0 LISTENING 20556
TCP 127.0.0.1:36962 mysvr:36963 ESTABLISHED 20556
TCP 127.0.0.1:36963 mysvr:36962 ESTABLISHED 20556
TCP 127.0.0.1:36964 mysvr:36965 ESTABLISHED 20556
TCP 127.0.0.1:36965 mysvr:36964 ESTABLISHED 20556
TCP 127.0.0.1:36966 mysvr:36967 ESTABLISHED 20556
TCP 127.0.0.1:36967 mysvr:36966 ESTABLISHED 20556
TCP [::]:7474 mysvr:0 LISTENING 20556
So this may be a consequence that socketpairs are not supported on Windows, so pipes are implemented as loopback connections. While this originated with Windows XP it looks like it has continued for Windows systems. It looks as if the port numbers used have likewise been changed or at least left unrestricted.
This is captured at the end of this older Java bug:
Windows doesn't support socketpairs so Pipe is implemented as a loopback connection on that platform. The "client" side of that connection doesn't explicitly bind and so it allocated an ephemeral port in the range 1024-5000.
java.io.OutputStream usages would be implemented as a loopback in this way.
I'll do some followup to see if we can get any more info.