Is there a way or tool to reset connection to one side only?
For example tcpkill sends RST packets to both sides, but I need to simulate case when connection left half opened: client closed connection, but server doesn't know about it.
Iptables configuration could be also helpful, when by some short timeout it starts rejecting client packets: --reject-with tcp-reset
In a Java program, calling close() on an input stream or output stream obtained from a Socket will close one side of the connection.
However, you seem to be asking if one application can "half close" a TCP/IP connection belonging to another proccess
As far as I am aware, an ordinary use-space application (Java or not) cannot close or half-close another application's sockets. Certainly not on a UNIX / Linux system. Ditto for a privileged user-space application.
So this leaves you with tools like tcpkill that work by injecting raw packets. It is possible to do the same kind of thing in Java using a native library (e.g. jpcap) to do the "dirty work" .... provided that your Java application is privileged. But it would be simpler to just use Process.exec(....) to run tcpkill or equivalent.
Iptables configuration could be also helpful, when by some short timeout it starts rejecting client packets:
--reject-with tcp-reset
I wouldn't do that. You risk leaving your system or virtual machine with an iptables configuration that causes problems, or worse.