It seems like whenever adb client cannot communicate with adb server, it automatically starts it. Is there a way to prevent adb server from automatically starting like this?
You can specify server address and if adb think it is not a local address, then adb won't start a local server.
Addresses start with "local" are considered as local address, but "127.0.0.1" is not.
So you can set ADB_SERVER_SOCKET=127.0.0.1:5037 or ANDROID_ADB_SERVER_ADDRESS=127.0.0.1 to prevent starting local server.
If you are interested in the detail, see is_local_socket_spec in https://android.googlesource.com/platform/system/adb/+/d7285b7cdd139ada5d662b5f3c1c5a4e16462299/socket_spec.cpp#128