Socket serialization slowdown

Question

I just want to preface that I'm new to socket serialization and before downvoting please suggest what I can add to edit in. I've tried to break code down as small as possible as the project is pretty large.

I'm trying to a create a very simple RPC type middleware where client can invoke methods on server and retrieve object back if the method make such an object. I'm using FST-Serializer library to serialize my object and send them over the network/socket.

While I've gotten it up and running, I have found a very weird issue where the performance of serialization slowdown significantly when this occurs (code examples below):

Client Serialize params/method name -> Server retrieves information and executes method.

Which would mean network slow or implementaion cannot invoke it fast enough, however if this occurs it speeds up significantly:

Client Serialize params/method name -> Server retrieves information and executes method -> Server serializes an object sends it back to client.

While this at first doesn't seem like a big deal, for void methods it can become annoying as I have to send back dummy data meaning I have to spent a network trip for no reason.

I thought maybe this might be buffer is not big enough but I've experiement with different buffer sizes and nothing seems to solve it, I also want to clarify that app still works fine (i.e no blocking) just a performance timing hit. So my question is what can cause such as slow down and is it preventable/fixable?

I ran YouKit performance to see hotspot analysis (new to this too) and it seems BufferedReader.Read method slows down a lot.

With Dummy data sent back: Without dummy data sent back:

Snippet of ServerThread that listens to incoming calls (Client code is similar creates input/output same with same buffersize):

 public ServerThread(Socket connection){
        this.connection = connection;
        isActive = true;
    }

@Override
public void run() {
    try {
        input = new BufferedInputStream(connection.getInputStream(), BUFFER_SIZE);
        output = new BufferedOutputStream(connection.getOutputStream(), BUFFER_SIZE); //Buffersize is 512_000 atm

        do{
            Method m = (Method) readObject(input);
            //Invoke method via reflection
            Server.methodNames.get(m.getName()).invoke(this, m.getParams());
        }while(isActive);

        connection.close();
    } catch (Exception e) {
        e.printStackTrace();
    }
}

@Override
public void transactionWillEnd() {
    this.currentTransactionLog = null;
    // This sends dummy data back to client or slowdown will occur
    try{
        writeObject(output, "SUCCESS");
        output.flush();
    } catch(Exception e){
        e.printStackTrace();
    }
}

Method class that's serialized from client sent to Serverthread

public class Method implements Serializable{
    private String name;
    private Object[] params;

    public Method(String name, Object...params) {
        this.name = name;
        this.params = params;
    }

    public String getName() {
        return name;
    }

    public Object[] getParams() {
        return params;
    }
}

TCPSerializer based off TCPObjectSocket in FST (inherited by serverthread&client:

public class TCPSerializer {

private static FSTConfiguration config;
public static int BUFFER_SIZE = 512_000;

AtomicBoolean readLock = new AtomicBoolean(false);
AtomicBoolean writeLock = new AtomicBoolean(false);

public TCPSerializer() {
    config = FSTConfiguration.createDefaultConfiguration();
}

public Object readObject(InputStream input) throws Exception{
    try {
        while ( !readLock.compareAndSet(false,true) );
        return config.decodeFromStream(input);
    } finally {
        readLock.set(false);
    }
}

public void writeObject(OutputStream output, Object toWrite) throws Exception{
    try {
        while ( !writeLock.compareAndSet(false,true) );
        config.encodeToStream(output, toWrite);
    } finally {
        writeLock.set(false);
    }
}

Example on how Client calls method:

@Override
public void transactionWillEnd() {
    String methodName = Helpers.getMethodName(0);
    Method m = new Method(methodName);
    try {
        client.writeObject(client.getOutput(), m);
        client.flush();
//Read dummy data before continuing.
        String verify = (String) client.readObject(client.getInput());
        if(!verify.equals("SUCCESS")) throw new Exception(verify);
    } catch (Exception e) {
        e.printStackTrace();
    }
}

Answer 1

Nagle's algorithm can slow down a connection significant if it not performing a request/response action. I have seen it wait 40 ms for another block of bytes to coalesce into a packet.

I suggest you try turning it off if you are streaming data without a reply.

socket.setTcpNoDelay(true);

https://docs.oracle.com/javase/9/docs/api/java/net/Socket.html#setTcpNoDelay(boolean)