Search code examples
rmi

RMI java.rmi.NoSuchObjectException and Port reuse in heavy concurrency

I am working on an application which has following structure

APPLICATION DESIGN:

  • One machine say MACHINE1 has running 6 different JVMs. Each JVM is running APP1 in it. So collectively MACHINE1 has 6 parallel executions of java application APP1
  • Another machine MACHINE2 runs Single JVM which has application APP2 running.
  • APP2 has different kind of JAVA Service Libraries like Service1, Service2 and service3 etc. If Any APP1 Of MACHINE1 wants to Perform Service1 then it will invoke Same instance of APP2 and pass Request Object. APP2 reads the Request object and on the basis of that it invokes the requested service and returns the result to APP1.

  • Services of APP2 are using input data in form of xml files of APP1. So we are transferring data of APP1 to APP2 in form of file InputStream

  • APP1 and APP2 are using RMI for communication. Services of APP2 perform operation on APP1 xml files to generate results. To send data from APP1 to APP2 we pick a port of MACHINE1 and export the remote Object of input Stream. In Machine1 we are maintaing POOL of integers which refers to Ports. APP1 picks a port integer from the pool and export the inputStream(We have created RMIInputStream).

PROBLEM:

When we execute application with less concurrency then it worked fine. But when we increase the concurrency then we starts getting following exception "java.rmi.NoSuchObjectException: no such object in table" This exception arises when we are reading the data from RemoteInputStream.

I have tried to solve this problem and focused on RemoteInputStream Object. I have seen lot many responses which states about the weak reference of remote object. I have seen it and found that reference is not weak in my case.

I have also modified the dfc properties by increasing the lease value but no effect in my case.

MY OBSERVATION ON THIS PROBLEM:

During testing I observed that every thing works fine if all ports used single time. Suppose If I allocate 20 ports in pool than I will not get any exception in first 20 times. I will get exception when I will start using the same port second time. If I increase the ports count from 20 to 600 then I wll not get exception in first 600 times. It looks like some problem is there with port re-use. I have added a wait time of 1 minute before sending port integer back to pool. In this case my problem got reduced drastically.

So my conclusion if I export an input Stream object in MACHINE1's port say 55800 and MACHINE2 will read it quicky and APP1 unexport the data and again try to export another remote input Stream object then it will create problem.

I need you all experts view on my problem. Is this problem related to TCP/IP protocol? Am I trying to reuse the same port which is still active in terms of TCP/IP life cycle?
Is there any good way to handle this problem rather than introducing the Wait in returning port.

In production environment in MACHINE1 we have been allocated certain range of Ports integers. Our application is allowed to use only ports between this range. Firewall is open for this specific range of Ports. So when we want to export inputStream we have to be sure that the data should be exported in allowed Port range. So we are using following code

UnicastRemoteObject.exportObject(remoteInputStreamObj, portNo.intValue());

For this purpose we have developed a POOL which maintains the free port integers. If new thread created in APP1 and having 3 XML files which should be processed at APP2 so APP1 will create a request object and create new three RemoteInputStream objects. At the time of creating object 3 ports integers will be removed from the Pool. Once the request got processed at APP2 side and APP1 got the result, in that case APP1 will unexport the object and return the Ports integer back to POOL.

Here Ports POOL means a Pool of integers which maintains the free ports integers. This pool does not means RemoteInputStream objects pool. We create everytime new instance of RemoteInputStream. In MACHINE1 eventhough 6 JVM's are running we have another 7th JVM which maintains singleton reference of Port POOL. So One POOL instance is shared by all the other APP1 JVMs.

In our case we have high level of concurrency. Other than concurrency, One request of APP1 can have more than one XML. In few cases 25-30 files can be there in single request which needs to be read by MACHINE2 APP2 simultaneously. In this case we need to create 25-30 instances of RemoteInputStream object and put it in Request Object. So here we need more that one port at a time.

In actual production environment it may happen that 50 threads of Service1 can be execute simultaneously and each Request Object can have 2-3 XML files whose inputStream object needs to export. In this case we need more than one port.

Solution

  • we starts getting following exception "java.rmi.NoSuchObjectException: no such object in table"

    This exception has exactly one meaning. You tried to call a remote method via a stub whose corresponding remote object has been unexported.

    Once the request got processed at APP2 side and APP1 got the result, in that case APP1 will unexport the object and return the Ports integer back to POOL.

    And here, as stated above, is the source of the problem. You're unexporting objects. And then some client is calling a method on a stub whose remote object has been unexported, and so you get this exception, exactly as described above.

    You don't need all this complexity. You can use a single port per JVM for all RMI objects. You don't even need to unexport the remote objects, which is what is causing the original problem. Just let them be DGC'd in the normal course of events.

    Throw all this away and return a new RemoteInputStream per request.