I have been working on a project which is about data copy from one db to another. Written in java6. And working on distributed machines. We have 9 server machines. One is the main node which i have my main control module working on which gets copy requests and distributes the job to other 8 machines.
It was started to be coded with JMS to distribute tasks to that 8 machine before.There was an Apache Active MQ server working on another computer. But i saw it was not much appropriate and the machines need to be more tightly coupled as it was causing some code overhead and it was necessary to return a response message for all the messages sent to slave machines. I decided to change the interconnection between main node and other 8 slave machines and coded that with RMI.
I coded an RMI server for slave machines and client for main node machine. Then created threads on main node to trigger distributed-tasks at slave machines.
The thing is performance decreased drastically. Normally i was able to copy about 6GB of data in about 6 mins from one specific db to another. Now it takes more than 1 and a half hour to copy 9GB of data. It used to consume lots of CPU when the task was executing on slave machines. I was observing CPU usage ratio more than 90%. Now it never uses more than 15%.
What i need is to understand what is causing this performance decrease. What should i do ? Should i use a troubleshooting tool ?
Edit------------
Ok i create a slave module instance on my laptop only, and sent 16 tasks to work on it and profiled CPU using jvisualvm.The results are in the pic CPU Profiler result.
The control methods in the package (e.g. failTaskIfAbort(), performSanityCheck(), etc) were also present when i was using JMS for machine communication. It makes me think, RMI threads are somehow low priority or something.
I also uploaded the nps file exported from jvisualvm. You can get it here: exported profile result
From the comments,
I suspect its something unrelated. It appears your bottle neck is how long it takes to get its data from your database. If you are downloading to 8 clients at once, you might only be getting 1/8th of the database's bandwidth per client. I would see how it performs when you have only one client (i.e. is the CPU utilisation much higher)
BTW: You may find the data is limited by the network adapter of your server, or the speed it can retrieve data from the database. If its the former, you may have a spare network adapter on your server you can use to increase your bandwidth.
