i have a huge file (2GB) which contains employee numbers only. i have to read this file and get the employee number and call database to get the salary of the employee and then write this in another file with employee name and salary as its row.
Now here is the problem, By direct reading this huge file by simple nio in java my STS goes out of memory or it takes 4-5 hrs to complete the entire read-fetch-write process.
So i thought to use the Java concurrency to rescue me.
for that i have a class EmployeeDetails which implements Runnable, in its main method , i have Create a thread pool executor with core size and maximum size as 2, specify a time out and pass the blocking queue.
BlockingQueue workQueue = new LinkedBlockingQueue();
In ThreadPoolExecutor along with corePoolSize=2 , maximumPoolSize =20 , keepAliveTime = Long.MAX_VALUE, workQueue is been passed.
then created inputFile File object to getting the input file from path and outputFile Object where i need to write the data.
then i created two object of EmployeeDetails ,
EmployeeDetails readFile = new EmployeeDetails(inputFile);
EmployeeDetails writeFile = new EmployeeDetails (outputFile);
then ThreadPoolExecutor. submit(readFile) and ThreadPoolExecutor submit (writeFile ) called.
In class EmployeeDetails , i have created run method,
run method open the database connection,
then BufferedReader get the file,
while loop is run untill br.readLine() !=null
then for each employee db is called to get the salary as employee are in different shard (MySQL) so as its salary.
once the salary is retrieved i put it in a list and as soon as the list size it 10000. i need to wait a thread and call the wrie thread to write it into file and flush the list to get another 10000 records and thats the problem.
Please suggest a way to read 10000 records from a file , call the db to get details and then populating a list and once he list it 10000 record calling write thread to pic the list and write into another file.
I don't think it gives significant performance boost to just split reading and writing of the files.
Rather consider splitting the actual work - fetching from db - into threads.
Create ExecutorService at the beginning and from original thread just read the lines one by one and wrapped into Callable doing the fetch pass them to executor service. Pass the resulting Future into LinkedBlockingQueue which will be consumed by writer thread. That will wait for incoming futures and write the results (this is to keep the order, otherwise you could write directly from the Callable).