Search code examples
javamultithreadingniojava-io

How to read from different sections of a file using multiple threads in java and will it be efficient?


I would need to read a file in efficient way.

I would read the file and each word from the read file would looked into custom dictionary and decide the offset of the word in the file.

I came up with the following solution for this

Producer thread

Reads line by line from file and puts each line to ConcurrentHashMap.

Consumer threads

Reads the line from map and looks up into dictionary for the word.

I am quite sure about implementing the consumer but not sure if using threads to read file would be helpful. Unsure about using java IO or java NIO

Updated sample code for reading from file update to map for producer

public class DocumentManager {


    Map<Location, String> map = null;


    public DocumentManager(Map<Location, String> map) {
        this.map = map;
    }

    public void readFile(String path) throws IOException{
        BufferedReader bufferedReader = new BufferedReader(new FileReader(new File(path)));

        String line = "";

        while((line = bufferedReader.readLine()) != null){
            map.put(new Location(0, 0), line);
            //location is dummy at the moment
        }
  }

}

Solution

  • Using threads is not free, and passing a object between threads via a blocking queue can be surprisingly expensive. However updating a Map, while not free, is much cheaper than passing work between threads.

    Most likely you will spend most of your time reading and parsing the file. If you have a text file it is very hard to do this using multiple threads, and even harder to make it faster than just reading it in one thread.