write from multiple threads on same file without locking, Java

Question

I am making a download manager and I want multiple threads downloading different segments of file to write to file at different places at a time. Just for every one's clarification I dont want file to lock because it will kill the purpose of different threads writing at a time. I am using Apache HttpClient library and FileChannel transferFrom(). Current code only downloads the first segment and simply ignores other segments.

Code Explanation: The startDownload method creates a new file and checks if link support partial content, If it does it starts threads for each segment otherwise a single thread will download the whole file.The getFileName is function for extracting file name from URI. The Download method contains the code which actually downloads the file using HttpClient and then writes it using transferFrom.

    public void startDownload() {
    Thread thread = new Thread(() -> {
        try {
            String downloadDirectory = "/home/muhammad/";
            URI uri = new URI("http://94.23.204.158/JDownloader.zip");
            int segments = 2;
            // Create a HttpClient for checking file for segmentation.
            CloseableHttpClient Checkingclient = HttpClients.createDefault();
            // get request for checking size of file.
            HttpGet checkingGet = new HttpGet(uri);
            CloseableHttpResponse checkingResponse = Checkingclient.execute(checkingGet);
            long sizeofFile = checkingResponse.getEntity().getContentLength();
            // Create a new file in downloadDirectory with name extracted from uri.
            File file = new File(downloadDirectory + getFileName(uri));
            if (!file.exists()) {
                file.createNewFile();
            }
            // set range header for checking server support for partial content.
            checkingGet.setHeader("Range", "bytes=" + 0 + "-" + 1);
            checkingResponse = Checkingclient.execute(checkingGet);
            // Check if response code is 206 (partial content response code).
            if (checkingResponse.getStatusLine().getStatusCode() == 206) {
                //find size of each segment.
                final long sizeOfEachSegment = sizeofFile / segments;
                //Download each segment independently.
                for (int i = 0; i < segments; i++) {
                    Download(i * sizeOfEachSegment, (i + 1) * sizeOfEachSegment, sizeOfEachSegment, file, uri);
                }
                // Thread used for last few Bytes and EOF.
                Download(sizeOfEachSegment * segments, sizeofFile, Long.MAX_VALUE, file, uri);
            } else {
                System.err.println("server dont support partial content");
                System.out.println(checkingResponse.getStatusLine().getStatusCode());
                // Download complete file using single thread.
                Download(0, 0, Long.MAX_VALUE, file, uri);
            }
        } catch (IOException | URISyntaxException ex) {
            Logger.getLogger(Downloader.class.getName()).log(Level.SEVERE, null, ex);
        }
    });
    thread.start();
}
public void Download(long start, long end, long sizeOfEachSegment, File file, URI uri) {
    Thread thread = new Thread(() -> {
        try {
            FileChannel fileChannel = new FileOutputStream(file).getChannel();
            CloseableHttpClient client = HttpClients.createDefault();
            HttpGet get = new HttpGet(uri);
            // Range header for defining which segment of file we want to receive.
            if (end != 0) {
                String byteRange = start + "-" + end;
                get.setHeader("Range", "bytes=" + byteRange);
            }
            CloseableHttpResponse response = client.execute(get);
            ReadableByteChannel inputChannel = Channels.newChannel(response.getEntity().getContent());
            fileChannel.transferFrom(inputChannel, start, sizeOfEachSegment);
            response.close();
            client.close();
            fileChannel.close();
        } catch (IOException | IllegalStateException exception) {
            Logger.getLogger(Downloader.class.getName()).log(Level.SEVERE, null, exception);
        }
    });
    thread.start();
}

Some fix to existing code that can make multiple threads to write to same file at same time without waiting will be nice but I am also interested in studying other more efficient techniques if they can do the above task. If in any case writing to a file without waiting is impossible then any other efficient solution is more then welcome. Thanks in advance :)

Answer 1

Writing to the same file from different threads is not going to help you at all - it would probably even dramatically harm throughput.

You should use one thread to write to the file and feed it from a queue.

Something like:

class WriteBlock {
    long offset;
    byte[] data;
}
BlockingQueue<WriteBlock> writeQueue = new LinkedBlockingQueue<>();

Now each downloading thread should read a block from the download, create a WriteBlock and post it into the queue.

Meanwhile the writing thread sucks WriteBlocks out of the queue and writes them as fast as it can.

There may be optimizations to resequence the blocks while in the queue (perhaps with a PriorityBlockingQueue) but do it the simple way first.