Vala `load_contents` error with large files

Question

I'm trying to read the content of a file (should be easy...) and I have no problems until I try to read huge files (more than 2 GB). I get an error that i don't know how to solve and I have not found any alternative way to read the content which works with huge files. This is my code:

File file = File.new_for_path (abs_path);
uint8[]  contents;
try {
    string etag_out;
    file.load_contents (null, out contents, out etag_out);
}catch (Error e){
    error("Error reading from file: %s", e.message);
}

The content should contains all the bytes readed from abs_path. But I got an error saying:

 Error reading from file: Bad address

I don't know why I get the Bad address error. The file exists. Another file with the same name but whose size is only 50 MB works well.

Does anybody have an alternative solution to read huge files in Vala? Or knows how to make this works?

Note: I use the load_content method because I want the bytes (uint8 []), not the string.

Answer 1

As you've noticed, reading large files into memory can be troublesome and very slow. Since you're able to work with the file data in chunks, you can read the file a little bit at a time using a buffer like so:

size_t BUFFER_SIZE = 256;
File file = File.new_for_path (abs_path);
Cancellable cancellable = new Cancellable ();
...

try {
    // Stream the file in chunks rather than loading the entire thing into memory
    FileInputStream file_input_stream = file.read (cancellable);
    ssize_t bytes_read = 0;
    uint8[] buffer = new uint8[BUFFER_SIZE];
    while ((bytes_read = file_input_stream.read (buffer, cancellable)) != 0) {
        // Send the buffer content as needed
    }
} catch (GLib.Error e) {
    critical ("Error reading file: %s", e.message);
}

This will read the file in BUFFER_SIZE chunks that you can then send as needed via SOAP message, etc.