Read large f64 binary file into array

I'm looking for help/examples on how to read a relatively large (>12M) binary file of double precision numbers into a rust array. I have metadata on the number of f64 values in the file.

I've read on this and seen the byteorder crate but did not find the documentation/examples particularly helpful.

This is not something that needs to be BufRead, since that likely won't help performance.

Thank you!

Solution

The easiest way to do it is to read 8 bytes and convert it to f64 using one of the f64::from_byte-order_bytes() methods:

These methods are used like that:

let mut buffer = [0u8; 8]; // the buffer can be reused!
reader.read_exact(&mut buffer) ?;
let float = f64::from_be_bytes(buffer);

So you can either read the file 8 bytes at a time or on some larger chunks:

fn main() -> Result<(), Box<dyn Error>> {
    let file = File::open("./path/to/file")?;
    let mut reader = BufReader::new(file);

    let mut buffer = [0u8; 8];
    loop {
        if let Err(e) = reader.read_exact(&mut buffer) {
            // if you know how many bytes are expected, then it's better not to rely on `UnexpectedEof`!
            if e.kind() == ErrorKind::UnexpectedEof {
                // nothing more to read
                break;
            }

            return Err(e.into());
        }

        // or use `from_le_bytes()` depending on the byte-order
        let float = f64::from_be_bytes(buffer);
        //do something with the f64
        println!("{}", float);

    }

    Ok(())
}

If you don't mind adding an additional dependency to your project, then you can also use the ByteOrder crate which has convenience methods to read whole slices:

use byteorder::{ByteOrder, LittleEndian};

let mut bytes = [0; 32]; // the buffer you've read the file into
let mut numbers_got = [0.0; 4];

LittleEndian::read_f64_into(&bytes, &mut numbers_got);
assert_eq!(numbers_given, numbers_got)