Bypass borrow checker in file chain implementation

Question

I am trying to implement a simple chain of files. Unfortunately, my implementation for BufRead::fill_buf produces a borrow checker error.

impl BufRead for FileChain {
    fn fill_buf(&mut self) -> io::Result<&[u8]> {
        loop {
            let buf = self.stream.fill_buf()?;
            if !buf.is_empty() {
                return Ok(buf);
            } if let Some(filename) = self.future_files.pop() {
                self.stream = BufReader::new(File::open(&filename)?);
            } else {
                return Ok(&[])
            }
        }
    }
}

error[E0506]: cannot assign to `self.stream` because it is borrowed
40 |     fn fill_buf(&mut self) -> io::Result<&[u8]> {
   |                 - let's call the lifetime of this reference `'1`
41 |         loop {
42 |             let buf = self.stream.fill_buf()?;
   |                       ---------------------- `self.stream` is borrowed here
43 |             if !buf.is_empty() {
44 |                 return Ok(buf);
   |                        ------- returning this value requires that `self.stream` is borrowed for `'1`
45 |             } if let Some(filename) = self.future_files.pop() {
46 |                 self.stream = BufReader::new(File::open(&filename)?);
   |                 ^^^^^^^^^^^ `self.stream` is assigned to here but it was already borrowed

You can find the full class definition here (56 lines).

I think this is similar to already known borrow checker issue, described here, here and here. Basically, the checker thinks that buf reference blocks stream from changing. However, return buf and stream update cannot happen together.

Existing solutions (for example from here) cannot be applied here as I don't have a HashMap/Vector, and I am limited by BufRead API. I also cannot use Polonius as I can only use stable features.

My question is: is it possible to modify this code without sacrificing speed using safe code? And if this is not possible, is there a way to override borrow checker using unsafe code?

Answer 1

is there a way to override borrow checker using unsafe code?

While using unsafe doesn't turn off the borrow checker, it does allow you to circumvent some of its safeguards. For example, this compiles and is, I believe, sound:

fn fill_buf(&mut self) -> io::Result<&[u8]> {
    loop {
        // unsafe: we either return a reference into self (which prevents
        // the caller from mutating self until dropping that reference),
        // or we proceed to mutate self. This is safe under borrowing
        // rules, but fails to compile due to limitations of the current
        // borrow checker. Here we use `transmute()` to temporarily
        // decouple the reference from self (it is reassigned the lifetime
        // of self on returning). Note that the safe code without
        // `transmute()` compiles under Polonius.
        unsafe {
            let buf = std::mem::transmute::<_, &'static [u8]>(self.stream.fill_buf()?);
            if !buf.is_empty() {
                return Ok(buf);
            }
        }
        if let Some(filename) = self.future_files.pop() {
            self.stream = BufReader::new(File::open(&filename)?);
        } else {
            return Ok(&[]);
        }
    }
}

When using unsafe, you're on your own to prove that the code is actually sound - the above code documents the reasoning in a comment next to unsafe, which is considered best practice. Keep in mind that unsafe enables you to cheat much further than what would be safe. For example, modifying the signature to return the incorrect Result<&'static [u8]> still compiles, although it makes the function patently unsound. This is the reason such use of unsafe is best avoided in situations where performance is not paramount.