rust: Single trait to implement read/write/list files from local and remote sources

Question

I'm trying to create a web service that can stream files from various sources. I want to declare a Source trait that each source must implement with methods for listing, reading and eventually writing files but I have a hard time finding the right pattern.

In the code below I get problems with Source not being "object safe" due to the generic parameter R.

What would be a good pattern to use to have multiple source types some local, some remote/network ones implement the same Source trait to read/write/list files?

use std::collections::HashMap;

use anyhow::{anyhow, Result};
use async_std::path::PathBuf;
use async_trait::async_trait;
use tokio::{io::{BufReader, AsyncRead}, fs::File};

#[async_trait]
pub trait Source {
    // async fn list(&self, path: PathBuf, path_prefix: PathBuf) -> Result<Vec<FileMeta>>;
    async fn reader<R: AsyncRead>(&self, path: PathBuf) -> Result<BufReader<R>>;
    // async fn writer<W: AsyncWrite>(&self, path: PathBuf) -> Result<BufWriter<W>>;
}

#[derive(Clone)]
pub struct Local {
    root: PathBuf
}

impl Local {
    pub async fn new(root: PathBuf) -> Result<Self> {
        Ok(Self { root: root.canonicalize().await? })
    }

    fn root(&self) -> PathBuf {
        self.root.clone()
    }

    async fn resolve(&self, path: PathBuf) -> Result<PathBuf> {
        let path = path.strip_prefix("/").unwrap_or(&path);

        let mut result = self.root();
        result.push(path);

        result.canonicalize().await?;

        if !result.starts_with(self.root()) {
            return Err(anyhow!("Requested path is outside source root"));
        }

        Ok(result)
    }
}

#[async_trait]
impl Source for Local {
    async fn reader<R: AsyncRead>(&self, path: PathBuf) -> Result<BufReader<R>> {
        let file = File::open(self.resolve(path).await?).await?;
        let reader = BufReader::new(file);
        Ok(reader)
    }
    
    /*
    async fn writer<W: AsyncWrite>(&self, path: PathBuf) -> Result<BufWriter<W>> {
        todo!()
    }
    */
}

/*
The idea is to allow other file sources, HTTP, SSH, S3 ect. as long as they implement 
the Source trait 

#[derive(Clone)]
pub struct RemoteHTTP {
    server_url: String
}

#[async_trait]
impl Source for RemoteHTTP {
    async fn reader<R: AsyncRead>(&self, path: PathBuf) -> Result<BufReader<R>> {
        todo!()
    }
    
    async fn writer<W: AsyncWrite>(&self, path: PathBuf) -> Result<BufWriter<W>> {
        todo!()
    }
}
*/

pub struct Config {
    sources: HashMap<String, Box<dyn Source>>,
}

impl Config {
    pub async fn load() -> Result<Self> {
        let local = Local::new("/tmp/".into()).await?;
        // let remote = RemoteHTTP::new("https://example.org".into());

        let mut sources: HashMap<String, Box<dyn Source>> = HashMap::new();
        sources.insert("local".into(), Box::new(local));
        // sources.insert("remote".into(), Box::new(remote));

        Ok(Self { sources })
    }

    pub fn sources(&self) -> HashMap<String, Box<dyn Source>> {
        self.sources.clone()
    }
}

#[tokio::main]
async fn main() -> Result<()> {
    let config = Config::load().await;

    // Store various sources into a config map
    let local = Local::new("/tmp".into()).await?;
    config.sources.insert("local".into(), Box::new(local));



    // Create a read stream from one of hhe sources
    if let Some(source) = config.sources.get("local".into()) {
        let reader = source.reader("a-file".into()).await?;
        // stream data with an actix HTTP service using: HttpResponse::Ok().streaming(reader)
    }

    Ok(())
}

Answer 1

You cannot use generics in methods of traits that are intended to be used dynamically. But even if you could, the signature of Source::reader() wouldn't work because it'd allow the caller to choose which reader type to return, whereas the implementation would surely want to return a concrete type. Thus every concrete implementation would fail to compile with "returned <some concrete type>, generic type R expected". The correct return type would be something like BufReader<impl AsyncRead>, but that wouldn't work because impl in return position is not yet allowed in traits, and because you need your trait to be object-safe.

Instead, Source::reader() should return a boxed AsyncRead. For example:

#[async_trait]
pub trait Source {
    async fn reader(&self, path: PathBuf) -> Result<BufReader<Box<dyn AsyncRead + Unpin>>>;
}

The implementation then looks like this:

#[async_trait]
impl Source for Local {
    async fn reader(&self, path: PathBuf) -> Result<BufReader<Box<dyn AsyncRead + Unpin>>> {
        let file = File::open(self.resolve(path).await?).await?;
        let reader = BufReader::new(Box::new(file) as _);
        Ok(reader)
    }
}

Your example fixed up to compile on the playground.