How to establish relation with respect to higher ranked trait bounds

Question

I am trying to create a high order function which takes in (a function that takes &str and returns iterator of &str). What I am having difficulty is to relate lifetime variables of for<'a> for the function and the return type of that function. It is hard to describe in words, so let's jump right into the code:

use std::fs::File;
use std::io::{BufRead, BufReader, Result};

fn static_dispatcher<'b, I>(
    tokenize: impl for<'a> Fn(&'a str) -> I,
    ifs: BufReader<File>,
) -> Result<()>
where
    I: Iterator<Item = &'b str>,
{
    ifs.lines()
        .map(|line| tokenize(&line.unwrap()).count())
        .for_each(move |n| {
            println!("{n}");
        });
    Ok(())
}

fn main() -> Result<()> {
    let ifs = BufReader::new(File::open("/dev/stdin")?);
    static_dispatcher(|line| line.split_whitespace(), ifs)
    // static_dispatcher(|line| line.split('\t'), ifs)
}

The compiler complains that the lifetime relation of the tokenize's input 'a and output 'b is not specified.

  --> src/main.rs:21:30
   |
21 |     static_dispatcher(|line| line.split_whitespace(), ifs)
   |                        ----- ^^^^^^^^^^^^^^^^^^^^^^^ returning this value requires that `'1` must outlive `'2`
   |                        |   |
   |                        |   return type of closure is SplitWhitespace<'2>
   |                        has type `&'1 str`
   |
   = note: requirement occurs because of the type `SplitWhitespace<'_>`, which makes the generic argument `'_` invariant
   = note: the struct `SplitWhitespace<'a>` is invariant over the parameter `'a`
   = help: see <https://doc.rust-lang.org/nomicon/subtyping.html> for more information about variance

I want to specify 'a = 'b, but I can't because 'a comes from for<'a>, which is not visible for the type I.

I also tried

fn static_dispatcher<'a, I>(
    tokenize: impl Fn(&'a str) -> I,
    ifs: BufReader<File>,
) -> Result<()>
where
    I: Iterator<Item = &'a str>,

but this does not work either b/c the lifetime of tokenize argument must be generic, i.e., must be used with for <'a>.

How can I fix this problem?

Answer 1

You can use a custom trait:

trait IteratorReturningSingleArgFn<Arg>: Fn(Arg) -> Self::Iter {
    type Iter: Iterator<Item = Self::Item>;
    type Item;
}

impl<Arg, F, Iter> IteratorReturningSingleArgFn<Arg> for F
where
    F: Fn(Arg) -> Iter,
    Iter: Iterator
{
    type Iter = Iter;
    type Item = Iter::Item;
}

fn static_dispatcher(
    tokenize: impl for<'a> IteratorReturningSingleArgFn<&'a str, Item = &'a str>,
    ifs: BufReader<File>,
) -> Result<()> {
    // ...
}

This makes inference mad (and I can understand it :) ) and requires you to specify the type in the closure inside static_dispatcher() explicitly:

    ifs.lines()
        .map(|line: Result<String>| tokenize(&line.unwrap()).count())

And in main():

static_dispatcher(|line: &str| line.split_whitespace(), ifs)

But unfortunately it still doesn't work: because of the nasty lifetime inference in closures, the compiler doesn't use HRTB lifetime here and instead try to apply concrete lifetimes. This can be fixed by using the nightly closure_lifetime_binder feature:

#![feature(closure_lifetime_binder)]

static_dispatcher(for<'a> |line: &'a str| -> std::str::SplitWhitespace<'a> { line.split_whitespace() }, ifs)

Or by using a function instead of a closure:

fn tokenize(line: &str) -> impl Iterator<Item = &str> {
    line.split_whitespace()
}
static_dispatcher(tokenize, ifs)