Search code examples
rustrust-polars

Rust Polars .par_iter() for ChunkedArray<Float64Type>?


In Rust, using Polars, I am writing a custom function to be used within apply/map. This works well:

fn capita(x: Series) -> Result<Series> {
    let y = x
            .utf8()
            .unwrap()
            .par_iter() //ParallelIterator

However, if my Series was of type f64, then this doesn't work:

fn other_fn(x: Series) -> Result<Series> {
    let y = x
            .f64()
            .unwrap()
            .par_iter() // <--- no method par_iter found for reference ChunkedArray<Float64Type>

Is there a possible workaround? Looking at utf8.rs I need something like that but for Float64Chunked type.

Many thanks

EDIT I think this workaround has worked, although turned out to be slower, AND giving an unexpected result. using par_bridge:

pub fn pa_fa(s: &mut [Series])->Result<Series>{
    let u = s[2].f64()?;
    let n = s[1].f64()?;
    let n_iter = n.into_iter();

    let c: Vec<f64> = n_iter.zip(u).par_bridge().map(
    // ignore this line   let c: Vec<f64> = n_iter.zip(u).map( 
        |(n,u)|{
            n.unwrap().powf(1.777)*u.unwrap().sqrt()
        }
    ).collect();
    Ok(Series::new("Ant", c))

Solution

  • The alternative I found was to collect ChunkedArray into a Vector:

    use polars::prelude::*;
    use rayon::prelude::*;
    
    fn round_series_float64(series: Series, decimals: u32) -> PolarsResult<Series> {
    
        let chunkedarray_floatf64 = series
            .f64()?
            .into_iter()
            .collect::<Float64Chunked>();
    
        let vec_opt_f64: Vec<Option<f64>> = chunkedarray_floatf64
            .into_iter()
            .collect();
    
        // No sorting required as rayon collects already sorted!
    
        let new_series: Series = vec_opt_f64
            //.into_iter()
            .into_par_iter() // rayon: parallel iterator
            .map(|opt_f64: Option<f64>|
                opt_f64.map(|float64| round_f64(float64, decimals))
            )
            .collect::<Float64Chunked>()
            .into_series();
    
        /*
        // No use of rayon:
        let new_series: Series = series
            .f64()?
            .into_iter()
            .map(|opt_f64: Option<f64>|
                opt_f64.map(|float64| round_f64(float64, decimals))
            )
            .collect::<Float64Chunked>()
            .into_series();
        */
    
        Ok(new_series)
    }
    

    Such that:

    pub fn round_f64(value: f64, decimals: u32) -> f64 {
        if decimals == 0 {
            value.round()
        } else {
            let multiplier = 10.0_f64.powf(decimals as f64);
            (value * multiplier).round() / multiplier
        }
    }