Search code examples
juliajulia-dataframe

Row wise median for julia dataframes

I want to compute the median values of all rows in a dataframe. Some columns contain NaN values. Some rows even have all NaN values. The problem with median is

  1. if there's any NaN values in a vector it returns NaN. In this case I would like to skip NaNs (like in Pandas).
  2. it is undefined for empty vectors (throws an error). In this case I want to return a NaN (like in Pandas)

I came up with the following solution:

df = DataFrame(rand(100, 10), :auto) 
df[1, :x3] = NaN
df[20, [:x3, :x6]] .= NaN
df[5, :] .= NaN

safemedian(y) = all(isnan.(y)) ? NaN : median(filter(!isnan, y))
x = select(df, AsTable(:) => ByRow(safemedian∘collect) => "median")

This works however it's rather slow.

Question 1) Is there a way to speed this up?

I think the collect method is causing the sluggish performance. But I need to use the collect method otherwise I get an error:

safemedian(y) = all(isnan.(y)) ? NaN : median(filter(!isnan, y))
x = select(df, AsTable(:) => ByRow(safemedian) => "median")

# results in ArgumentError: broadcasting over dictionaries and `NamedTuple`s is reserved

This is because AsTable(:) passes each row a named tuple.

Question 2) Is there a way to pass rows as vectors instead?

This way I could pass the row to any function that expects a vector (for example the nanmedian function from the NaNStatistics.jl Package). Note I would not need to use the collect method if the AsVector(:) method was implemented (see [here]). Unfortunately it didn't get the go ahead and I'm not sure what the alternative is.

Question 3) This one is more philisophical. Coming from Python/Pandas some operations in Julia are hard to figure out. Pandas for example handles NaNs seemlessly (for better or worse). In Julia I artificially replace the missing values in my dataframe using mapcols!(x -> coalesce.(x, NaN), df). This is because many package functions (and functions I've written) are implemented for AbstractArray{T} where {T<:Real} and not AbstractArray{Union{T, Missing}} where {T<:Real} (ie. they don't propagate missings). But since there is no skipnan yet there is a skipmissing function in Julia, I'm thinking I've got it all wrong. Is the idiomatic way to keep missing values in Julia and handle them where appropriate? Or is it ok to use NaN's (and keep the type fixed as say Float64)?

Solution

  • Try:

    filter.(!isnan,eachrow(Matrix(df))) .|>
  v->isempty(v) ? NaN : median(v)

    Each library has idiosyncracies which melt away with practice. So coming from paNdas you are familiar with pandas and it feels natural. After a while, it is entirely possible you would find things that are natural in Julia to be awkward in panads.

    For more perspective, ask this question on Discourse, which has very recently had a thread directly on these issues.