I am trying to benchmark the performance of functions using BenchmarkTools as in the example below. My goal is to obtain the outputs of @benchmark as a DataFrame.
In this example, I am benchmarking the performance of the following two functions:
"""Example function A: recodes negative values to 0"""
function negative_to_zero_a!(x::Array{<:Real,1})
for (i, v) in enumerate(x)
if v < 0
x[i] = zero(x[i]) # uses 'zero()'
end
end
end
"""Example function B: recodes negative values to 0"""
function negative_to_zero_b!(x::Array{<:Real,1})
for (i, v) in enumerate(x)
if v < 0
x[i] = 0 # does not use 'zero()'
end
end
end
Which are meant to mutate the following vectors:
int_a = [1, -2, 3, -4]
float_a = [1.0, -2.0, 3.0, -4.0]
int_b = copy(int_a)
float_b = copy(float_a)
I then produce the performance benchmarks using BenchmarkTools.
using BenchmarkTools
int_a_benchmark = @benchmark negative_to_zero_a!(int_a)
int_b_benchmark = @benchmark negative_to_zero_b!(int_b)
float_a_benchmark = @benchmark negative_to_zero_a!(float_a)
float_b_benchmark = @benchmark negative_to_zero_b!(float_b)
I would now like to retrieve the elements of each of the four BenchmarkTools.Trial objects into a DataFrame similar to the one below. In that DataFrame, each row contains the results of a given BenchmarkTools.Trial object. E.g.
DataFrame("id" => ["int_a_benchmark", "int_b_benchmark", "float_a_benchmark", "float_b_benchmark"],
"minimum" => [15.1516, 15.631, 14.615, 14.271],
"median" => [15.916, 15.731, 15.916, 15.879],
"maximum" => [149.15, 104.108, 63.363, 116.181],
"allocations" => [0, 0, 0, 0],
"memory_bytes" => [0, 0, 0, 0])
4×6 DataFrame
Row │ id minimum median maximum allocations memory_estimate
│ String Float64 Float64 Float64 Int64 Int64
─────┼────────────────────────────────────────────────────────────────────────────
1 │ int_a_benchmark 15.1516 15.916 149.15 0 0
2 │ int_b_benchmark 15.631 15.731 104.108 0 0
3 │ float_a_benchmark 14.615 15.916 63.363 0 0
4 │ float_b_benchmark 14.271 15.879 116.181 0 0
How can I retrieve the results of the benchmarks into a DataFrame like this one?
As usual with Julia, there are multiple ways to do what you want. I present here maybe not the simplest way, but the one, which hopefully shows interesting approach, which allows for some generalizations.
But before we start, small notice: your benchmarks are not quite correct, since your functions mutate argument. For proper benchmarking you should copy your data before each run and also do it every time you execute function. You can find more information here: https://juliaci.github.io/BenchmarkTools.jl/dev/manual/#Setup-and-teardown-phases
So, for now, we assume that you prepared benchmarks like this
int_a_benchmark = @benchmark negative_to_zero_a!(a) setup=(a = copy($int_a)) evals=1
int_b_benchmark = @benchmark negative_to_zero_b!(b) setup=(b = copy($int_b)) evals=1
float_a_benchmark = @benchmark negative_to_zero_a!(a) setup=(a = copy($float_a)) evals=1
float_b_benchmark = @benchmark negative_to_zero_b!(b) setup=(b = copy($float_b)) evals=1
Main idea is the following. If we can represent our benchmark data as a DataFrame, then we can combine them together as a single large DataFrame and do all necessary calculations.
Of course one can do it in super easy way, just by making command
df = DataFrame(times = int_a_benchmark.times, gctimes = int_a_benchmark.gctimes)
df.memory .= int_a_benchmark.memory
df.allocs .= int_a_benchmark.allocs
but this is too boring and too verbatim (but simple and should be done 99% of time). It would be nice, if we can just do DataFrame(int_a_benchmark) and get the result immediately.
As it turns out, it is possible, because DataFrames supports Tables.jl interface for working with table-like data. You can read details in the manual of Tables.jl, but generally you need to define some meaningful things, like names of columns, and column accessors and package will do everything else. I show the results here, without further explanations.
using Tables
Tables.istable(::Type{<:BenchmarkTools.Trial}) = true
Tables.columnaccess(::Type{<:BenchmarkTools.Trial}) = true
Tables.columns(m::BenchmarkTools.Trial) = m
Tables.columnnames(m::BenchmarkTools.Trial) = [:times, :gctimes, :memory, :allocs]
Tables.schema(m::BenchmarkTools.Trial) = Tables.Schema(Tables.columnnames(m), (Float64, Float64, Int, Int))
function Tables.getcolumn(m::BenchmarkTools.Trial, i::Int)
i == 1 && return m.times
i == 2 && return m.gctimes
i == 3 && return fill(m.memory, length(m.times))
return fill(m.allocs, length(m.times))
end
Tables.getcolumn(m::BenchmarkTools.Trial, nm::Symbol) = Tables.getcolumn(m, nm == :times ? 1 : nm == :gctimes ? 2 : nm == :memory ? 3 : 4)
and we can see that it really works (almost magically)
julia> DataFrame(int_a_benchmark)
10000×4 DataFrame
Row │ times gctimes memory allocs
│ Float64 Float64 Int64 Int64
───────┼──────────────────────────────────
1 │ 309.0 0.0 0 0
2 │ 38.0 0.0 0 0
3 │ 25.0 0.0 0 0
4 │ 37.0 0.0 0 0
⋮ │ ⋮ ⋮ ⋮ ⋮
Next step is combining all dataframes in a single dataframe. We should make following steps:
name with the name of relevant benchmarkvcat function)Of course, you can do all of this steps one by one for each dataframe, but it's too long (and boring, yes). Instead, we can use amazing mapreduce function and so called Do-Block syntax. map part will prepare necessary dataframes and reduce will combine them together
df_benchmark = mapreduce(vcat, zip([int_a_benchmark, int_b_benchmark, float_a_benchmark, float_b_benchmark],
["int_a_benchmark", "int_b_benchmark", "float_a_benchmark", "float_b_benchmark"])) do (x, y)
df = DataFrame(x)
df.name .= y
df
end
And now for the final part. We have nice, large DataFrame, which we want to aggregate. For this we can use Split-Apply-Combine strategy of DataFrames
julia> combine(groupby(df_benchmark, :name),
:times => minimum => :minimum,
:times => median => :median,
:times => maximum => :maximum,
:allocs => first => :allocations,
:memory => first => :memory_estimate)
4×6 DataFrame
Row │ name minimum median maximum allocations memory_estimate
│ String Float64 Float64 Float64 Int64 Int64
─────┼────────────────────────────────────────────────────────────────────────────
1 │ int_a_benchmark 22.0 24.0 3252.0 0 0
2 │ int_b_benchmark 20.0 23.0 489.0 0 0
3 │ float_a_benchmark 21.0 23.0 134.0 0 0
4 │ float_b_benchmark 21.0 23.0 129.0 0 0
As a bonus, last calulcation can look even better with the help of Chain.jl package:
using Chain
@chain df_benchmark begin
groupby(:name)
combine(:times => minimum => :minimum,
:times => median => :median,
:times => maximum => :maximum,
:allocs => first => :allocations,
:memory => first => :memory_estimate)
end