I have a dataset that looks like the following but with much more rows and groups:
df2 <- data.frame(
"group" = c(1, 1, 1, 1, 2, 2, 2, 3, 3, 3, 3, 4, 4, 4, 4, 5, 5, 5, 5, 5),
"R1" = c(1, 0, 0, 1, 0, 1, 1, 1, 0, 1, 0, 1, 1, 1, 1, 1, 0, 0, 0, 0),
"R2" = c(1, 1, 0, 1, 0, 0, 1, 1, 0, 1, 0, 0, 1, 0, 1, 1, 0, 1, 0, 0),
"R3" = c(1, 1, 1, 1, 0, 1, 1, 1, 0, 1, 0, 1, 1, 0, 1, 1, 0, 0, 0, 0))
where R1, R2, and R3 are different raters. I want to calculate the inter-rater reliability (inter-coder reliability) within each group. How could I loop over the groups to do that?
I found a similar questions here: Inter-rater reliability per category but there is no answer. I appreciate any help even it is only about the looping over the groups without the calculation of the inter-rater reliability.
I discovered the excellent tidycomm package recently. Calculating icr on groups is no implemented yet but it works nicely with group_map.
library(tidyverse)
library(tidycomm)
df2 %>%
# tidycomm expects tidy data so we have to do some reshaping first
mutate(post_id = row_number()) %>%
pivot_longer(R1:R3, names_to = "coder_id", values_to = "code") %>%
# first group by, then apply the function once per group
group_by(group) %>%
group_map(.f = function(.x, .y) {
out <- test_icr(.x, unit_var = post_id, coder_var = coder_id, code)
add_column(out, group = .y$group, .before = 1L)
}) %>%
bind_rows()
#> # A tibble: 5 × 9
#> group Variable n_Units n_Coders n_Categories Level Agreement Holstis_CR
#> <dbl> <chr> <int> <int> <int> <chr> <dbl> <dbl>
#> 1 1 code 4 3 2 nominal 0.5 0.667
#> 2 2 code 3 3 2 nominal 0.667 0.778
#> 3 3 code 4 3 2 nominal 1 1
#> 4 4 code 4 3 2 nominal 0.5 0.667
#> 5 5 code 5 3 2 nominal 0.8 0.867
#> # … with 1 more variable: Krippendorffs_Alpha <dbl>
Created on 2022-03-23 by the reprex package (v2.0.1)