Search code examples
machine-learningstatisticsnlpcomputer-science

Tools for calculating annotators agreement

I am looking for a tool for calculating inter annotators agreement for two annotators in the case of multi-label classification.

I was trying to use ReCal online tool, but it seems that ReCal doesn't support multi-label cllassification.

Are you aware of any tool for multi-label classification?

Solution

  • Look at https://dkpro.github.io/dkpro-statistics/ . However, it is not a 'ready-to-use tool', it is a library and it requires a bit of coding to fit your data into its data model.