Measures, for each species, the agreement between reviewers on the clips reviewed more than once: the proportion of agreement and kappa, which discounts the agreement expected by chance. When a species has exactly two reviewers, this is Cohen's kappa, with the marginal frequencies of each reviewer (Cohen 1960, doi:10.1177/001316446002000104 ); with more reviewers, or different pairs of reviewers on different clips, Fleiss' kappa (Fleiss 1971, doi:10.1037/h0031619 ). Only positive and negative verdicts count.
Arguments
- reviews
One table of reviewed clips (as returned by
classify_review()) with areviewercolumn, or a named list of such tables, named by reviewer.