Compare calibration models by cross-validation

Fits each candidate model with k-fold cross-validation and scores the predicted probabilities on the held-out folds. Folds are stratified within site x species by default. With fold_group, recordings or nights are kept intact across species; see calibration_folds(). Models with random effects need the 'lme4' package.

Usage

compare_calibration_models(
  validations,
  models = NULL,
  k_folds = 5,
  seed = 2026,
  fold_group = NULL
)

Arguments

validations

Table returned by read_validations().

models

Named list of model formulas. NULL chooses the family from the observed species and sites: site-effect models for one species, the original family for multiple species, and no site random effects when only one site is observed. Supply an explicit list to reproduce a particular candidate set.

k_folds

Number of folds.

seed

Seed of the fold assignment. The global random number generator is left as it was.

fold_group

Optional character vector of grouping columns. Do not include species, verdicts or confidence scores: these would split recordings or nights by species or by outcome. Group IDs must be nonmissing and nonempty. NULL keeps the original clip folds.

Value

A tibble with model, formula, auc_mean, auc_sd, n_failed (folds whose fit failed, left out of the means), logloss_mean, logloss_sd, brier_mean, brier_sd, cal_intercept and cal_slope (means over the folds), prop_singular (share of the fold fits that were singular) and complexity (position of the model in models), ordered from the highest mean AUC. Failed folds also raise a warning. A test fold with only one verdict has missing AUC but contributes log-loss and Brier; it is not a failed fit.

Details

The AUC measures discrimination (whether true detections get higher probabilities than false ones). The log-loss and the Brier score are proper scoring rules: they reward probabilities close to the observed outcomes, which is what matters when the probabilities are compared with p_min (Gneiting & Raftery 2007, doi:10.1198/016214506000001437 ). The calibration intercept and slope come from a logistic regression of the verdicts on the logit of the predicted probabilities in each test fold: a well calibrated model has intercept 0 and slope 1; a slope below 1 means probabilities too extreme (overfitting).