The posterior of abc_fit() for an observation is a weighted sample of the reference table,
with weights from each random forest. For each observation and parameter this gives the
effective sample size of the weights (Kish) and the Pareto k-hat of their upper tail
(Vehtari et al. 2024), computed by posterior::pareto_khat() on the positive weights, and
for each observation the statistics that fall outside the central coverage of the
reference table.
Value
A data frame with observation, parameter, n_positive (simulations with positive
weight), ess, khat, khat_threshold, n_outside (statistics of the observation outside
the central coverage of the table) and outside (their names).
Details
The weights of random forests are not importance ratios, so the thresholds of k-hat, derived
for importance sampling, do not carry over. In our checks (data-raw/abc_diagnose.R), k-hat
and the effective sample size did not predict which estimates were wrong within the model,
while data from another model had about half the effective sample size and more statistics
outside the table. Read them as warnings that the observation is unlike the simulations, not
as measures of the precision of an estimate. An observation outside the range of the
simulated statistics is the clearer warning, since the forests then extrapolate from the
edge of the table without saying so.
References
Vehtari, A., Simpson, D., Gelman, A., Yao, Y., & Gabry, J. (2024). Pareto smoothed importance sampling. Journal of Machine Learning Research, 25(72), 1–58.
See also
Other inference:
abc_choose(),
abc_cv(),
abc_fit(),
abc_ppc(),
abc_recovery(),
abc_table(),
fold_summarise(),
observe_detection(),
spatial_folds()
Examples
set.seed(1)
table <- data.frame(a = runif(500))
table$s1 <- table$a + rnorm(500, 0, 0.05)
fit <- abc_fit(table, "a", "s1", ntree = 100)
abc_diagnose(fit, data.frame(s1 = c(0.5, 3)))
#> Warning: Number of tail draws cannot be less than 5. Changing to 5.
#> Warning: Number of tail draws cannot be less than 5. Changing to 5.
#> observation parameter n_positive ess khat khat_threshold n_outside
#> 1 1 a 13 5.532480 NA NA 0
#> 2 2 a 11 4.602561 NA NA 1
#> outside
#> 1
#> 2 s1
