library(CausalStress)
library(dplyr)
library(ggplot2)
board_path <- tempfile("causalstress-suite-")
dir.create(board_path)
board <- pins::board_folder(board_path)Suites are named collections of DGP IDs. They make repeated benchmark setup convenient, but membership is not evidence that a DGP is stable or that every aggregate is scientifically meaningful. Inspect registry status and each DGP report before designing a study.
Discover suites
cs_suite_registry() |>
select(suite_id, description)# A tibble: 5 × 2
suite_id description
<chr> <chr>
1 placebo All placebo (sharp-null) DGPs
2 heavytail Heavy-tail signal and placebo heavy-tail DGPs
3 signal Signal-focused DGPs (baseline, QTE heterogeneity, nonlinear heteros…
4 stress Stress-test DGPs (overlap and tilt stressors)
5 all All available synthetic DGPs
cs_get_suite("placebo")[1] "synth_placebo_tau0" "synth_placebo_nonlinear"
[3] "synth_placebo_heavytail" "synth_placebo_tilted"
[5] "synth_placebo_tilted" "synth_placebo_kangschafer"The placebo suite currently contains experimental DGPs. The call below requests that status explicitly and uses a small bootstrap only to make interval plots executable. Increase the number of seeds and bootstrap replicates for scientific work.
placebo_runs <- cs_run_suite(
suite_id = "placebo",
estimator_ids = "lm_att",
n = 150,
seeds = 1:2,
status = "experimental",
bootstrap = TRUE,
B = 20,
board = board,
show_progress = FALSE
)
placebo_tidy <- cs_tidy(placebo_runs)
cs_collect_scores(placebo_runs) |>
count(estimator_id, estimand_target_id, score_status)# A tibble: 1 × 4
estimator_id estimand_target_id score_status n
<chr> <chr> <chr> <int>
1 lm_att att scored 12Visualize ATT evidence
cs_plot_placebo() shows each placebo ATT estimate and interval; color records whether the interval covers the sharp-null target. cs_plot_att_error() shows the finite set of per-run ATT deviations.
cs_plot_placebo(placebo_tidy) +
ggtitle("Placebo ATT intervals")
cs_plot_att_error(placebo_tidy) +
ggtitle("ATT deviations in the example runs")
Interpret the gatekeeper components
cs_summarise_gatekeeper() returns separate target-specific components. The ATT component compares interval coverage with threshold. ATE remains structurally present but unverified pending recalibration. For QST, a run fails when more than 10% of its tau intervals exclude the placebo truth, and an estimator fails when more than 10% of verified runs fail. Missing intervals produce UNVERIFIED, not a pass.
gate <- cs_summarise_gatekeeper(placebo_runs, threshold = 0.90)✖ lm_att: coverage = 0.67 (threshold 0.90)ℹ Culprits (below threshold):- lm_att on synth_placebo_kangschafer: coverage = 0.00- lm_att on synth_placebo_nonlinear: coverage = 0.50- lm_att on synth_placebo_tau0: coverage = 0.50ℹ QST Gatekeeper (10/10 rule):! lm_att: run failure rate = NA (threshold 0.10) [UNVERIFIED]
gate$att_verdict# A tibble: 1 × 4
estimator_id n_verified coverage_rate status
<chr> <int> <dbl> <chr>
1 lm_att 12 0.667 FAIL
gate$att_culprits# A tibble: 3 × 4
dgp_id estimator_id n_verified dgp_coverage
<chr> <chr> <int> <dbl>
1 synth_placebo_kangschafer lm_att 2 0
2 synth_placebo_nonlinear lm_att 2 0.5
3 synth_placebo_tau0 lm_att 2 0.5
gate$ate_verdict# A tibble: 1 × 8
estimand_target_id estimator_id n_verified coverage_rate threshold status
<chr> <chr> <int> <dbl> <dbl> <chr>
1 ate lm_att 0 NA NA UNVERIFIED
# ℹ 2 more variables: policy_status <chr>, registry_consequence <chr>
gate$qst_verdict# A tibble: 1 × 4
estimator_id n_verified run_fail_rate status
<chr> <int> <dbl> <chr>
1 lm_att 0 NA UNVERIFIEDThe two seeds above demonstrate the API; they are far too few to support a scientific gatekeeper decision.
Inspect execution metadata and persisted evidence
placebo_tidy |>
select(
dgp_id, estimator_id, seed,
run_time_dgp, run_time_est, run_time_total
) |>
arrange(desc(run_time_total))# A tibble: 12 × 6
dgp_id estimator_id seed run_time_dgp run_time_est run_time_total
<chr> <chr> <int> <dbl> <dbl> <dbl>
1 synth_placebo_no… lm_att 1 0.00418 0.0322 0.0738
2 synth_placebo_ta… lm_att 1 0.00446 0.0353 0.0650
3 synth_placebo_he… lm_att 2 0.00421 0.0346 0.0626
4 synth_placebo_no… lm_att 2 0.00421 0.0341 0.0616
5 synth_placebo_he… lm_att 1 0.00414 0.0338 0.0609
6 synth_placebo_ta… lm_att 2 0.00438 0.0312 0.0600
7 synth_placebo_ti… lm_att 1 0.00420 0.0323 0.0598
8 synth_placebo_ti… lm_att 2 0.00421 0.0311 0.0590
9 synth_placebo_ti… lm_att 2 0.00404 0.0317 0.0589
10 synth_placebo_ti… lm_att 1 0.00415 0.0308 0.0581
11 synth_placebo_ka… lm_att 1 0.00404 0.0286 0.0556
12 synth_placebo_ka… lm_att 2 0.00394 0.0287 0.0555# A tibble: 10 × 4
dgp_id estimator_id seed fit_fingerprints
<chr> <chr> <int> <list>
1 synth_placebo_heavytail lm_att 1 <chr [1]>
2 synth_placebo_heavytail lm_att 2 <chr [1]>
3 synth_placebo_kangschafer lm_att 1 <chr [1]>
4 synth_placebo_kangschafer lm_att 2 <chr [1]>
5 synth_placebo_nonlinear lm_att 1 <chr [1]>
6 synth_placebo_nonlinear lm_att 2 <chr [1]>
7 synth_placebo_tau0 lm_att 1 <chr [1]>
8 synth_placebo_tau0 lm_att 2 <chr [1]>
9 synth_placebo_tilted lm_att 1 <chr [1]>
10 synth_placebo_tilted lm_att 2 <chr [1]> Timing fields are provenance, not scientific identities. Persisted results are written only because this example supplied board; skip_existing = TRUE is a separate opt-in for strict fingerprint-checked resume.