Skip to contents

Experiment Store

Source: vignettes/experiment-store.qmd

Every committed run is stored with its evidence. By the end of this article you can find your runs again, label and compare them, see what produced each one, and reopen them in a later session. For snapshot creation and data-input boundaries, read vignette("data-input-and-snapshots", package = "ledgr").

Running this yourself

This article is evaluated when rendered. It writes to temporary DuckDB stores so package builds and local previews do not leave project artifacts behind. In real work, use a project-local path such as artifacts/ledgr_store.duckdb.

Pre-CRAN compatibility

ledgr is pre-CRAN. Store schemas, config hashes, provenance formats, and experimental APIs may change before the first CRAN release. Treat stores created with pre-CRAN ledgr as research artifacts for the version that produced them, and expect to rerun experiments after upgrading.

The examples use dplyr for data preparation and compact display. It is a suggested package used by the vignettes, not part of the experiment-store contract.

library(ledgr)
library(dplyr)
data("ledgr_demo_bars", package = "ledgr")

Temporary Snapshot Setup

This article uses a small snapshot in a temporary store from ledgr_temp_store(), so the store examples are self-contained. A real project passes a persistent path, so its runs outlive the R session. The full snapshot lifecycle is covered in vignette("data-input-and-snapshots", package = "ledgr").

db_path <- ledgr_temp_store(file.path(tempdir(), "ledgr_store_demo.duckdb"))

bars <- ledgr_demo_bars |>
  filter(
    instrument_id %in% c("DEMO_01", "DEMO_02"),
    between(
      ts_utc,
      ledgr_utc("2019-01-01"),
      ledgr_utc("2019-06-30")
    )
  )

snapshot <- ledgr_snapshot_from_df(
  bars,
  db_path = db_path,
  snapshot_id = "store_demo_snapshot"
)

After snapshot creation, store operations take snapshot, not db_path. In a new R session, recover the handle with ledgr_snapshot_open(db_path, snapshot_id).

Record Two Variants For Comparison 

features <- list(ledgr_ind_sma(20))

trend_strategy <- function(ctx, params) {
  above <- ctx$vec$close > ledgr_signal_feature(ctx, "sma_20")
  ctx |>
    ledgr_selection(where = above, missing = "exclude") |>
    ledgr_target_quantity(ctx, params$qty)
}

exp <- ledgr_experiment(
  snapshot = snapshot,
  strategy = trend_strategy,
  features = features,
  opening = ledgr_opening(cash = 10000),
  cost_model = ledgr_cost_zero()
)

bt_small <- exp |>
  ledgr_run(params = list(qty = 5), run_id = "trend_qty_5")

bt_large <- exp |>
  ledgr_run(params = list(qty = 15), run_id = "trend_qty_15")

Discover Runs

ledgr_run_list() is the store discovery view.

ledgr_run_list(snapshot)
# ledgr run list
# A tibble: 2 x 11
  run_id       label tags  status final_equity total_return complete_performance
  <chr>        <chr> <chr> <chr>         <dbl> <chr>        <lgl>
1 trend_qty_5  <NA>  <NA>  DONE         10042. +0.4%        NA
2 trend_qty_15 <NA>  <NA>  DONE         10125. +1.3%        NA
  achieved_end_utc valuation_state execution_mode reproducibility_level
  <dttm>           <chr>           <chr>          <chr>
1 NA               <NA>            audit_log      tier_1
2 NA               <NA>            audit_log      tier_1

# i 2 runs attempted: 2 done, 0 incomplete, 0 failed.
# i INCOMPLETE metrics describe the achieved prefix only.
# i Full identity and telemetry columns remain available on this tibble.
# i Inspect one run with ledgr_run_info(snapshot, run_id).

Both runs finished with status DONE. complete_performance and achieved_end_utc are NA because these runs had no point-in-time rules that could stop them early, and the footer’s note about INCOMPLETE metrics applies only to runs whose status says so; neither run here does.

Use labels and tags for mutable human-facing organization.

snapshot <- snapshot |>
  ledgr_run_label("trend_qty_5", "Baseline quantity") |>
  ledgr_run_tag("trend_qty_5", c("baseline", "trend")) |>
  ledgr_run_tag("trend_qty_15", c("trend", "larger-size"))

ledgr_run_list(snapshot)
# ledgr run list
# A tibble: 2 x 11
  run_id       label             tags               status final_equity total_return
  <chr>        <chr>             <chr>              <chr>         <dbl> <chr>
1 trend_qty_5  Baseline quantity baseline, trend    DONE         10042. +0.4%
2 trend_qty_15 <NA>              larger-size, trend DONE         10125. +1.3%
  complete_performance achieved_end_utc valuation_state execution_mode
  <lgl>                <dttm>           <chr>           <chr>
1 NA                   NA               <NA>            audit_log
2 NA                   NA               <NA>            audit_log
  reproducibility_level
  <chr>
1 tier_1
2 tier_1

# i 2 runs attempted: 2 done, 0 incomplete, 0 failed.
# i INCOMPLETE metrics describe the achieved prefix only.
# i Full identity and telemetry columns remain available on this tibble.
# i Inspect one run with ledgr_run_info(snapshot, run_id).

Tags and labels do not alter snapshot hashes, strategy hashes, parameter hashes, config hashes, or result artifacts.

The returned objects are still tibbles. When you need a custom view, convert to a tibble and select the columns you want.

ledgr_run_list(snapshot) |>
  as_tibble() |>
  select(run_id, label, tags, status, final_equity, execution_mode)
# A tibble: 2 x 6
  run_id       label             tags               status final_equity execution_mode
  <chr>        <chr>             <chr>              <chr>         <dbl> <chr>
1 trend_qty_5  Baseline quantity baseline, trend    DONE         10042. audit_log
2 trend_qty_15 <NA>              larger-size, trend DONE         10125. audit_log

Inspect And Compare 

info <- ledgr_run_info(snapshot, "trend_qty_5")
info
ledgr Run Info
==============

Run ID:           trend_qty_5
Label:            Baseline quantity
Status:           DONE
Archived:         FALSE
Tags:             baseline, trend
Snapshot:         store_demo_snapshot
Snapshot Hash:    1f315cc620d0849849114d32cb44b042ec567f9512db99ad802c4ac89efe3a43
Feature Set Hash: 790156753ee13f87911742cd1a70cfcf05124271e95c0f352b4ca10370601860
Risk Chain Hash:  71863d276abfadf01e5451b8feb3ae38690b42c350db22b2740bf990358c0a11
Config Hash:      1909d4e5ecf028b692d41d85bee64ea4e5a1bf880aee5afb3d485b733f5caf99
Strategy Hash:    2e232e77ee059fecf01d125877ea3b9bca61e6a484825fd059ddd3d50985f3fc
Params Hash:      69e7ad01d1e85237d7f1593f9505f7c45d29bb55766b05abe6c067f0324ba47e
Reproducibility:  tier_1
Execution Mode:   audit_log
Fill Timing:      dense_bar_timestamp
Timing Version:   N/A
Elapsed Sec:      1.447

ledgr_run_info() is the detailed metadata view. It includes execution mode, compact telemetry, status, identity hashes, and reproducibility tier. The hashes identify every input and are the same whenever the same inputs run; Elapsed Sec describes how this execution went on this machine and can differ between otherwise identical runs.

Useful fields include:

Field Meaning
run_id, status, label, tags, archived mutable and immutable run organization fields
snapshot_id, snapshot_hash sealed data identity
strategy_source_hash, strategy_params_hash, config_hash strategy, parameter, and run-configuration identity
strategy_schedule, schedule_hash canonical decision schedule and its identity
reproducibility_level strategy preflight tier recorded with the run
execution_mode, elapsed_sec, pulse_count execution telemetry
decision_pulse_count, scheduled_skip_count invoked decisions and schedule-controlled skips
error_msg failure diagnostic for non-completed runs

A scheduled skip means ledgr deliberately did not construct a decision context or invoke the callback on that pulse. It is not an explicit ctx$hold() from the strategy. Economic events, reconciliation, strategy-state persistence and checkpointing still run, so the two counts describe callback cadence rather than missing market time.

comparison <- ledgr_run_compare(snapshot, run_ids = c("trend_qty_5", "trend_qty_15"))
comparison
# ledgr comparison
# A tibble: 2 x 11
  run_id       label final_equity total_return sharpe_ratio max_drawdown n_trades win_rate
  <chr>        <chr>        <dbl> <chr>               <dbl> <chr>           <int> <chr>
1 trend_qty_5  Base~       10042. +0.4%               0.838 -0.5%              12 25.0%
2 trend_qty_15 <NA>        10125. +1.3%               0.851 -1.5%              12 25.0%
# i 3 more variables: valuation_state <chr>, assumed_terminal_outcomes <int>,
#   reproducibility_level <chr>

# i 2 runs attempted: 2 done, 0 incomplete, 0 failed.
# i Only DONE runs are compared; incomplete and failed runs stay listed by ledgr_run_list().
# i Fill timing comparable: yes (same_timing_convention).
# i Full identity and telemetry columns remain available on this tibble.
# i Inspect one run with ledgr_run_info(snapshot, run_id).

The two runs differ only in quantity: tripling it roughly triples the profit and the drawdown while the Sharpe ratio barely moves. Both Sharpe ratios are annualized from half a year of two demo instruments, so they describe this sample, not the rule. Comparison is read-only and does not rerun strategies. n_trades counts closed, realised trade observations, not every fill. A run can have fills but no closed trades yet, in which case win rate is not defined.

ledgr_run_compare() starts from the durable snapshot handle because it reads stored run artifacts. When you want the comparison to use an experiment’s metric assumptions, pass that context explicitly:

exp_comparison <- ledgr_run_compare(
  snapshot,
  run_ids = c("trend_qty_5", "trend_qty_15"),
  metric_context = ledgr_metric_context(exp)
)

The printed comparison formats some columns for reading, and a select() on the comparison keeps that print. The columns underneath are raw numbers, so code can use them directly, as the ranking below does. For a report table that prints the numbers as they are, convert with as_tibble() first and format percentages yourself:

comparison_report <- comparison |>
  as_tibble() |>
  select(run_id, final_equity, total_return, sharpe_ratio, max_drawdown, n_trades)

comparison_report
# A tibble: 2 x 6
  run_id       final_equity total_return sharpe_ratio max_drawdown n_trades
  <chr>               <dbl>        <dbl>        <dbl>        <dbl>    <int>
1 trend_qty_5        10042.      0.00418        0.838     -0.00499       12
2 trend_qty_15       10125.      0.0125         0.851     -0.0148        12

After selecting a run, reopen it and inspect the underlying result tables rather than parsing the printed comparison:

best_run_id <- comparison |>
  arrange(desc(total_return)) |>
  pull(run_id) |>
  first()

best_bt <- ledgr_run_open(snapshot, best_run_id)
tail(ledgr_results(best_bt, what = "equity"), 3)
# A tibble: 3 x 6
  ts_utc     equity   cash positions_value running_max drawdown
  <date>      <dbl>  <dbl>           <dbl>       <dbl>    <dbl>
1 2019-06-26 10125. 10125.               0      10201. -0.00743
2 2019-06-27 10125. 10125.               0      10201. -0.00743
3 2019-06-28 10125. 10125.               0      10201. -0.00743
close(best_bt)

Inspect Stored Strategy Source

Completed runs keep strategy provenance in the experiment store: the source text, parameters, hashes and dependency metadata that produced each run. ledgr_run_strategy(snapshot, run_id, trust = FALSE) reads them without executing anything; vignette("reproducibility", package = "ledgr") teaches that reader and when trusted recovery is appropriate.

When a run ID is missing, store lookup helpers fail with class ledgr_run_not_found, so code can tell a typo from other store errors:

tryCatch(
  ledgr_run_info(snapshot, "missing_run"),
  ledgr_run_not_found = function(error) class(error)[[1]]
)
[1] "ledgr_run_not_found"

Reopen A Completed Run In A Later Session

ledgr_run_open() reconstructs a completed run handle from stored artifacts. It does not recompute the strategy. This is useful when you want full result tables or plots after restarting R.

reopened <- ledgr_run_open(snapshot, "trend_qty_5")
summary(reopened)
ledgr Backtest Summary
======================

Performance Metrics:
  Total Return:        0.42%
  Annualized Return:   0.82%
  Max Drawdown:        -0.50%

Risk Metrics:
  Risk-Free Rate:      0.00% annual
  Annualization:       252 periods/year (US equity daily (default))
  Volatility (annual): 0.98%
  Sharpe Ratio:        0.838

Trade Statistics:
  Closed Trades:       12
  Win Rate:            25.00%
  Avg Trade:           $3.48

Exposure:
  Time in Market:      66.67%

Execution Evidence:
  Fill Timing:         dense_bar_timestamp

Corporate-Action Evidence:
Economic events: NOT SUPPLIED - returns may omit distributions
Price basis: UNDECLARED - distribution double counting cannot be ruled out
  Full policy record: ledgr_corporate_action_summary(bt)
tail(ledgr_results(reopened, what = "equity"), 3)
# A tibble: 3 x 6
  ts_utc     equity   cash positions_value running_max drawdown
  <date>      <dbl>  <dbl>           <dbl>       <dbl>    <dbl>
1 2019-06-26 10042. 10042.               0      10067. -0.00251
2 2019-06-27 10042. 10042.               0      10067. -0.00251
3 2019-06-28 10042. 10042.               0      10067. -0.00251
close(reopened)

Only completed runs can be reopened. Failed or incomplete runs remain inspectable through ledgr_run_info().

Store-level helpers such as ledgr_run_info(), ledgr_run_list(), and ledgr_run_compare() use the snapshot handle and remain available after a completed run handle is closed. Result-table helpers such as ledgr_results() need a live or reopened backtest handle.

Recovery

Most ledgr recovery workflows should start with the high-level store helpers above:

The lower-level recovery pair is public for restart inspection and for tools that need to work directly against the store connection.

ledgr_db_init(db_path) opens a DBI connection to a ledgr DuckDB store and ensures the ledgr schema exists. In normal workflows, ordinary users usually do not need it because snapshot and run helpers open, verify, and close the required connections for their own operations.

ledgr_state_reconstruct(run_id, con) reconstructs ledgr’s expected simulated state for one stored run from ledger-backed evidence. It returns a list of positions, cash, profit and loss (pnl) and an equity curve (equity_curve) rebuilt from the stored run records. It is useful when you are inspecting a restart boundary, debugging stored evidence, or building a low-level tool that already owns a DBI connection.

con <- ledgr_db_init(db_path)
state <- ledgr_state_reconstruct("trend_qty_5", con)
DBI::dbDisconnect(con, shutdown = TRUE)

names(state)
[1] "positions"    "cash"         "pnl"          "equity_curve"

This pair is intentionally not a broker or migration layer. It does not perform broker reconciliation, prove live restart safety, migrate old schemas, repair a sealed snapshot, or recover strategy dependencies that were never captured in the run provenance. Treat it as low-level inspection over ledgr’s own stored evidence, not as an escape hatch around the sealed-data and provenance contracts.

Archive Without Deleting 

snapshot <- snapshot |>
  ledgr_run_archive("trend_qty_15", reason = "larger position kept for reference")

ledgr_run_list(snapshot)
# ledgr run list
# A tibble: 1 x 11
  run_id      label             tags            status final_equity total_return
  <chr>       <chr>             <chr>           <chr>         <dbl> <chr>
1 trend_qty_5 Baseline quantity baseline, trend DONE         10042. +0.4%
  complete_performance achieved_end_utc valuation_state execution_mode
  <lgl>                <dttm>           <chr>           <chr>
1 NA                   NA               <NA>            audit_log
  reproducibility_level
  <chr>
1 tier_1

# i 1 runs attempted: 1 done, 0 incomplete, 0 failed.
# i INCOMPLETE metrics describe the achieved prefix only.
# i Full identity and telemetry columns remain available on this tibble.
# i Inspect one run with ledgr_run_info(snapshot, run_id).

Archiving hides a run from default listings without deleting artifacts.

Back Up A Closed Store

The experiment store is an ordinary DuckDB file. Back it up only after every run and snapshot handle using it has been closed.

Back up closed stores

Close live handles, then copy or sync the store with the rest of the project’s research artifacts.

dir.create("backups", showWarnings = FALSE)
file.copy(
  "artifacts/ledgr_store.duckdb",
  file.path("backups", paste0("ledgr_store_", Sys.Date(), ".duckdb")),
  overwrite = TRUE
)

Do not copy a file while another ledgr process has it open. For a larger project, keep the same closed-file rule and use your normal versioned backup or sync system.

Resource Cleanup

ledgr_run() and ledgr_run_open() return live handles for durable run artifacts. The artifacts are already durable when a run completes, and ordinary result inspection opens and closes read connections per operation. Use close(bt) as explicit resource cleanup in long sessions, tests, and explicit-open workflows. Close snapshot handles when the workflow is finished.

Where Next