Skip to contents

On Leakage: ledgr Design Choices

Source: vignettes/leakage.qmd

A backtest is valid only if each decision uses information that was knowable at that point in time – and nothing else. By the end of this article you will be able to spot an obvious lead, distinguish expanding from full-sample preprocessing, and state which leakage risks ledgr cannot certify for you.

library(ledgr)
library(dplyr)
data("ledgr_demo_bars", package = "ledgr")

bars <- ledgr_demo_bars |>
  filter(
    instrument_id %in% c("DEMO_01", "DEMO_02"),
    between(
      ts_utc,
      ledgr_utc("2019-01-01"),
      ledgr_utc("2019-06-30")
    )
  )

Leakage is what happens when that boundary is violated. It can be obvious, or it can be hidden inside preprocessing that looks harmless.

ledgr’s workflow is designed around this problem. The strategy interface, sealed snapshots, and registered feature definitions each make a common leakage pattern harder to write by accident – not just documented as bad practice. They are not a sandbox: R code can still reach data ledgr did not hand it. This article explains where that protection applies and where your responsibility begins.

Leakage breaks the question

A leaky backtest can have correct arithmetic and still answer the wrong research question. ledgr narrows common leakage paths, but it cannot certify your data availability or research process.

Definition

Leakage is any path by which a decision uses information that was not knowable at the simulated decision time.

The Obvious Leak

The blunt example is lead(close): tomorrow’s close placed onto today’s row.

leaky_signals <- bars |>
  group_by(instrument_id) |>
  arrange(ts_utc, .by_group = TRUE) |>
  mutate(
    tomorrow_close = lead(close),
    buy_signal = tomorrow_close > close
  )

leaky_signals |>
  select(instrument_id, ts_utc, close, tomorrow_close, buy_signal) |>
  slice_head(n = 4)
#> # A tibble: 8 x 5
#> # Groups:   instrument_id [2]
#>   instrument_id ts_utc              close tomorrow_close buy_signal
#>   <chr>         <dttm>              <dbl>          <dbl> <lgl>     
#> 1 DEMO_01       2019-01-01 00:00:00  91.5           91.3 FALSE     
#> 2 DEMO_01       2019-01-02 00:00:00  91.3           90.5 FALSE     
#> 3 DEMO_01       2019-01-03 00:00:00  90.5           89.8 FALSE     
#> 4 DEMO_01       2019-01-04 00:00:00  89.8           89.2 FALSE     
#> 5 DEMO_02       2019-01-01 00:00:00  73.9           74.3 TRUE      
#> 6 DEMO_02       2019-01-02 00:00:00  74.3           74.8 TRUE      
#> 7 DEMO_02       2019-01-03 00:00:00  74.8           74.9 TRUE      
#> 8 DEMO_02       2019-01-04 00:00:00  74.9           73.8 FALSE

The table is still grouped by instrument, so slice_head(n = 4) shows the first four rows of each. Every TRUE marks a day whose next close is higher, which is known only the day after. The resulting buy_signal looks like an ordinary column, but it answers a question the strategy could not have answered at today’s decision time.

A strategy that trades on this signal is contaminated even if the resulting trades are not all profitable. The signal was created from tomorrow’s outcome, so the backtest is no longer measuring a decision rule that could have existed in real time. The arithmetic may be correct; the question being measured is not.

The Subtle Leak

Real leakage is often less cartoonish. A full-sample threshold can leak without calling lead().

leaky_features <- bars |>
  group_by(instrument_id) |>
  arrange(ts_utc, .by_group = TRUE) |>
  mutate(
    ret_5 = close / lag(close, 5) - 1,
    strong_return = ret_5 > quantile(ret_5, 0.75, na.rm = TRUE)
  )

leaky_features |>
  select(instrument_id, ts_utc, ret_5, strong_return) |>
  filter(!is.na(ret_5)) |>
  slice_head(n = 4)
#> # A tibble: 8 x 4
#> # Groups:   instrument_id [2]
#>   instrument_id ts_utc                 ret_5 strong_return
#>   <chr>         <dttm>                 <dbl> <lgl>        
#> 1 DEMO_01       2019-01-08 00:00:00 -0.0316  FALSE        
#> 2 DEMO_01       2019-01-09 00:00:00 -0.0302  FALSE        
#> 3 DEMO_01       2019-01-10 00:00:00 -0.0167  FALSE        
#> 4 DEMO_01       2019-01-11 00:00:00 -0.0157  FALSE        
#> 5 DEMO_02       2019-01-08 00:00:00  0.00559 FALSE        
#> 6 DEMO_02       2019-01-09 00:00:00 -0.00942 FALSE        
#> 7 DEMO_02       2019-01-10 00:00:00 -0.0123  FALSE        
#> 8 DEMO_02       2019-01-11 00:00:00 -0.0230  FALSE

There is no future row reference in the final rule. The leak happened earlier: the January rows are compared against a return distribution that includes later months. Future information has been baked into a normal-looking feature.

A threshold estimated on a prior training sample and then frozen for later evaluation can be valid. The leak appears when the threshold is estimated from the same future-inclusive sample on which early decisions are judged.

The gap is concrete. Suppose the first quarter of the sample has strong returns and the remaining three quarters are weak:

set.seed(42)
ret_5 <- c(
  rnorm(63,  mean =  0.004, sd = 0.010),  # first quarter: strong
  rnorm(189, mean = -0.001, sd = 0.012)   # remaining three quarters: weak
)

full_sample <- unname(quantile(ret_5, 0.75, na.rm = TRUE))
expanding <- vapply(seq_along(ret_5), function(i) {
  if (i <= 20L) {
    return(NA_real_)
  }
  unname(quantile(ret_5[seq_len(i - 1L)], 0.75, na.rm = TRUE))
}, numeric(1))

first_quarter <- seq_len(63L)
tibble(
  rule = c("full sample", "expanding, prior rows only"),
  threshold_at_quarter_end = c(full_sample, expanding[[63L]]),
  first_quarter_signals = c(
    sum(ret_5[first_quarter] > full_sample),
    sum(ret_5[first_quarter] > expanding[first_quarter], na.rm = TRUE)
  )
)
#> # A tibble: 2 x 3
#>   rule                       threshold_at_quarter_end first_quarter_signals
#>   <chr>                                         <dbl>                 <int>
#> 1 full sample                                 0.00775                    24
#> 2 expanding, prior rows only                  0.0104                      8

The full-sample threshold is lower because the later weak-return period drags the distribution down. The expanding rule uses only rows that precede each decision, so it is the causal comparison. Here the future-inclusive rule emits many more first-quarter signals. The exact count is less important than the direction: later observations changed earlier decisions.

The Strategy Boundary 

flowchart LR
  data["sealed data"]
  features["registered features"]
  pulse["pulse context"]
  target["target holdings"]
  ledger["ledger events"]

  data --> features --> pulse --> target --> ledger

At execution time, ledgr strategies receive one pulse context. The strategy can read current bars, current registered features, current positions, cash, equity, and helper functions such as ctx$flat() and ctx$hold().

The strategy does not receive the full future market-data table. It has no market-data object from which it can casually index tomorrow’s bar.

strategy <- function(ctx, params) {
  up <- ctx$vec$close > ctx$vec$open
  ctx |>
    ledgr_selection(where = up) |>
    ledgr_target_quantity(ctx, params$qty)
}

Everything this rule reads comes from ctx: today’s close and open for every instrument. Nothing in it can reach tomorrow’s bar.

This boundary prevents one common class of leakage. It does not cleanse contaminated data that were already handed to ledgr, and it does not stop a strategy from reading data it was not handed. A function can close over an object from your session, such as the full bars table or a column of precomputed signals, and index it by ctx$ts_utc. Preflight classifies such a strategy as Tier 2 and records a note naming the captured object; it does not refuse it, and ledgr cannot tell whether that object holds future rows.

The Feature Boundary

Registered indicators are ledgr’s feature boundary. A feature is not just a casual column added to a data frame. It has an ID, parameters, required history, warmup/stability rules, and a deterministic place in the pulse context.

Scalar indicator functions are evaluated on bounded historical windows ending at the current bar. Vectorized series_fn functions, which compute a custom indicator over an instrument’s whole series (vignette("custom-indicators", package = "ledgr")), must return a numeric series aligned to the input bars, with valid warmup and post-warmup values. Unknown feature IDs fail loudly at pulse time rather than silently becoming missing values.

Those rules make accidental leakage harder than ad hoc full-sample signal columns.

The Sharp Edge: series_fn

Vectorized custom indicators are useful because many indicators are naturally computed over a full series. They are also a residual risk. Output shape and value checks can prove that a vector has the right length and valid values. They cannot prove that the function avoided future rows internally. Availability-aware experiments recompute each feature from truncated history at a few fixed sessions and refuse when the value changes, which catches a function that reads later rows at those sessions; it is a sample, not a proof.

The custom-indicator article explains this boundary in more detail. The core warning is the same: full-series custom feature code must be written with causal discipline.

What ledgr Enforces At These Boundaries

Boundary ledgr behavior
Strategy reads future rows Strategy receives a pulse context, not a future table; an object captured from your session is noted by preflight, not blocked.
Unknown feature ID ctx$feature() fails loudly instead of returning warmup.
Known feature in warmup Values are explicitly NA_real_ until usable.
Invalid target shape Missing or extra target instruments are rejected.
Scalar indicator history Scalar indicators are evaluated on bounded windows ending at the current bar.
Vectorized feature output series_fn output must be aligned, numeric, and valid after warmup.
Feature look-ahead (availability-aware) Each definition is recomputed from truncated history at no more than four fixed sessions and refuses with ledgr_feature_lookahead_detected when its value changes.

What Remains Your Responsibility

ledgr does not certify that the dataset, event timestamps, universe construction, parameter search, or custom vectorized feature code are causally clean.

Use vignette("point-in-time-inputs", package = "ledgr") for knowledge-time facts and vignette("survivorship-bias", package = "ledgr") for membership, lifetime, and terminal-event evidence. Those tools make declared evidence executable; they cannot repair vendor data that never recorded when a value was actually knowable.

Risk Why ledgr cannot fully solve it
Survivorship-biased universe A snapshot may already exclude dead or unavailable instruments.
Bad availability timestamps Event, fundamental, or macro data may be timestamped by label date rather than when it was tradable information.
Restated data Vendors may revise values after the simulated decision time.
Research-loop leakage Trying many ideas and reporting the best one turns the sample into training data.
Semantically leaky series_fn A shape-valid vector can still be future-aware internally.
Captured session objects A strategy can close over a data frame from your session and read rows the pulse context would not give it.
Split-adjusted price levels Split-adjusted bars restate history at today’s share basis, so a rule that uses an absolute price level, such as a minimum price or a lot size, sees levels that did not exist before a later split.

Checklist Before Trusting A Run

  • Was the snapshot built from data available at the simulated decision times?
  • Was the universe chosen without future survival information?
  • Are event or fundamental timestamps availability timestamps?
  • Are features registered as indicators rather than full-sample signal columns?
  • Does the strategy fill after the information it used?
  • Did parameter choices survive out-of-sample or regime checks?
  • Can the run be reopened and explained from stored provenance?

Try it

Change the weak later regime to a strong one and rerun the expanding comparison. Which early decisions change under the full-sample rule, and why can the expanding rule not see that regime yet?

What To Remember

ledgr protects the strategy boundary and makes the feature boundary explicit. That is necessary, not sufficient.

A causally honest backtest still depends on clean data availability, honest universe construction, disciplined feature authoring, and a research process that does not turn the test sample into training data.

Where Next