trader.replay.scoring module

Score replayed decisions against an always-BUY null.

Pure functions over a row list — no I/O, no model calls — so every scoring rule is testable with no Ollama and no database. score_all is the one function that reaches into outcomes.py, turning one replayed decision plus its symbol’s ForwardPrices into a ScoredRow; everything else here operates on ScoredRows alone.

class trader.replay.scoring.EdgeResult(model_return, null_return, edge, n_acting, n_rows)

Bases: object

The model’s edge over a null, at one horizon.

Parameters:

• model_return (float | None)

• null_return (float | None)

• edge (float | None)

• n_acting (int)

• n_rows (int)

model_return: float | None

null_return: float | None

edge: float | None

n_acting: int

n_rows: int

class trader.replay.scoring.ScoredRow(symbol, decision_at, split, entry_day, action, confidence, returns, mae, dense)

Bases: object

One replayed decision, with its forward returns and MAE at every horizon.

Parameters:

• symbol (str)

• decision_at (datetime)

• split (Literal['train', 'holdout'])

• entry_day (date)

• action (Action)

• confidence (float | None)

• returns (dict[int, Decimal | None])

• mae (dict[int, Decimal | None])

• dense (bool)

symbol: str

decision_at: datetime

split: Literal['train', 'holdout']

entry_day: date

action: Action

confidence: float | None

returns: dict[int, Decimal | None]

mae: dict[int, Decimal | None]

dense: bool

trader.replay.scoring.abstention_value(rows, n)

mean(r | HOLD) - mean(r | BUY). Negative means the HOLDs carry information.

None when either side has no rows with a defined horizon — there is no abstention value to report from an empty comparison.

Parameters:

• rows (Sequence[ScoredRow])

• n (int)

Return type:

Decimal | None

trader.replay.scoring.confidence_tertiles(train_rows)

Two cut points splitting train_rows’ confidences into thirds.

Computed on train and applied to holdout unchanged by the caller — recomputing on holdout is how a calibration result becomes circular.

Parameters:

train_rows (Sequence[ScoredRow])

Return type:

tuple[float, float]

trader.replay.scoring.counts(rows)

Row counts by action, symbol, year, density (dense/thin), and split.

Parameters:

rows (Sequence[ScoredRow])

Return type:

dict[str, dict[str, int]]

trader.replay.scoring.edge(rows, n, *, include_holds_in_null)

edge_A (include_holds_in_null=False) or edge_B (True).

edge_A’s null is always-BUY over the same acting rows the model traded — never over every row, which would let an unrelated HOLD’s raw return leak into the comparison and swing the edge on rows the model never acted on. edge_B’s null spans every row with a defined horizon, which is what makes it answer “does abstaining have value” instead.

Parameters:

• rows (Sequence[ScoredRow])

• n (int)

• include_holds_in_null (bool)

Return type:

EdgeResult

trader.replay.scoring.edge_a_is_degenerate(rows, n)

True when edge_A’s null cannot possibly differ from its model side.

edge_A’s null is always-BUY over the same acting rows the model traded (see edge’s docstring): for each acting row it takes the raw return r, exactly as signed_return does for a BUY. If there is no SELL among the acting rows at horizon n, signed_return returns +r for every one of them, so the model-side list and the null-side list are the identical sequence of values — edge_A == 0.0 by construction at every draw of a bootstrap over this row set, never by measurement (issue #29). Built the same way edge builds its own two lists, so a change to one without the other cannot silently drift them apart.

The vacuous case (zero acting rows at this horizon) also returns True: both lists are empty and therefore identical, and edge already reports None rather than a misleading 0.0 in that case — flagging it degenerate too costs nothing and is the conservative side to be wrong on.

Parameters:

• rows (Sequence[ScoredRow])

• n (int)

Return type:

bool

trader.replay.scoring.hit_rate(rows, n)

(acting hit rate, base rate) — fraction of rows with signed_return > 0.

The acting rate is over BUY/SELL rows scored by their signed return; the base rate is r > 0 across every row with a defined horizon, acting or not, so the two are comparable as “the model” versus “the market”.

Parameters:

• rows (Sequence[ScoredRow])

• n (int)

Return type:

tuple[float, float]

trader.replay.scoring.market_adjusted(rows, benchmark)

Subtract SPY’s same-date, same-horizon return, dropping SPY’s own rows.

benchmark maps (entry_day, horizon) -> SPY’s forward_return, built by the caller from SPY’s own ForwardPrices over the sample’s entry days.

Parameters:

• rows (Sequence[ScoredRow])

• benchmark (dict[tuple[date, int], Decimal])

Return type:

list[ScoredRow]

trader.replay.scoring.score_all(*, symbol, decision_at, split, entry_day, action, confidence, prices, dense)

Build a ScoredRow from a decision and the symbol’s forward prices.

prices=None (forward outcomes not yet fetched, e.g. a –dry-run pass) scores every horizon as None rather than raising: a dry run exercises the prompt and the point-in-time guards without pricing a single row.

Parameters:

• symbol (str)

• decision_at (datetime)

• split (Literal['train', 'holdout'])

• entry_day (date)

• action (Action)

• confidence (float | None)

• prices (ForwardPrices | None)

• dense (bool)

Return type:

ScoredRow

trader.replay.scoring.signed_return(action, r)

+r for BUY, -r for SELL, None for HOLD (not an acting row) or a missing return.

Parameters:

• action (Action)

• r (Decimal | None)

Return type:

Decimal | None

trader.replay.scoring.tertile_means(rows, cuts, n)

Mean signed_return at horizon n, bucketed by cuts (from confidence_tertiles).

Parameters:

• rows (Sequence[ScoredRow])

• cuts (tuple[float, float])

• n (int)

Return type:

dict[str, Decimal | None]