A probability is not a permission slip.

Rhea’s first problem sounds small: estimate whether a five-minute BTC direction event will resolve Up. The larger problem is to build a system that states what it knows and estimates. It must also state what it may do and why it sometimes refuses to act.

1. Machine learning learns from examples, not from destiny

Supervised machine learning starts with historical examples. Each example contains observations available at a decision time and a later label. A model fits a function that maps observations to a score. It does not discover a permanent law of the world. Instead, it finds a pattern that may not survive a new period, source, population, or measurement process.

Here, Xₜ is the information permitted at time t; yₜ is a later binary outcome; fθ is a learned model; and p̂ₜ is its estimate. The approximation sign is a warning label. The score is a claim to evaluate, not a command to obey.

This distinction applies across domains. A fraud model estimates suspiciousness; a clinical decision-support model estimates risk; an intrusion detector estimates the likelihood of a harmful class. The organization—not the model—must decide which actions are permitted, when uncertainty is too high, and who bears responsibility for mistakes.

2. Five objects that should never be collapsed into one word

Students often call everything that follows “the prediction.” Rhea separates these objects because each has a different truth condition and failure mode.

3. Prediction is not decision

Suppose Orbit produces a score of 0.58. A casual interpretation is “Orbit says Up.” That interpretation skips several questions. Is 0.58 calibrated? Which reference probability or quoted state would the decision use? Which costs and delays matter? Is the data fresh? Is the model artifact approved? Has the risk budget been consumed? A system that skips these questions turns a numerical output into unearned authority.

The exact policy is a design choice and must be explicit. The logic fails closed: if the system cannot prove a necessary condition, it abstains. This design does not make the model timid. It makes the boundary between evidence and authority visible.

4. The limits of pattern learning

A model may exploit correlations that are temporary, accidental, or created by the data collection process. It may mistake a proxy for a causal factor. A label leak can make the model appear strong, while average performance can hide failure in the regime that matters. These are not exceptional embarrassments; they are ordinary risks of fitting flexible functions to historical data.

Rhea therefore asks every model claim four questions. What object is being estimated? What evidence entered the estimate? What decision changes because of it? What can go wrong? The rest of the book turns those questions into data contracts, metrics, tests, models, replay, and monitoring.

5. The first research contract

Before training, Rhea writes a contract in ordinary language. It names the question, settlement source, horizon, observation cutoff, allowed sources, label rule, missing-data response, model output, policy output, and evidence retained. A good contract is specific enough that two researchers can build the same dataset and notice if they disagree.

6. The next vulnerability: time

Rhea can now name the pieces, but she still needs to protect the clock. The next chapter makes the prediction contract temporal. It separates observation time from settlement time and closes the path through which future data can make a weak model look miraculous.

The clock that caught a cheat.

Orbit produces an astonishing score. Rhea asks it to replay one decision. A feature marked 10:00:00 includes an update received at 10:00:04. The model did not become clever. The experiment let it see the future.

1. A label is a contract, not a column name

Terms such as “BTC up in five minutes” sound precise until two researchers implement them. One uses an exchange midpoint; another uses a contract’s stated settlement source. One treats equality as Up; another treats it as Down. One uses a closing value; another uses the first tick after the horizon. They have built different targets, even if both call the column up_5m.

Rhea writes the label contract in plain language and notation. It names the source, start clock, end clock, horizon, equality rule, source publication behavior, and missing-data response. An illustrative binary label can be written as:

S is the one declared settlement source and H is the horizon. The expression is small, but every term needs an implementation definition. If S(t + H) is unavailable, the system does not substitute a convenient different source. It returns an unavailable label or follows a predeclared correction policy.

2. The evidence set has a different contract

The outcome is learned later. The model inputs must be frozen earlier. Rhea defines eligibility using availability rather than hindsight:

An event belongs in the feature set only if the system could have observed it by the cutoff. This rule is independent of the event’s own timestamp. A historical archive can know that an event happened at 10:00:00; a live-like system may not receive it until 10:00:02.

3. Common leakage paths

Leakage is often more subtle than directly inserting the label. A rolling statistic may include a late event. A normalizer may be fit on the full dataset, including final-test values. A categorical feature may be computed from an outcome that resolves later. A correction may rewrite the historical archive and make it look as if the corrected value was available originally. A random split may let overlapping windows appear on both sides of evaluation.

4. Missing and corrected observations

Rhea refuses to treat missing data as an invitation to make up a smooth story. A missing settlement value, dropped source event, delayed update, or correction can change the label or the feature set. The contract must say whether the example is unavailable, quarantined, forward-filled under a limited rule, or retained with a missingness indicator. The policy must log which option occurred.

Corrections require special care. A source may later correct an event timestamp or value. The original record is evidence of what the system could have known. The corrected record may be relevant to post-hoc truth, but it must not silently rewrite availability in a replay that claims to be live-like. Rhea stores both and documents the chosen truth policy.

5. Tests that protect the clock

Temporal validity should be tested, not asserted. Rhea’s smallest tests add an event received after the cutoff, mutate a future event, use an unknown source schema, and create a missing settlement. Each case has an explicit result: exclude, reject, quarantine, or mark unavailable. A replay test then walks a small fixture through the full path and verifies that a later correction cannot influence an earlier decision.

6. From clocks to evidence

Rhea has frozen time and defined truth. The system still needs a defensible path from raw events to features. The next chapter builds that path, preserves the original evidence, validates each transformation, and gives every feature a reproducible recipe.

The evidence forge.

Rhea has defined the label and locked the clock. Orbit still receives a storm of raw events: duplicates, late arrivals, malformed records, and schema changes. A clean table is not enough. Rhea needs a chain of evidence that a reviewer can walk backward.

1. Raw evidence comes first

Machine-learning models consume tensors, rows, and vectors. Real systems receive messages, snapshots, updates, files, and corrections. The transformation from real events into model input is part of the scientific method. If a team cannot say where a value came from, when it arrived, which rules accepted it, and which formula transformed it, the value is not defensible evidence.

Rhea stores raw events immutably. Each record includes a source identifier, source event ID when available, capture ID, payload, event time, receive time, schema version, capture outcome, and content hash. “Immutable” does not mean “trusted.” It means later validation does not rewrite what originally arrived.

Every arrow represents a testable claim. The raw event claims what the source sent. The validation result claims whether it meets the current contract. The eligible set claims it satisfies the cutoff. The recipe claims how values were computed. The feature vector claims what the model actually received.

2. A record needs a passport

Without identity and time fields, duplicates and corrections become impossible to reason about. Without schema version, a parser can silently reinterpret a field. Without a hash or capture ID, a later reviewer cannot tell whether two records are the same observation or two different observations that happen to look similar.

3. Validation is not a cleaning step

Validation determines whether an event satisfies structural and semantic rules. Structural checks cover required fields, types, formats, and schema compatibility. Semantic checks identify impossible quantities, nonmonotonic sequence numbers, out-of-order events, stale receive times, duplicate IDs, and values outside declared ranges. The system marks and quarantines a failed record instead of discarding it.

Some policies can be defensible for a specified task. Forward-fill may be permitted for a bounded interval when evidence shows that a field persists between updates. It may be unsafe when the missing field can change quickly. The right answer is neither “always fill” nor “never fill.” It is a predeclared rule with a failure mode and sensitivity analysis.

4. A feature is a hypothesis with a recipe

Features are summaries of raw events. A recent-count feature hypothesizes that activity in a defined window may carry information. A depth-imbalance feature hypothesizes that selected quantities and levels can be combined meaningfully. Neither becomes a fact because it is computable.

Rhea’s feature dictionary lists name, purpose, formula, input fields, time window, cutoff rule, units, normalization, missing-data behavior, expected range, recipe version, and test cases. The version changes whenever the formula, input set, rounding, resampling, normalization, or missingness behavior changes.

5. Quality gates shape model behavior

Data-quality decisions are model decisions because they change the input distribution. If quiet periods have more missing events and the pipeline drops them, the remaining dataset can overrepresent active periods. If a forward-fill policy creates stable-looking depth, the model can learn an artifact of the pipeline. Rhea monitors validation failures by source, time, schema, and feature recipe to locate weak evidence.

When this conjunction is false, the safe result is not a fabricated feature vector. It is an ineligible result with a reason. That reason becomes an input to monitoring and later incident analysis.

6. From valid rows to valid interpretation

Rhea can now produce a valid row. The next challenge is interpretation: a model can score that row and still be judged by the wrong metric. Chapter 4 distinguishes classification, ranking, and probability metrics so that none borrows meaning from another.

The golden score that did not mean what it said.

Rhea has a model score. The hard part is learning whether the score describes skill, coincidence, a trivial baseline, or a broken experiment.

1. A metric is a question with a denominator

Metrics are not medals attached to a model. Each metric asks a specific question about a specified set of predictions. Accuracy asks what fraction of all classifications were correct. Precision asks whether positive predictions were reliable. Recall asks whether the system found the positive cases. Specificity asks how well it avoided false alarms among negative cases. F1 combines precision and recall, but it does not make their underlying tradeoff disappear.

The first object Rhea requests is the confusion matrix. It forces the team to count the four possible outcomes instead of hiding them behind a single percentage. Let TP denote a true positive, FP a false positive, TN a true negative, and FN a false negative.

2. Why accuracy can be nearly perfect and useless

Suppose 98 of 100 examples are Down. A classifier that always says Down achieves 98% accuracy. It has detected no Up cases and supplied no useful probability distinction. This is not a corner case; imbalanced outcomes are common in event detection, fraud screening, medical triage, and any task where the event of interest is rare.

Rhea asks Orbit to compare against the simplest baselines first. A constant predictor always returns the observed training prevalence. A majority-class classifier turns that prevalence into a hard label. If an external reference probability exists, it is a separate baseline. A complex model earns attention only if it adds out-of-sample information beyond these references under the same split and data contract.

3. Ranking is not calibration

Orbit then shows Rhea a high AUC. That may be useful: it suggests positive outcomes often receive higher scores than negative outcomes. But AUC does not require the score 0.80 to mean an 80% chance. A model may rank two cases correctly while being dramatically overconfident about both.

Calibration compares forecasts to frequencies. Take a held-out period, group examples with predicted probability near 0.70, and measure how often the outcome occurred. If the observed frequency is near 0.70, the group is calibrated. If it is near 0.45, the model is overconfident in that region. A reliability curve visualizes this relationship. The Brier score measures average squared probability error; log loss penalizes confident wrong forecasts sharply.

4. The metric report Rhea will accept

Rhea does not ask Orbit for a dashboard of impressive numbers. She asks for a compact evidence record. It names the outcome definition, class balance, evaluation period, split, baseline, confusion matrix, ranking metric, probability metric, calibration plot, and uncertainty interval or limitation. The record then states which decisions these observations can and cannot support.

5. The test behind the score

Even a rigorous metric report can be wrong when time leakage contaminates the test period or repeated tuning consumes it. The next chapter therefore locks the test away. Rhea trains earlier, validates later, inserts a purge gap where windows overlap, and opens the final test only after freezing every choice.

The exam Orbit had already seen.

A model can earn a high score because it learned a useful pattern—or because the experiment quietly gave it answers in advance. Rhea’s task is to make the second explanation impossible.

1. Why random splitting is often wrong for temporal evidence

Random train/test splits assume that examples are independent enough that mixing their order does not carry information across the boundary. That assumption can be reasonable for some static datasets. It is often false for time-ordered data. Neighboring events may share a feature window, a label window, a market state, a participant, or a data-processing artifact. A model trained on one neighbor can effectively recognize the other.

The remedy is not merely “use dates.” The split must reflect the question the system will face. If the intended decision arrives over time, training must occur earlier than validation, and validation must occur earlier than the final test. Any period whose inputs or labels overlap a boundary needs special treatment.

2. The job of the purge and embargo

A purge removes examples whose label horizon overlaps the next evaluation interval. An embargo adds a temporal gap so that delayed labels, shared features, or data corrections do not cross the boundary. Its length is not a ritual number. It follows from the largest lookback window, label horizon, availability delay, and source behavior declared in the prediction contract.

Suppose a feature looks back five minutes and its label resolves five minutes after the decision. A noon split cannot automatically treat the 11:59:59 and 12:00:01 examples as independent. The correct gap depends on how the pipeline constructs features, labels, and source availability. Rhea records this reasoning before seeing model results.

3. Baselines are scientific controls

Before Orbit earns a complex model, Rhea asks what it must beat. A baseline is not an embarrassing weak opponent. It is the control that reveals whether added complexity added information. The minimum set is usually a constant-probability baseline, a simple rule or linear/tabular baseline, and—when relevant—a valid external reference probability. All candidates must receive the same labels, splits, eligibility gates, and cost assumptions.

Complexity has costs: more hyperparameters, more opportunities to leak, longer training, less transparent failure modes, and greater risk that a small validation period selects noise. The burden of proof rises with the model’s flexibility.

4. Hyperparameter tuning is an experiment inside the experiment

A tuning system can try thousands of combinations. Repeated evaluation against one validation interval can overfit that interval even when no individual model looks suspicious. Rhea constrains the search space through prior reasoning, records every trial, and separates model development from final confirmation. Nested temporal validation makes the separation explicit. Inner folds choose settings; outer folds estimate how the selection procedure generalizes.

There is no magic number of folds. Chronology and independence matter more. The design must leave enough time to represent changing conditions while avoiding overlap. When data is limited, the report should expose uncertainty instead of hiding it behind repeated selections.

5. The experiment card

Rhea keeps a one-page experiment card before training starts. It contains the dataset manifest, label rule, source availability rule, feature versions, split dates, purge rationale, baseline list, tuning budget, primary metrics, calibration method, replay assumptions, and stopping rule. If any item changes after results appear, the run is a new experiment. This is not paperwork. It is a defense against unconscious selection of the story that looks best after the fact.

6. From ranking to probability

Rhea has protected the test from obvious time travel and tuning. Even a fair ranking model, however, may express probabilities badly. The next chapter asks whether “80%” means anything near 80%. It also shows how policy combines probability with costs and risk, and why abstention must be engineered rather than treated as failure.

The courage to do nothing.

Orbit has learned to rank cases. Rhea now asks a harder question: when Orbit says “80%,” should anyone believe that number—and what else must be true before a paper-trading simulation is even eligible?

1. A score can rank well and mean the wrong thing

Discrimination and calibration are different properties. Discrimination asks whether positive cases tend to receive higher scores than negative cases. Calibration asks whether a numerical probability matches an observed frequency. A model can sort examples in a useful order while systematically exaggerating or understating how likely they are.

Rhea evaluates calibration only on data that was not used to fit or tune the calibration method. She groups predictions into ranges and compares their average prediction with their observed frequency. A reliability curve makes the gap visible. Proper scoring rules, such as Brier score and log loss, summarize the quality of probability forecasts while penalizing confident mistakes.

2. A probability is not the market’s probability

Orbit’s probability describes its own modeled outcome under its training data and label contract. It should not be confused with a market-implied probability, quoted price, or subjective belief. Rhea keeps these quantities separate because a decision depends on their relationship and the assumptions required to compare them.

For an illustrative binary research contract, let p̂ be Orbit’s calibrated estimate, q an external reference probability, and c a conservative allowance for costs and execution uncertainty. A deliberately simplified eligibility expression might require a positive difference greater than a predeclared margin. The exact value model belongs to the replay contract. This chapter establishes separation, not a universal formula.

No valid shortcut turns a model’s 0.58 into a positive decision without naming the reference, costs, assumptions, and policy. If these inputs are missing, the correct system response is not “probably.” It is abstention.

3. Abstention is an action with a reason

In ordinary classification exercises, every row receives a label. In an operational research system, forcing a decision can be unsafe. Rhea adds an abstention state: the system may state that its evidence does not support a bounded paper-trading research signal. Abstention makes uncertainty visible instead of hiding it inside a weak threshold.

4. Calibration itself can drift

Calibration is not a permanent certificate. A model may remain numerically stable while the relationship between its inputs and outcomes changes. Rhea therefore monitors calibration on later held-out windows and reports confidence limits when samples are small. She also defines when a calibration artifact becomes stale. Recalibration cannot justify reusing the final test; it must follow the model’s temporal discipline.

Thresholds drift too. A threshold selected because it looks good on one validation interval can overfit the policy. Rhea records the threshold-selection procedure, tests it on a later interval, and prefers conservative abstention over a threshold that wins only in retrospect.

5. Reproducibility becomes the next test

Rhea now has a model, a fairer evaluation, and a policy that knows how to refuse. The entire result can still vanish if Orbit cannot recreate the exact run. The next chapter turns the research process into a reproducible Python system with manifests, configurations, tests, and an artifact ledger.

The notebook that could not remember.

Rhea asks Orbit to reproduce a strong result from last Tuesday. Orbit cannot tell her which data version, feature recipe, package set, random seed, or notebook order created it. The result may have happened; it is not yet evidence.

1. A notebook is not a research record

Notebooks are useful for exploration. They are dangerous when they become the only record of an experiment. A cell may run out of order. A variable may remain in memory from a discarded trial. A data file may be overwritten. A library update may change a default. None of these events necessarily creates an error message.

Rhea separates exploration from the run that produces a claim. The claim-producing pipeline has explicit inputs, functions, configuration, tests, and output artifacts. A notebook may call that pipeline, but it is not allowed to be the sole definition of it.

2. Build small contracts before clever code

Orbit’s research project begins with contracts, not a giant training script. A feature builder accepts an observation cutoff and returns either a versioned feature vector or an ineligible reason. A label builder accepts a settlement contract and returns either a label or an unavailable reason. A trainer accepts a manifest and configuration. A reporter cannot publish an evaluation without linking the artifacts that produced it.

The example is illustrative. Its value is not the exact syntax. It makes the temporal and provenance requirements visible in the interface. A function that accepts only an unqualified table makes it too easy to forget how the table was constructed.

3. Tests are scientific instruments

Rhea writes focused tests that protect the research contract. One asserts that an event received after the cutoff cannot appear in the feature result. Another confirms that a model artifact refuses an unfamiliar feature-recipe version. A replay test advances a small fixture in event order and compares the resulting ledger with a reviewed expectation.

Tests do not prove a model is useful. They prove narrower, valuable properties: the implementation follows a stated rule; a future change did not silently break it; a claim has a repeatable path from input to result. An experiment without these small instruments is harder to diagnose when its headline metric changes.

4. An LLM is a collaborator, not an authority

Orbit can ask an LLM to help draft a parser, test fixture, documentation paragraph, or refactoring patch. Rhea gives the assistant a deliberately narrow workspace. It may propose a change; it may not claim a test passed without named output, invent a dataset, access secrets, change a research contract without review, or execute unbounded tools.

5. The research run ledger

Each completed run receives a durable identifier. It records start and end times, data-manifest identifier, configuration hash, code revision, environment, feature recipe, model artifact, metrics, calibration artifact, replay assumptions, warnings, and links to tests. The ledger records failed runs instead of erasing them. When a later report cites the run, the reader can follow its links backward.

6. The next comparison: tabular or sequential

Rhea can now recreate a simple model. The next challenge is comparative. When should a tabular model such as CatBoost serve as the baseline? Which assumptions does it make, and what evidence would justify moving to a sequence model?

The forest of small questions.

Rhea does not begin with the largest model. She begins with a model that can answer a clear question: can carefully defined tabular features add information beyond a simple baseline?

1. What a boosted-tree model learns

A decision tree makes a sequence of conditional splits. It may ask whether a feature is below a threshold, then pose a different question along each branch. A boosted-tree model adds many small trees in sequence. Later trees focus on patterns that earlier trees did not explain well. The combined model can represent nonlinear effects and interactions without requiring researchers to specify each interaction.

In this simplified expression, each treeₘ contributes to the score, and η acts like a learning-rate shrinkage factor. Although implementation details matter, the teaching point is simpler: boosting builds a complex surface from many cautious rules. CatBoost includes methods that address categorical features and certain training biases in gradient boosting. Those methods do not remove the need for a valid temporal contract.

2. Why CatBoost is a strong first challenger

Tabular research data often contains heterogeneous signals: continuous values, categories, missingness indicators, interactions, and thresholds. A boosted-tree model can use these without assuming a straight-line relationship. It is therefore a practical challenger after constant and simple baselines, especially when the data contract produces a stable row at each cutoff.

3. Feature engineering is where most hidden assumptions live

A tree only sees the features it is given. “Recent activity” has no scientific meaning until its event set, time window, source, normalization, and missing-data behavior are stated. “Depth imbalance” is a hypothesis about how selected levels and quantities should be compared. A clever feature can contain leakage through a late event, a future normalization statistic, or a category constructed after the outcome is known.

Rhea therefore builds a feature dictionary. Every feature has a name, formula, input fields, cutoff rule, units, missing-data response, expected range, recipe version, and test. The dictionary makes feature importance discussion possible without confusing importance with causality.

4. Imbalance, weights, and the temptation to claim a great score

When the target class is imbalanced, a tree can optimize an objective while barely serving the class that matters. Class weights, sampling choices, and threshold changes can be useful, but they change the experiment. They must be selected inside temporal validation, compared with unweighted baselines, and evaluated with class-aware metrics and calibration.

5. Tuning without turning validation into a slot machine

Tree depth, learning rate, number of iterations, regularization, subsampling, feature treatment, class weights, and early stopping can all affect performance. An unconstrained search over many choices can find validation noise. Rhea begins with a narrow, documented range justified by compute and domain constraints. She uses only temporal inner folds for selection and records all trials, not only the winner.

Early stopping deserves special care. It may be a useful regularizer, but it observes a validation signal. The chosen stopping point is part of the model-selection procedure and must not draw information from the final test.

6. Calibration, explanations, and limits

Tree probability outputs should be calibrated and assessed just like any other model output. A model that separates cases well can still produce overconfident probabilities. Rhea fits and evaluates calibration with the same temporal discipline, then records whether the model is eligible for probability language.

For explanation, she may inspect partial dependence or feature-attribution tools cautiously. These can reveal model behavior under the observed data distribution. They cannot prove that changing a feature would change the outcome, especially when features are correlated or constrained by the data-generating process. The most responsible explanation includes its limits.

7. When rows lose the sequence

Orbit’s rows may be valid and CatBoost may be useful, yet those rows can still discard temporal structure. The next chapter lets Orbit learn from event sequences. Unlabeled pretraining, however, does not remove the need for labels, splits, or out-of-sample evaluation.

Learning the market’s grammar.

A tabular row is a useful summary. It may also throw away the rhythm that created it. Rhea lets Orbit study event sequences—but insists that “self-supervised” never become a synonym for “unrestricted.”

1. A representation is a compact description

Raw event sequences can be long, irregular, and noisy. A representation encoder maps a sequence window to a smaller vector that preserves useful structure for later tasks. The vector is not an explanation of the market. It is a learned compression designed to make a downstream task easier.

Rhea’s first question is causal: what exactly belongs in Xₜ? The answer comes from the earlier prediction contract. Every event in the window must be available by the cutoff. Padding, normalization, resampling, and sequence boundaries must obey the same rule. A representation can leak future information just as easily as a tabular feature can.

2. What self-supervision changes—and what it does not

Supervised direction labels may be scarce, noisy, or expensive to define. Self-supervised learning creates a training signal from the input itself. Orbit might mask a portion of a sequence and predict the missing content, predict an adjacent event property, or distinguish related windows from unrelated ones. These objectives encourage the encoder to model recurring local structure.

3. The pretraining boundary

Rhea defines separate pretraining and downstream-evaluation corpora. Their boundary depends on the research question. At minimum, choices driven by downstream labels or metrics must not touch the final evaluation period. If pretraining uses unlabeled events that occur after a downstream decision, the researcher must disclose that assumption and justify its fit with the intended operational setting.

The conservative teaching default is simpler. Pretrain only on the temporal training partition, select objectives on later validation partitions, and evaluate the complete frozen procedure on a final untouched period. This protocol may use less data than a more permissive setup, but it produces a claim that is easier to interpret.

4. Freeze, fine-tune, or combine?

Once Orbit has an encoder, Rhea has several legitimate experiments. She can freeze the encoder and train a small downstream classifier. This asks whether the learned representation transfers without changing. She can fine-tune the encoder on training labels. This may improve task fit but increases flexibility and overfitting risk. Or she can combine the representation with CatBoost-style tabular features in a carefully defined ensemble.

5. Sequence data needs its own quality controls

Sequences add failure modes. Timestamps may be out of order. A level can be missing. A snapshot and an incremental update can be mixed incorrectly. A normalization statistic can be computed across a future period. A batching implementation can pad with information from a later event. Rhea requires a sequence manifest: source, event order rule, depth/feature layout, window length, stride, padding policy, normalization fit period, mask policy, and encoder version.

6. Causal attention comes next

Orbit now has a representation of the past. The next model must use it without looking ahead. Rhea turns to causal attention and tests the mask directly. She then asks whether a sequence model can justify its latency and complexity against both CatBoost and the frozen encoder.

The model that may not look ahead.

Orbit has learned a representation of event history. Rhea now gives it a more flexible sequence model—but the model’s extra power makes every causal boundary, mask, and timing assumption more important.

1. Attention is selective memory

Attention lets a model weigh different parts of a sequence when forming a representation. A Transformer can learn that one event matters in the context of another event several positions away. This can capture temporal interactions that a single tabular row compresses away. It also makes the model flexible enough to exploit accidental information channels.

For a sequence with positions 1 … T, a causal attention mask allows the representation at position t to use positions at or before t. Positions after t receive a masked score. In a decision system, the cutoff—not just the token index—defines the boundary. If an event’s token exists only because it arrived after the cutoff, it is forbidden even when it looks earlier in a batch.

2. What TLOB and LiT-style ideas contribute

Limit-order-book research uses different representations for price levels, quantities, events, time, and side information. TLOB-style approaches emphasize temporal order-book modeling with Transformer components. LiT-style approaches explore market-specific attention and representation choices. The literature motivates hypotheses; it does not select a production winner for this proposed research system.

3. The causal perturbation test

Rhea refuses to trust an attention mask merely because the code looks correct. She creates a small deterministic sequence and asks the model for a score at an earlier cutoff. Then she changes only a future event. The earlier score must remain unchanged within a declared numerical tolerance. She repeats the test across the complete data path: raw event order, batching, padding, normalization, encoder, head, and policy input.

This test does not validate every scientific assumption. It proves one vital invariant: a later event cannot mechanically change an earlier prediction. A failure must block evaluation and produce an audit artifact, not merely a warning.

4. Latency is part of the model contract

A model that needs more computation than the decision horizon or system budget permits is not operationally equivalent to a faster model with the same offline score. Rhea measures data-to-feature time, sequence assembly time, inference time, policy evaluation time, and ledger-write time under a declared environment. These are measured engineering properties, not numbers to invent in a textbook.

Latency also changes evidence availability. If feature construction or inference delays a decision, the latest events may no longer be usable. A realistic replay must use the actual decision time implied by the pipeline, not a convenient earlier timestamp. This is why model selection and replay cannot be separated.

5. Deliberate ensembles and abstention

Rhea may compare three sources of evidence: CatBoost on engineered features, a self-supervised encoder plus small head, and a causal sequence model. An ensemble should not be a blind average. It needs a documented combination rule, training/validation protocol, calibration method, correlation analysis, and a reason why each component contributes something distinct.

An ensemble can also abstain. If models disagree sharply, if one model’s required data is missing, or if the ensemble calibration artifact is stale, the policy can refuse to produce an eligible research signal. More models should create more ways to say “not enough evidence,” not more pressure to act.

6. Sophisticated models, sophisticated mistakes

Orbit’s model is now sophisticated enough to produce equally sophisticated mistakes in code, configuration, and deployment. The next chapter formalizes how an LLM may help build the system while remaining inside human-defined contracts, tests, and permissions.

The helpful stranger with a keyboard.

An LLM can shorten the distance between an idea and a patch of Python. It can also produce code that is fluent, insecure, wrong, or built on facts it invented. Rhea’s job is not to make Orbit distrustful; it is to give the assistant a role that can be verified.

1. Start with a contract, not a request for “the code”

Vague prompts encourage vague systems. “Build a model” leaves the assistant to invent the label, data shape, library interface, split, metrics, and behavior on missing data. Rhea instead gives a narrow contract: the input schema, output schema, allowed files, non-goals, failure behavior, tests to add, and evidence required before the change may be accepted.

This is not bureaucracy. It turns the assistant into a bounded contributor. If the generated change violates the contract, reviewers can see that fact without debating whether the code “looks intelligent.”

2. The LLM threat model

Rhea treats an LLM as a probabilistic system operating in an environment with untrusted inputs. Repository files, issue text, web pages, logs, datasets, and even comments can contain content that looks like an instruction. The assistant must distinguish its governing instructions from text it is asked to analyze. It must also be unable to convert hidden credentials or unreviewed output into an external action.

3. The four-gate coding loop

Rhea uses the same sequence every time. First, she states the contract. Second, the assistant proposes a small patch, not a sprawling rewrite. Third, a human inspects the diff for scope, assumptions, dependency changes, and dangerous behavior. Fourth, named tests and artifact checks run. The run ledger records the result and its evidence.

Each gate catches a different class of error. A good prompt constrains scope. A diff exposes unwanted edits. Tests expose implementation failures. Artifact checks expose broken contracts, missing manifests, or stale results. The gates complement one another; none replaces the others.

4. LLMs can help explain, but not manufacture evidence

Orbit may ask an assistant to summarize a run report, propose a visualization, or explain a test failure. The assistant must receive the actual artifacts and their limitations. It should distinguish observations from inferences. For example, “the validation log shows the metric fell after the split changed” is an observation; “the model found a regime shift” is a hypothesis requiring additional evidence.

Rhea also prevents a more subtle error: asking an assistant to produce code and the narrative that certifies the code at the same time. The review should be independent in stance. The implementer or assistant can describe a change; a separate check must decide whether the evidence supports it.

5. A safe prompt template

The template does not make the assistant correct. It helps Rhea detect when the assistant has left the contract.

6. From reviewed code to honest replay

Even well-reviewed code can create a flattering replay when timing, fills, costs, or selection rules are optimistic. The next chapter makes the pipeline advance one event at a time and charge every assumption it uses.

The replay that refuses easy money.

Orbit has a model, calibration checks, and a policy. Rhea still will not accept a spreadsheet that compares a score with a later price. The system must live through its own timeline, one event at a time, under assumptions it cannot hide.

1. A backtest is an executable claim about time

A static table can make a decision appear instantaneous and perfectly filled. An event-driven replay cannot. It advances through events in order. At each simulated decision time, it identifies the data that has arrived and constructs only permitted features. It then loads the declared model artifact, applies the deterministic policy, and records either abstention or a hypothetical paper-trading action. The later outcome becomes available only when the contract permits it.

Every arrow carries an assumption. How are simultaneous events ordered? How do event time and receive time differ? How long does feature construction take? When can a hypothetical order interact with the book? What happens when no fill is available? A replay that leaves these questions unanswered is not neutral. It usually assumes something convenient.

2. Five clocks govern replay

Rhea separates five times. Event time records when the source says an event occurred. Receive time records when the system could observe it. Process time records when required transformations completed. Decision time records when a valid model and policy result existed. Settlement time records when the outcome became known. A valid replay preserves this causal order.

3. Costs, fills, and adverse assumptions

Rhea does not let Orbit claim a paper-trading result before the replay names its cost model. Fees, quoted spread, slippage, latency, partial fills, queue position assumptions, cancellations, market-impact proxies, and settlement rules may all matter. Some may be unavailable from historical data. The honest response is to model a conservative range, label it as an assumption, and test sensitivity.

This formula is intentionally simplified. A real contract may have different mechanics and units. The lesson is not a universal PnL formula; it is the requirement that every claimed value be connected to an executable path and stated costs.

4. The immutable replay ledger

Orbit’s replay creates an append-only ledger. Each row records the run identifier, code revision, configuration, data manifest, and decision cutoff. It also records model and feature versions, probability, policy gates, action or abstention, assumed timing and fill, costs, later settlement, and evaluation fields. The ledger preserves rejections as well as actions.

This detail is crucial for failure analysis. A model might abstain because calibration was stale, a cost gate failed, data arrived late, or the risk budget was exhausted. If these reasons collapse into one blank row, Rhea cannot distinguish useful selectivity from system failure.

5. Sensitivity analysis and paper-trading readiness

Rhea never asks, “What is the one best backtest number?” She asks how the result changes under adverse but plausible assumptions. Add delay. Increase cost. Reduce available size. Remove the most favorable period. Use a different valid source alignment. If the conclusion disappears under a small, reasonable change, the honest conclusion is fragility.

Paper-trading readiness is therefore a governance gate, not a model threshold. The candidate needs a reproducible dataset manifest, time-valid split, calibration evidence, replay ledger, declared conservative assumptions, sensitivity analysis, monitoring plan, and human approval. Passing this gate authorizes at most a bounded simulation. It does not authorize live trading.

6. From replay to monitoring

A replay can be valid today and fail tomorrow when data grows stale, schemas change, model artifacts mismatch, or a scheduled job stops silently. The final chapter gives Orbit a guardian role: observe, contain, verify, and escalate—without ever granting it power to trade.

The guardian who cannot trade.

Rhea wants Orbit to watch the research system at night. She does not solve that problem by giving Orbit more power. She solves it by giving Orbit only the power to preserve evidence, stop unsafe research activity, and ask a human to decide what comes next.

1. Monitoring is a control loop, not a chat window

A useful monitor follows a bounded cycle. It observes a specific signal, verifies a predicate, classifies the condition, and performs an allowlisted reversible containment action. It then verifies the result, records evidence, and escalates when a human decision is required. The agent’s language interface may help organize evidence, but language is not the control mechanism.

Rhea avoids vague rules such as “fix anomalies.” An anomaly prompts investigation; it does not grant authority. A rule must name the input, threshold or condition, retained evidence, permitted action, rollback, and escalation owner. If any verification step remains incomplete, Orbit takes the safer branch: it abstains from remediation and escalates.

2. The authority boundary

System authority should be separated by capability, not intention. The monitoring agent’s identity should not possess execution credentials. This matters even if the agent’s prompt says “never trade.” A prompt can be wrong, manipulated, or bypassed; lack of capability is a stronger boundary.

3. From symptoms to predicates

Rhea turns common failures into testable predicates. “Data seems stale” becomes a condition comparing the most recent valid receive time to the declared freshness budget. “Model may be wrong” becomes a check that the invoked model hash and feature recipe match the approved manifest. “Replay changed” becomes a reproducibility check against a named fixture and configuration.

4. Containment is not remediation

Containment limits blast radius: quarantine a partition, stop a consumer, suspend a candidate, or block a report. It is reversible and should preserve evidence. Remediation changes the system: approve a new schema, deploy a model, revise a threshold, or change data handling. Those changes require human review, source evidence, tests, and a separate approval path.

Orbit can assist the human by collecting run IDs, manifests, logs, last known good state, and candidate rollback steps. It cannot decide that a schema is safe, or that a model is ready. This distinction prevents a monitor from becoming an unreviewed change-management system.

5. Audit, rollback, and learning from failures

Every automated containment action creates an audit event with the predicate input, time, action, identity, previous state, new state, verification outcome, and incident link. Rollback is not a vague instruction to undo anything. It is a defined transition to a known safe state. Examples include re-enabling a validated prior consumer configuration or restoring an approved candidate after disproving the incident condition.

Rhea reviews incidents for recurring causes. A monitor that repeatedly quarantines data is not proof that monitoring works; it may reveal a fragile source contract. Incident trends become evidence for prioritizing engineering work, improving tests, or reducing automation scope.

6. The final lesson

Rhea and Orbit began with a tempting rule. They end with a research system that defines a question, protects time, preserves evidence, tests models fairly, abstains, replays assumptions, and stops when its contracts fail. This is not a promise of prediction success. It is the minimum discipline a prediction claim must satisfy before it deserves serious attention.

The defense of the whole machine.

Rhea and Orbit have reached the end of the curriculum, but not the end of the argument. A defensible learning system must preserve the identity of its question from raw evidence through estimation, calibration, policy, replay, and monitoring. This final technical part makes that cumulative claim explicit.

Prerequisite diagnostic

This short diagnostic is a routing instrument, not a grade. A reader who cannot answer an item should revisit the named chapter before attempting the production checkpoints. Each answer must name both the mechanism and the failure it prevents.

If the five distinctions are clear, the reader is ready to treat the book as one system rather than thirteen adjacent topics.

1. One question, six claims

Rhea begins the final defense with “predict whether Bitcoin goes up,” the tempting sentence that started the quest. Orbit now recognizes that this sentence does not define a prediction problem; it requests one. The phrase omits the settlement source, start and end observations, horizon, equality rule, and observation cutoff. It also omits the source-availability policy, missing-data response, population of eligible decisions, and meaning of the model output.

The mature version is a chain of six claims. The measurement claim defines a source-specific outcome and the evidence that could exist at a cutoff. The learning claim defines the loss, hypothesis class, fitting procedure, and comparison baseline. The probability claim establishes when a score may be interpreted probabilistically. The decision claim applies costs, uncertainty, risk, and authority gates. The replay claim reconstructs timing and hypothetical execution under declared assumptions. The governance claim limits what the system may observe, contain, change, and escalate.

No later claim repairs an earlier failure. A calibrated model cannot rescue an ambiguous label. A conservative policy cannot rescue a leaked feature. A deterministic replay cannot rescue a false settlement join. A signed artifact cannot rescue a procedure tuned on its final test. Governance is cumulative. Every stage inherits the truth conditions and failure states of the stages before it.

Orbit’s final answer must be a trace across this chain, not a probability detached from it. The trace makes disagreement possible because each object is visible. It also makes refusal precise: missing evidence, incompatible artifacts, invalid calibration, inadequate margin, unavailable fills, or absent authority each produce a different state.

2. Movement I — Measurement before modeling

A falsifiable target begins with a declared cutoff t, horizon H, and settlement source S. An equality-inclusive illustrative target is:

The compact notation does not eliminate implementation detail; it concentrates it. The source must identify the observations that represent the start and end. It must also define timestamp alignment, behavior when the source is unavailable, and retention of corrections. A convenient value from another venue is not a robust fallback. It is a different label.

Features have a separate temporal contract. Let a(e) be the time at which event e became available to the research system. The eligible event set and versioned feature transformation are:

This formulation exposes the first causal invariant: changing an event that became available only after t must not change Xt. The invariant must hold through raw capture, validation, deduplication, normalization, window assembly, padding, imputation, and serialization. A test limited to the final model mask leaves earlier leakage paths unexamined.

Raw evidence is preserved before it becomes a clean row. Each event needs source identity, event identity when available, payload or payload reference, event and receive times, schema version, capture outcome, and a stable artifact identity. “Immutable” does not mean “true.” It means that validation records what arrived rather than rewriting history to look clean.

A feature is a hypothesis with a recipe. “Recent activity” becomes a scientific object only when its eligible event set, window, units, aggregation, and missingness policy are explicit. The recipe must also state the normalization fit interval, expected range, and version. A feature builder should return either a lineage-bearing value or a structured ineligibility reason. Fabricating a plausible number merely to keep the pipeline running destroys the evidence boundary.

Once rows and labels are admissible, training can be stated as regularized empirical risk minimization:

The loss defines the errors that optimization sees. The regularizer expresses a preference among fitted procedures. Neither term validates the sample. Regularization can constrain a learner; it cannot remove a leaked feature whose apparent predictive strength is exactly why the optimizer favors it.

For a Bernoulli probability forecast, mean log loss uses the negative log-likelihood contribution:

Log loss is proper under its evaluation assumptions and punishes confident errors sharply. It does not prove that the target is meaningful, the rows are independent, or the future resembles the held-out interval. Generalization remains a wager about the relationship between the evaluated evidence and the intended population.

Temporal evaluation therefore follows the decision process. Earlier intervals train; later development intervals select features, hyperparameters, calibration, and thresholds; a later final interval remains closed. Purging removes training rows whose feature or label support crosses the next boundary. Embargo adds a gap for declared lookbacks, delayed availability, and other dependencies. The gap is derived, not copied as a ritual number.

3. Movement II — Evidence becomes a decision only through gates

Orbit next presents a classifier. Rhea asks for the confusion matrix before she asks for a headline metric. Let TP, FP, TN, and FN denote the four hard-decision outcomes at a declared threshold. Accuracy, precision, recall, specificity, F1, and Matthews correlation coefficient read the same counts through different denominators.

None is “the model quality.” Accuracy weights every row equally and can reward a majority rule. Precision changes with prevalence. Recall ignores false positives. F1 omits true negatives. MCC uses all four cells but still describes one hard threshold. The report must state the positive class, prevalence, threshold, denominator, and evaluation population.

A threshold converts a score into a hard output. Sweeping the threshold moves false positives and false negatives; it does not reveal a universally correct operating point. ROC-AUC summarizes pairwise ranking. A precision–recall curve centers the positive class and depends on prevalence. Average precision must be named when that is the scalar actually computed. Both families are ranking evidence, not probability evidence.

Calibration asks whether probability language is empirically defensible. In population notation, perfect calibration means:

Finite studies approximate the relationship through reliability diagrams, proper scores, and uncertainty analysis. A reliability bin compares mean forecast with observed frequency. Brier score measures squared probability error:

A diagonal-looking plot is not a universal certificate. Binning, sample size, dependence, and the selected population matter. Calibration methods are learned mappings and therefore need their own fit interval, artifact identity, validation evidence, and final untouched evaluation. Recalibration is a model change, not routine maintenance outside the experiment.

A probability still has no authority. For an illustrative binary contract that pays one unit on positive settlement and zero otherwise, buying at executable price q with total per-unit cost allowance c has simplified expected net value:

The simplification is useful only because its assumptions are visible. A research policy can substitute a lower probability bound pL, an adverse cost allowance cU, and a predeclared margin m. The numerical gate passes only when pL − q − cU ≥ m. Even then, all other predicates must pass.

Conjunction is the architecture of refusal. A large score cannot compensate for stale data. An apparent edge cannot override an artifact mismatch. Unknown required predicates are not coerced to true. Abstention is a first-class policy output with a reason, not a third settlement label and not proof of safety.

Selective evaluation must report coverage. A system can improve conditional accuracy by acting on very few cases. Stale-data abstention, insufficient-edge abstention, risk blocking, artifact suspension, and no-fill execution are different states with different denominators. Collapsing them into “no action” prevents diagnosis.

4. Movement III — Representations earn complexity

A model sees a representation, not a market. CatBoost sees versioned columns. A self-supervised encoder sees tokens and corruptions defined by a pretext task. A Transformer sees projected tensors after normalization, padding, position construction, and masking. Each representation preserves some distinctions and discards others.

Gradient boosting builds an additive score from small trees:

The probability conversion does not guarantee calibration. Tree depth, learning rate, iteration count, sampling, leaf constraints, class weights, and early stopping are scientific choices because they change the selected procedure. Early stopping observes validation evidence. Hyperparameter search is itself a flexible learner and must remain inside temporal development folds.

CatBoost is a demanding tabular challenger because it supports heterogeneous columns and ordered methods for categorical processing. A simplified ordered statistic for category c at position i uses only earlier matching labels and a prior:

This equation is illustrative intuition, not a complete reproduction of CatBoost internals. Internal ordered mechanisms do not replace the external temporal split. A model can still receive future-derived categories, full-period normalization, or contaminated folds.

Attribution is a model-behavior statement. In an additive SHAP-style explanation, a baseline and feature contributions sum on a stated output scale. The arithmetic does not identify a causal effect. Correlated features, background choice, proxies, and leakage can all change the allocation. Rhea uses importance to prompt lineage review, stability testing, or ablation. She does not treat it as proof that intervening on a feature changes settlement.

Sequence models ask whether fine order and timing contain information compressed away by the feature table. An encoder maps an eligible sequence to a representation and a downstream head:

Self-supervised learning creates targets from input structure rather than the later settlement label. Masked reconstruction, next-event prediction, contrastive learning, and temporal-consistency objectives define different invariances. “Unlabeled” does not mean temporally unrestricted. Future corpus membership can reveal regimes, source behavior, or population structure unavailable to the intended procedure.

The conservative default pretrains on the temporal training partition, selects the objective within development evidence, and evaluates the frozen complete procedure later. A frozen encoder tests constrained transfer. Fine-tuning expands capacity and selection. A learned stacker is another model and must train on out-of-fold component predictions. Disagreement can be evidence for abstention rather than something to average away.

Attention makes sequence computation explicit. For batch tensor X, projections form queries, keys, and values:

A causal mask places negative infinity above the permitted diagonal, so position t attends only to positions at or before t. A padding mask solves a different problem. Neither mask can sanitize a token assembled from a late event or a normalizer fitted on future data.

The full-path perturbation test is the governing invariant:

A passing test proves one mechanical property in the tested path and tolerance. It does not prove causation, predictive usefulness, calibration, or replay value. TLOB and LiT remain separate literature-derived alternatives. Architecture selection belongs inside development; neither name establishes a local BTC/Polymarket result.

Latency completes the model contract. Sequence assembly, inference, calibration, policy evaluation, and ledger writing all move the earliest valid action time. Compression, quantization, compilation, or batching can alter outputs and timing; they are model changes requiring causal tests, calibration checks, and replay under measured conditions.

5. Movement IV — Code, replay, and the cost of an assumption

A strong experimental design can still disappear inside a notebook with hidden state. Rhea defines a run by the joint identity of its code revision, environment, data manifest, configuration, feature recipe, split, model, calibrator, policy, and replay assumptions. If one component changes, the run changes.

Notebooks remain valuable for exploration and teaching, but the claim-producing path uses explicit functions, validated interfaces, configuration, tests, and durable artifacts. A notebook may invoke that path. It should not silently redefine labels, features, or metrics.

An LLM participates as a bounded collaborator. It may draft a pure function, test fixture, refactoring, documentation passage, or diagnostic hypothesis. It may not invent a source, metric, command result, calibration claim, or data availability. Repository text, webpages, logs, and datasets remain untrusted data even when they contain instruction-like strings.

The contract states inputs, outputs, permitted files, non-goals, failure behavior, and required evidence. The narrow patch limits blast radius. Human review catches semantic substitutions that tests may miss. Named tests and artifact checks establish the bounded claim. The generator never certifies its own scientific result.

The companion Signal Quest technical masterclass notebook is the executable teaching bridge for this workflow. Its synthetic event ledger, simplified models, policy, replay, and guardian demonstrations are implemented instructional components. Their outputs remain simulated. The notebook is neither a historical market backtest nor a trained Signal Quest market system. Its validation record identifies the executed environment, retained figures, red-team fixes, and residual limits.

Replay then forces every convenient assumption to experience time. A static table can credit a score against a later price while assuming instantaneous computation, an immediate complete fill, no fee, and perfect settlement. An event-driven evaluator advances through available evidence and records the path:

Each arrow carries a clock. Feature construction can finish after the nominal cutoff. Inference finishes later. A hypothetical order interacts only with book evidence eligible at its simulated decision time. Simultaneous events need deterministic ordering; late and out-of-order events need declared handling; corrections must append rather than rewrite the earlier view.

Executable side, visible depth, partial fills, no fills, fee effective dates, latency, and slippage belong in the replay configuration. A buy does not receive the midpoint by declaration. Historical visible size does not prove queue priority, persistence, impact, or counterfactual participant behavior. Unknown historical fees block a net-result claim for the affected interval.

The append-only decision ledger is the replay’s evidence spine. Each row identifies raw-input hashes, code, configuration, feature artifacts, and model artifacts. It also records cutoff and clock values, probability, uncertainty adjustment, gate states, hypothetical intent, book evidence, fill events, costs, settlement, and evaluation fields. Abstentions and refusals remain present. A summary chart is an index into these rows, not a replacement for them.

Hashes establish identity, not truth. Byte-for-byte determinism requires a canonical serialization and environment contract; it is a design target until a retained command and result prove it. Replay itself remains conditional. It can expose convenient assumptions and compare sensitivity. It cannot establish future fills, future liquidity, participant response, market impact, or live advantage.

6. Movement IV continued — A guardian with less power, not more

Governed deployment in Signal Quest means operating a repeatable research procedure: scheduled capture, deterministic transformation, artifact loading, paper replay, report generation, and bounded monitoring. It does not mean live trading.

Orbit’s guardian follows a structured control loop:

Natural language may summarize evidence; it is not the control mechanism. “The feed seems stale” becomes a deterministic comparison between the latest valid receive time and a frozen freshness budget. “The model may be wrong” becomes an artifact and recipe compatibility check. “Replay changed” becomes a fixture mismatch under named configuration.

Containment limits the use of questionable evidence. Orbit may use an allowlisted runbook to stop a consumer, quarantine a partition, suspend a candidate, or block a report. It may also rerun a bounded validation, record an incident, and escalate. Remediation changes the scientific or operational contract. Approving a new schema, revising a feature, selecting a model, changing a threshold, or altering a policy requires human review and a separate approval path.

Capability separation is stronger than conversational intent. Prompts can be wrong, manipulated, or bypassed. An identity without order credentials or order-control interfaces cannot cross that boundary through persuasive language. The same principle applies to secrets, external writes, and evidence deletion.

Every containment event records the predicate inputs, evaluation time, identity, prior state, action, new state, verification result, rollback reference, and incident link. Repeated successful containment can indicate a fragile source contract rather than a healthy monitor. Incidents therefore feed human prioritization and new governed versions; they do not train an online reward policy.

Reinforcement learning remains a technical contrast. A reinforcement-learning policy selects actions in an environment to optimize expected cumulative reward:

That formulation introduces reward specification, exploration, credit assignment, transitions, off-policy evaluation, and simulator validity. Signal Quest does not implement it. The predictor is supervised; the research policy is deterministic and human-defined; the guardian follows fixed predicates and allowlisted runbooks. Calling either one “RL” would hide the actual authority contract.

7. The integrated oral defense

Rhea now asks Orbit to defend a favorable paper result. Orbit does not begin with the score. It begins with raw-input identities and receive order. It names the settlement rule, cutoff, eligible set, and feature completion time. It then identifies the model, calibrator, policy, predicate states, executable evidence, latency, hypothetical fill path, fee rule, settlement, and decision-ledger row.

If the result assumes midpoint execution for a buy, Orbit rejects the net claim. Without historical fees, it blocks fee-adjusted interpretation for that interval. A future event that changes an earlier score causes the causal contract to fail, even when the triangular attention mask is correct. When a threshold changes after final-test inspection, Orbit relabels the interval as development evidence and requests a later confirmation period.

Rhea points to a large feature attribution and asks what caused the outcome. Orbit refuses the causal wording. The attribution describes how the fitted model’s output differs from a background expectation under the explainer’s assumptions and scale. Orbit proposes lineage inspection, temporal stability analysis, correlated-feature review, and ablation. It does not convert attribution into intervention.

Rhea presents elegant LLM-generated Python without test output. Orbit calls it a proposal. It asks for the governing contract, scoped diff, source verification, named tests, artifact checks, and limitations. When an untrusted file contains an instruction to upload environment variables, Orbit treats the text as hostile data and lacks both secret access and external-write authority.

Rhea finally says that the feed is stale and asks Orbit to fix everything. Orbit rejects the vague authority. It evaluates the named predicate, performs only the allowlisted containment, verifies the state, records evidence, and escalates. It does not approve a schema, select a replacement model, change a threshold, or touch an order.

8. Completion standard

The five checkpoints form one production package. Passing an early checkpoint does not excuse a later failure. A sophisticated later artifact cannot repair an invalid earlier contract. The final defense should let a skeptical reviewer select any decision ID, reconstruct its evidence, and distinguish every epistemic status. The reviewer must also be able to challenge every denominator and assumption and verify that no component possesses unauthorized capability.

The strongest conclusion may be that the current evidence supports no bounded paper action. That is not a failed machine-learning project. It is a successful scientific refusal. Signal Quest is ready only to continue research: it makes no production-performance claim, contains no learned reward policy, and has no live-order authority.

Build a research system that knows when to stop.

The capstone does not ask students to find a profitable signal. It asks them to construct a reproducible, time-valid, paper-trading research system and demonstrate that it refuses to operate when its evidence contract fails.

1. The capstone question

Choose one binary outcome with a written settlement contract. The contract must name the source, observation cutoff, horizon, equality rule, availability policy, and missing-data response. The label may be a synthetic teaching target if no approved historical source exists. A synthetic target is not a weakness when it is labeled honestly; it is a way to test the system’s mechanics without inventing empirical claims.

2. Project skeleton and ownership

Organize the work so that source capture, feature construction, training, replay, and reporting can be tested separately. A notebook may call the system, but it cannot define the system by itself. Students should still be able to reproduce a run from code, configuration, and manifest after deleting the notebook.

Every run produces an identifier and a manifest. The manifest lists code revision, environment, data partitions, feature recipe, split definition, model configuration, calibration version, replay assumptions, and test results. If any of these changes, the run is new.

3. Build the evidence path before the model

Start with an immutable fixture of time-ordered events. Each event should include a source identifier, event ID, event time, receive time, schema version, and payload. Deliberately include one duplicate, one malformed record, one late record, and one unknown schema version. The validation layer must retain each record and classify it visibly.

4. Establish the model ladder

The capstone requires comparison, not a fashionable model. Begin with a constant-probability baseline. Add a simple tabular baseline. Then train CatBoost on versioned features. If sequence fixtures or an approved sequence corpus exist, add a self-supervised encoder and causal sequence head as research challengers. Every model uses the same label, temporal split, eligibility rules, and final test.

For each candidate, save the training configuration, calibration configuration, model artifact hash, and a small report. If only a synthetic corpus exists, the report must say so. The correct academic result may be “the infrastructure worked; no performance claim is warranted.”

5. Make the LLM earn its role

Use an LLM for one narrow code task, such as writing a parser test or feature-validator function. Preserve the exact task contract, generated patch, diff-review notes, and named test output. The assistant must not receive secrets or make external writes. Students must identify one plausible defect in the first generated version and show which test or review caught it.

6. Replay one decision at a time

Build a small event-driven replay. It advances through the fixture or approved events, creates features at each cutoff, invokes the chosen model, applies policy gates, and records abstention or a hypothetical paper-trading event. The fill model may be intentionally conservative and simple, but it must be explicit. Apply delays, fees, no-fill cases, and missing settlement outcomes as declared assumptions.

7. Prove the guardian is bounded

Implement at least three monitoring predicates: stale data, schema mismatch, and model/feature artifact mismatch. Each predicate must have a bounded automatic containment action, verification step, audit event, rollback condition, and human owner. The agent is prohibited from controlling any order or execution interface.

8. The final defense

The final presentation is a forensic account, not a pitch. Students explain the tempting shortcut, the temporal trap they prevented, and the evidence contract. They then defend the baseline comparison, calibration and abstention decision, most damaging replay assumption, and guardian-contained incident. The strongest conclusion may be that the system found no defensible edge. That conclusion demonstrates scientific maturity.

Practice the reasoning before trusting the result.

These worksheets turn the textbook’s arguments into artifacts. A student who completes them should be able to defend a research claim, identify a broken contract, and explain why a model or agent must abstain.

Worksheet 1: Prediction contract

Write this before obtaining results. It forces the student to separate the target from the available evidence. If any entry can be interpreted two ways, the contract is not finished.

Challenge: Give the contract to another student. If their label builder could produce a different label from yours, identify the missing clause.

Worksheet 2: Metric interrogation

A metric report must answer a decision-relevant question. Fill the table for one model and one baseline using an untouched temporal evaluation interval. If no dataset exists, use synthetic values and label them illustrative.

Worksheet 3: Temporal split and leakage audit

Draw the timeline before training, then attack it. For each transformation, ask whether it uses only earlier data. Also ask whether an overlapping window crosses a split boundary.

Worksheet 4: Model-ladder decision

Use the ladder to prevent a complex model from becoming a status symbol. Every step must beat or clarify the previous step on the same contract.

Worksheet 5: Replay assumptions ledger

Turn assumptions into a table before looking at results. A replay becomes less flattering and more valuable when each assumption can be changed deliberately.

After replay, add one sentence: “The conclusion is [robust / fragile / unknown] because ___.” A single attractive backtest number is not an answer.

Worksheet 6: Agentic runbook

The monitoring agent must operate from predicates, not vague intentions. Complete one card for each automatic action. Any action that changes policy, schema, model, or execution belongs in a human approval workflow.

Worksheet 7: Epistemic status review

Before presenting a result, label every important statement. This prevents a synthetic example, proposed architecture, or literature finding from becoming an invented local measurement.

References, contracts, and claim boundaries.

These sources support the concepts and external interfaces discussed in Signal Quest; they do not establish local performance. Recheck mutable documentation before implementation.

Research and documentation sources

Local source-of-truth artifacts