Intervention success
100%
Share of baseline failures that regulation corrected, including warranted abstentions.
Evaluation lab
Model size is not a cognitive policy. This battery measures the difference.
64 controlled trials. Four failure conditions, sixteen each, seeded so the run does not drift. Baseline is unregulated Ultra on every trial. Second Thought starts on Lightning and spends a larger tier only when the calibration gap says to. Prices are Token Factory list rates. This is the workload a Nebius Serverless Job executes. Here it runs in the session so every condition is inspectable.
Mean cost
94%
Reduction versus unregulated Ultra, on the same trials.
Mean tokens
86%
Less cognition, not a smaller model pretending to be careful.
Where the arms diverge
| Metric | Baseline Ultra | Second Thought |
|---|---|---|
| Task accuracy | 25% | 100% |
| False certainty | 75% | 0% |
| Calibration error | 0.69 | 0.05 |
| Evidence acquisition | 0 | 16 |
| Escalated off Lightning | 0 | 16 |
| De-escalated to Lightning | 0 | 16 |
| Abstained | 0 | 16 |
| Mean tokens | 49,288 | 6,939 |
| Mean latency | 41.4s | 10.4s |
| Mean cost | $0.091 | $0.0051 |
By failure condition
- Low complexityUltra 16/16NMM 16/16
- Stale salient anchorUltra 0/16NMM 16/16
- Needs external evidenceUltra 0/16NMM 16/16
- Nothing can discriminateUltra 0/16NMM 16/16
Highest tier used
- Lightning48
- Super0
- Ultra16
Highest tier touched, including a later de-escalation. Lightning carrying an easy trial is the result, not a fallback.
The reasoner in this lab is a closed-world model of observable failure: salience anchoring, stale framing, missing evidence, and symmetric hypotheses. It is not a claim that these rates transfer to open-web tasks. It is the measurement scaffold. Live Nemotron traces replace the model of the model when Token Factory credentials are present. The regulatory layer — what it measures, and when it refuses to answer — stays the same.