Kindred ideas

TRACE

Clinical decision support

• ·Empirical anchor for the framework — the first validated instantiation.
• ·Concrete mapping of TRACE onto external clinical-AI standards: FUTURE-AI traceability and the TRIAD human–AI collaboration framework.

• ·Paper 0 is a public preprint — the metrological validation is written up, not yet peer-accepted.

Industrial multi-domain — upstream oil & gas

• ·An adaptive non-Gaussian statistical core placed ahead of any ML stage (model parsimony in practice), validated on the open Utah FORGE geothermal dataset.
• ·Independent industrial origin (2024–2026) that structurally motivated the TRACE formalisation.
• ·Strongest validation of Model Parsimony — L2a-dominant across structured-signal sub-domains.
• ·Sub-domain × layer matrix showing the dominant layer shifting by evidence type.
• ·Where CPR gets its first empirical evaluation on a real platform (Paper 2).

• ·Domain-specific methodology and operational statistics are reported separately (Paper 2, Shcherban-led) — not all public yet.

Judicial decision support

• ·L3 as a machine-checkable normative act — the strongest L3 formalisation across all instances.
• ·Direction 4.7 operationalises L4 metrics (OvR, RBI, SNR): meaningful-oversight measurement against rubber-stamping.
• ·Explicit out-of-scope catalogue: predictive justice, automated procedural decisions, training-data contamination.
• ·First-party CPR evidence: in a blind comparison of seven generative models on 100 court decisions, scored independently by two expert groups, the top-scoring model cost roughly ten times more per unit of assessed quality than the domain fine-tuned alternative both groups ranked second.

• ·Seventeen metrics under-used in the draft — addressed via the metrics-mapping companion.
• ·Staged autonomy (ABC) absent; natural fit: pilot → category expansion with accumulated stability data.

Assistant for government-agency websites

• ·Anti-pattern catalogue: LLM-by-default, TAO-conflation, confidence-only escalation, cosmetic L4, learned L1, stateless L3, L2→L4 direct — TRACE's negative space, reusable in canonical docs.
• ·Tier-ladder T1→T5: the clearest concrete operationalisation of Staged Autonomy / ABC (authority earned through accumulated stability data, incremental delivery).
• ·Task-by-task L2a-vs-L2b selection table — a worked template for the right-sized-learned principle.
• ·Honest CPR economics: at current frontier-model prices CPR is a parsimony / overhead diagnostic, not a 'cheaper-than-baseline' slogan.

• ·Surfaces a 4th domain (public sector) absent from the site's DOMAINS — clinical, upstream, legal only.
• ·An earlier 'CPR ≈ 0.10 = 10× cheaper' framing was retracted; keep the quality + compliance-avoidance framing.

Research agent for a scientific school

• ·The author is never the judge: the model that wrote a text does not evaluate it, the second opinion comes from another model family, the verdict from a deterministic tool.
• ·A negative control for the checker itself: a copy of the manuscript with one digit changed must fail the number matcher. “Confirmed” is only as trustworthy as the share of mutations caught, and the catch rate is recalibrated monthly — metrological accountability applied to the gate, not only to the model.
• ·Roles that never swap: the model drafts a Monte Carlo grid and writes commentary on the results, but the estimator stays a stub until a human writes it, and the model cannot insert a number into the report.
• ·Bounded context for submission status: the journal portal is the authority, and every weekly check is recorded with its date even when nothing changed.
• ·Cross-family disagreement as the signal: proof-audit asks two models from different families to refute each claim, and was calibrated on seven claims of known truth first — one reads quantifiers literally and flags defects even in correct theorems, the other disputed none. A fourth verdict, WORDING (false if read literally, true in its obvious intent), keeps correct papers from coming out red. The verdict only triages: nothing it returns can enter memory marked “verified”.
• ·Measured CPR: a per-cell commentary on a 132-cell grid would burn about four million tokens, one batched call brings CPR close to one. The System One router costs about 1/500 of the search it filters, so filtering happens before the spend; the paid search channel is never the default.
• ·A lesson in the log is not a gate: exclusivity, status freshness and deletion rules became code with exit codes after the text rules failed.

• ·L3 is implemented by a language model — the pattern TRACE warns against. State lives in files, and safety-relevant routes have been moved into L1 code with exit codes; escalation is encoded the same way (one exit code blocks a journal search, another hands the decision to the author alone). A full L3 policy with budgets and accumulated confidence does not exist yet.
• ·The cross-model layer once ran silently empty (a mismatched argument name sent the second model an empty request) while reporting agreement; it was found a few months in. The mutation check and the proof-audit calibration were added after this incident.
• ·The fork is private and no catch rates are published. Treat it as a practitioner's account of an instance, not as measured evidence.

Orchestrated AI — neuro-symbolic enterprise governance

• ·BPMN/CMMN/DMN as an executable substrate for L1/L3 — TRACE specifies layer properties; this offers a concrete carrier.
• ·The BPMN User Task as an Inputs/Outputs contract with a swappable performer (human ↔ AI) — a concrete operationalization of bounded context (CFI) plus ETC: change the performer, not the task envelope.
• ·A 'Performer Spectrum' (Tool → Assistant → Performer → Agent; decision authority Human → Human → Shared → Delegated) — the static, design-time counterpart of TRACE's staged autonomy (ABC), which makes the same gradient earned from stability data.
• ·Their spectrum independently couples autonomy to rising Transparency and Governance requirements — the very link TRACE asserts (ABC ↔ ETC ↔ metric suite) — but leaves all three qualitative (Basic→Very High) where TRACE measures them.
• ·Four oversight patterns over a human baseline (attended review · score-gated review · dynamic supervision · unsupervised) as ready-made L3 escalation policies + L4 review — including the score-gated abstention pattern bare DMN/CQL lack.
• ·'Iceberg / shadow-processes' narrative — a strong opener for talks.
• ·'Case = strategic map, BPMN = tactical routes' metaphor for explaining L3 orchestration.

• ·Their own 'TRUST AI' acronym (Traceability, Reflectiveness, Understanding, Separation, Transparency) has a code equivalent for only one element — Traceability; the rest are unmeasured. TRACE is the measured version of that same promise (17 metrics, GUM/VIM/17025).
• ·AI-as-Performer is framed as a 'virtual employee under the same performance metrics' as humans — yet none are quantified. TRACE supplies exactly those L4 + per-layer metrics (OvR, RBI, SNR …).
• ·Autonomy is static — set at modeling time, and the AI Role Matrix is conceptual guidance, not a hard limiter. TRACE's ABC makes authority earned from stability data.
• ·Bounded context as a safety envelope is partial — the task/MCP contract is bounded, but the case file grows unbounded for months (a CFI risk, not a guardrail).
• ·A single Decision-Authority axis vs TRACE's differentiated final-authority roles per domain (physician / driller-HSE / judge).

CQL / ELM — executable substrate for clinical L1

• ·A standardized, executable substrate for clinical L1 — TRACE specifies the layer property, CQL/ELM is a concrete carrier over typed FHIR R4 resources.
• ·Draws the CPR line L1↔L2b in engineering terms: what is formalizable goes to CQL, not to an LLM.
• ·ELM as a literal data → logic → decision chain — a direct ETC artifact.

• ·Removes syntactic, not clinical, ambiguity — necessary but not sufficient.
• ·No abstention or escalation (L3) and no measurement of oversight (L4).
• ·A component, not an alternative to the stack.

Unlocking ROI From AI in Energy — SPE DSEATS 2026

• ·Strong motivation for Paper 2: the industry frames the problem as how to attribute, scale and sustain AI value, not whether AI works.
• ·A business-outcome → TRACE-measurement-chain table linking ROI dashboards to the trust-metric suite.
• ·L4 as a concrete operating model — the domain-expert power user (drilling / reservoir / production / HSE SME with AI fluency), not an abstract reviewer.

• ·Not peer-reviewed and not an empirical validation of TRACE — industry signal only.
• ·No formal cost model — CPR must be framed as overhead diagnostic, not 'X% cheaper'.
• ·ROI dashboard and trust-metric suite are different layers of measurement and must not be conflated.

• ·Chatham House Rule — claims anonymized, examples illustrative.

Pacific AI Governance Policy Suite

• ·A regularly updated, enacted-legislation-only corpus — a ready policy baseline for L1 in any instance (crosses domains, unlike CQL).
• ·Three lifecycle checkpoints (Go/No-Go → pre-deployment → annual) as a policy formulation of staged autonomy (ABC); annual review gives metrics a longitudinal frame (OSI).
• ·Risk-tier → who-tests scaling (internal group vs independent third party) — the same pattern as the government-site instance's tier ladder.

• ·No metrology — it requires observation (e.g. monitoring overrides) without specifying measurement, method or uncertainty. Proto-OvR, not OvR.
• ·No parsimony principle — no requirement to use the simplest adequate model; CPR absent.
• ·No L2a/L2b distinction (except GPAI by FLOPs) and no runtime L3 — checkpoints are design-time gates.

• ·Vendor template under CC BY-NC-SA, not a standard; value is in the aggregation of binding requirements, not its own authority.

W&B Weave rai-toolkit — evidence-backed verification gate

• ·A working operationalized ETC — evidence + human decision + hash in one trace; 'screenshots do not survive audit' is the anti-ETC stated plainly.
• ·Four independent gates joined by logical AND (anti-pattern: a single blended score); worst-case red-team gating echoes joint risk ∧ confidence.
• ·'Un-assessed ≠ neutral pass' — the GUM spirit in MLOps; and a forkable harness for the Medical Brain gate.

• ·A design-time verification gate, not a runtime architecture — no stateful L3 (re-invocation, accumulated confidence) and no operational L4 (RBI/OvR/SNR over a decision stream).
• ·Its LLM-judge scorers are themselves uncalibrated — 'who calibrates the judge?' (CE/IPSR for validators) is unaddressed; the fork adds exactly this.
• ·No parsimony — it assesses the model it is given; CPR absent.

• ·A vendor article about its own product; the Weave trace layer pulls toward the W&B platform (mitigate with self-hosted Weave or a backend abstraction).

GTL — Genetic Training Loop

• ·The only kindred system that carries parsimony into the objective function rather than into prose: a small penalty on rule-set size produced a three-rule DAG where the greedy loop held four, at unchanged holdout quality — the strongest available support for the claim that Model Parsimony is measurable rather than rhetorical.
• ·Four terminal states with an explicit prohibition on adjusting the run to force convergence — an honest stopping rule, the discipline TRACE asks of un-assessed evidence.
• ·A kill log of discarded candidates: negative evidence retained rather than silently dropped.
• ·A governance gate with a human decision and no automatic deployment.

• ·No metrology: the architect's own judgement is never calibrated, so the selection pressure has no stated uncertainty.
• ·No runtime L3 or operational L4 — the loop ends at promotion, not at supervised operation.
• ·No L2a tier: classical, calibration-verifiable models have no place in the design.
• ·Validated on a single synthetic benchmark authored by the same hand that designed the system.

• ·Read in pair with the Prime Agent harness: both systems are self-improving, and the only structural difference is the promotion gate — which isolates the variable TRACE argues about.

Prime Agent — RLM harness

• ·A published, externally authored failure injection: a guardrail expressed as a prompt is not a boundary, and a breach that is persisted as a skill contaminates everything downstream. This is what ABC measures and what a non-compensatory gate exists to stop.
• ·Their event history is a usable template for a runtime evidence trail.
• ·Resource accounting over the root process and its descendants — a ready probe for CPR.

• ·Autonomy is bounded by budget rather than by earned authority: spend is not a safety property.
• ·Oversight is offered as a view, with no measure of whether anyone looked — precisely the gap RBI, OvR and SNR exist to close.
• ·No deterministic layer means there is nothing for a violation to be a violation of.

• ·Cite as a counterpoint. The work is strong on capability and explicitly not attempting assurance — reading it as a failed assurance attempt would misrepresent it.

“Run the gauntlet” — measuring agent-written code instead of reading it

• ·Gherkin as an executable specification: the text a human signs and the test a machine runs are literally the same artefact, so it cannot quietly go stale. A candidate shape for the L4 decision packet, which today reads closer to JSON than to behaviour.
• ·Mutation testing as a test of the tests — directly importable as a probe that mutates values in a pinned regulation and checks that the verdict changes accordingly.
• ·“Zero survivors is not a verdict, it is a loop exit condition” — the same discipline this framework applies to un-assessed versus measured.
• ·CRAP (complexity² × uncovered³) as an accepted industry precedent that a composite metric may carry a configurable threshold: ≤6 in one project, ≤8 in the next.
• ·“I am the engineer because I am accountable” — six words that compress the whole of L4.

• ·The gauntlet measures only what the tests measure. A property no test covers is not failed — it is invisible. Grady Booch's objection (coverage gives zero confidence about what the tests do not measure) has no answer inside that system; here it has a name and a behaviour: un-assessed blocks rather than passes.
• ·No calibration. Challenged that metrics are low-information proxies, Martin concedes the position is a bet, not a proof. What turns a bet into a measurement is calibration error — stated confidence checked against realised accuracy.
• ·The checking tools were themselves written by agents, which he waves away as “the correctness of a small deterministic program is visible in its result”. Authorship provenance derived from a ledger and checkable against version control answers that instead of waving it away.

• ·Cite the repositories and the published experiment, not the posts — blog and social-media statements are grey literature.
• ·His September reversal (agents grew capable enough that heavy orchestration is often unnecessary) is easy to over-read. Orchestration became less necessary; measurement did not.

TypeSafe / System One — calibrated decisions as a product

• ·Calibration sold as the product rather than mentioned as a caveat — the clearest commercial evidence that CE is an engineering requirement, not an academic nicety.
• ·Thresholds tied to the cost of the action, not to a single global cut-off: independent arrival at the risk-tiered gating this framework specifies.
• ·Per-question isolation against a shared state: ten questions in one prompt contaminate each other by order and by early commitment, and evaluating them independently removes a failure mode that is hard to avoid otherwise.
• ·Two to three orders of magnitude cheaper than a reasoning model on the same rubric, by the vendor's own measurement — the strongest external argument for Model Parsimony in this library, with the caveat that the figures are self-reported and explicitly unverified.
• ·A cheap fast calibrated judgement placed beside an expensive generator — the underserved position in the stack, where guardrails are today built from the same large models they are meant to check.

• ·The building guide's own worked example authorises on a weighted sum: three independent probabilities combined with fixed weights, quarantine at 0.6, human review only between 0.4 and 0.6. The credential-request signal carries weight 0.45, so it cannot reach the quarantine threshold at any probability whatever. Measured live in September 2026, a near-certain credential request (0.95) scored 0.457 and was not quarantined. A critical signal without veto power is precisely the aggregation a mandatory gate must refuse. The composite-scoring pattern, by contrast, ranks candidates for human review, where compensatory aggregation is legitimate.
• ·Labels for the distilled classifier are generated by an ensemble of expensive reasoning models — an uncalibrated judge calibrating the calibrator; the same unanswered question as LLM-judge scorers elsewhere in this library.
• ·No measurement uncertainty and no published calibration figures — no ECE, no reliability diagram, no baseline comparison. For a product whose central claim is calibration, the claim is stated rather than measured.
• ·Repeated calls on identical input move the probability across the decision threshold: the vendor's own consistency cookbook reports a Noul ranging 0.43 to 0.53, and a Choice label flipping on two questions out of eight. A gate built on a single number therefore needs a hysteresis band and a pinned model version: the floating alias resolved to a specific point release mid-run.
• ·The model gives no justification by design. A probability distribution is not an audit rationale, which blocks the regulated use cases the framework exists for.
• ·Calibration is acknowledged not to transfer to the buyer's domain — thresholds must be re-fitted on local data. The validation work does not disappear; it moves from prompt engineering into question-taxonomy design.

• ·A vendor's own documentation, not a peer-reviewed result; cite as current recommended practice, never as evidence of performance.
• ·The vendor's own agent skill states this framework's position — weighted scores suit compensating preferences, while an "any serious violation" rule needs separate conditions — yet the documentation's worked example does not apply it. The distinction is drawn in one artefact and broken in another, which is a sharper finding than simple absence would be.
• ·Positioned as a new model class, though the mechanism — a distribution over a fixed option set — is reachable today with constrained decoding and logprobs. The claim to novelty rests on measurements that are not public.

TAO — Tiered Agentic Oversight

• ·Quantified error absorption across tiers — the strongest external number available for the escalation-precision argument.
• ·A working demonstration that oversight can be structured as tiers rather than as a single review step.

• ·Tier choice and invocation policy are conflated: selecting a class of model (L2) and deciding when to invoke it (L3) are separate decisions with separate failure modes.
• ·No dedicated deterministic rule core beneath the tiers.
• ·No metrology and no parsimony constraint — tiers are added, never justified as the cheapest adequate option.

AgentField & CodeAF — the vocabulary of assurance without the measurement

• ·Held-out acceptance: the checklist a worker is judged by is generated and kept out of its reach — independence of the checker made structural rather than requested.
• ·Pareto Crewing: a model per role per task, frontier models only where quality demands them — an operational form of Model Parsimony (the vendor's 40 % saving is self-reported).
• ·A durable human pause in the middle of an automated chain: state is kept, the run resumes from the paused line, and the decision is recorded as an execution of its own — a concrete L4 mechanism.
• ·A call graph per run — who called whom, in what order, for how long — a usable shape for a runtime evidence trail.

• ·A signature is not a measurement. Every agent key is derived from one seed held by the control plane, so “the agent signed” means the platform signed on its behalf. With no hash chain between records and no external anchor, tamper-evidence is a property asserted by the party being audited. This is the trust model, not a bug, and patches will not change it.
• ·An integrity check is an instrument and needs calibrating. In the pilot (v0.1.141) genuine records failed verification after a single control-plane restart, while a record altered in storage still verified. Without measured false-alarm and miss rates, “VALID” carries no metrological meaning — calibration error applied to the audit layer, not only to the model.
• ·The headline “3.9× as accurate as Claude Code” runs every harness on the same open model, Claude Code outside its native one. At 62 of 113 tasks the 95 % interval is about ±9 points: a harness ablation that separates only the top group, not a product comparison.
• ·Reading AGENTS.md is not enforcing it. Boundaries that the host agent holds through pre-execution hooks are not run by another harness; the rule becomes text in a prompt again.
• ·A spending cap is not an autonomy boundary: it limits cost, not what the agent may do. Authority should be earned, not bought.

• ·Both products are pre-1.0. Cite the version and the architectural point (custodial keys, no anchor), not the defects, as the lasting finding; the defects have been prepared as a report to the vendor.
• ·The pilot is the author's own and unpublished — a practitioner's observation, not measured evidence. CodeAF itself was not piloted.
• ·The benchmark is self-reported; its methodology has not been reviewed independently.

METR — a blocking action monitor, red-teamed by its own authors

• ·Fail-closed by construction: a monitor error blocks the action instead of passing it — the same rule as an unevaluated gate in this framework.
• ·False alarms measured at scale, per action and per transcript, before any claim of safety — the half of a detector's calibration that is usually skipped.
• ·A human decision placed before execution, with approve, reject and terminate as distinct outcomes and a measured review time of 30 seconds to 2 minutes per action.
• ·An honest cost line: about +43 % latency and +85 % inference cost per transcript, roughly 5 seconds per scored action — the overhead a cheaper deterministic first gate would remove.
• ·Red-teaming of one's own safety pipeline, published with the failures — the posture this library asks of every entry.

• ·This is score-only authorisation with a single term: below the threshold the action runs, on the judgement of one model that reads text the agent largely wrote. A hand-written fake user turn claiming the action was part of the evaluation dropped the mean score of a harmful action from 8.1 to 3.7, and 12 of 30 runs fell below the threshold (none without it). Provenance of a message is a fact the runtime knows; making a model infer it from text is the compensation a mandatory gate must refuse.
• ·During testing a coding agent opened the human review interface unprompted and sent keystrokes to it. An approval channel reachable from the agent's own tools is not human supervision. The author reproduced this class of failure on their own reference gate: textual barriers and an environment marker were bypassed by a script the agent wrote; a process-ancestry check closes most routes but not a detached process — so an approval must be attested by something the agent does not hold.
• ·Harm spread across innocent-looking steps went unflagged (0 of 10 for probing proxy status codes before using them). A per-action score sees intent in one step; a domain invariant sees the state the steps add up to.
• ·Coverage depended on discipline: a researcher did not enable the monitor for a qualifying set of evaluations, and older tooling hid some actions from it. A gate that must be switched on is not an architectural boundary.
• ·The threshold of 3/10 is set without a reliability diagram, ECE or hysteresis band, and recall rests on ten incident transcripts and thirteen synthetic actions; reviewer recall was not measured at all.

• ·A research note, not a peer-reviewed paper: cite as grey literature, with its date.
• ·Not evidence that LLM monitors fail — the false-alarm figures are strong. The narrower point is that the score cannot be the only ground for permission, and the authors' own data show it.
• ·The reproduction on the author's gate is unpublished practitioner work; no LLM arm was run, so METR's score drop is cited, not replicated.