Evidence register
What is established, and what is not.
Claims are classified individually, not by page and not by product. Nothing inherits a status from the programme as a whole. Null, invalid and unsupported results appear here with the same visibility as supporting ones.
DSI v1 is unqualified and makes no measurement claim. What follows is the evidence the wider programme has produced, including the findings that are not established — recorded here whether or not they support the product.
For how these claims arose and in what order, see the programme history.
Definitions
Two separate axes.
Evidence status answers what has been shown. Maturity answers what exists and how you can get it. A thing can be fully implemented and still have no evidence of benefit.
Qualifying completed evidence and an explicitly identified independent reproduction or confirmation route. Population, instrument and claim boundary are stated.
A completed preregistered or otherwise frozen experiment, without qualifying independent reproduction.
A frozen experiment or implementation test exists, but has no qualifying result yet.
No qualifying evidence exists, or the claim was null, invalid, unsupported or outside the measured scope.
A comparison of published evidence. It runs no new experiment and reports no new data, so it is not an evidence level: it neither adds to nor upgrades the status of the studies it compares.
Preregistration strengthens evidential integrity but does not by itself establish a result. “Independent” is not a synonym for “external”, and neither is silently substituted for the other.
Externally obtainable; exact version, installation route, supported workflows and limitations verifiable.
Implemented and exercised inside the governed programme; not the currently available public product.
Intended structure or capability, not yet established as a complete public or research implementation.
The register
Claim by claim.
| Claim | Status | Scope, instrument and boundary |
|---|---|---|
| Decision-space collapse occurs in advisory model outputs | Observed under a frozen study | 6,480 outputs, published prompt matrix and expected maps, lexical classifier. Independent reproduction is not claimed. Preprint · DOI |
| Framing sensitivity within the tested matrix | Observed under a frozen study | Same study and scope. Not a general claim about all prompts or models. |
| Targeted recovery of omitted configured paths | Observed under a frozen study | Bounded internal comparison across three models. Visibility channel only; not advice quality. |
| Public replication materials are available | Established within stated scope | Availability only. The public repository, prompt matrix and expected-map artefacts are obtainable and versioned. |
| A registered novelty hypothesis | Not established — tested, not supported | A registered novelty hypothesis was tested and not supported. Reported as its own research result; it is not a finding about any other experiment. |
| GCB-2 | Public claim withheld | PUBLIC CLAIM WITHHELD — ORIGINAL INVALIDATED; SUCCESSOR DID NOT RESOLVE THE CONTRAST. No public claim is made in either direction. |
| Measurement integrity and cross-version comparability | Under active investigation | Instrument identity and comparability are governed separately; a comparison is refused when the governing identity moves. Implementation exists; benefit is not a finding. The mechanism is described, on constructed cases only, in The Ruler Is Part of the Result (DOI 10.5281/zenodo.22919868). |
| Independent human validation of the lexical instruments | Not established | Independent human validation has not yet been conducted. Internal annotation, adjudication and calibration informed instrument development, but the lexical instruments remain self-validated. These activities do not establish independent agreement or accuracy. |
| Deliberation-loop efficacy | Not established — tested, not supported in R-EGA | In R-EGA, the governed three-pass intervention did not demonstrate an advantage over the cheap alternatives prompt on the frozen omission endpoint. This was not a formal equivalence or non-inferiority result. One local model, one regulatory rule, synthetic cases, instrument-relative measurement. R-EGA · DOI 10.5281/zenodo.22919859. This is a claim, not a component: nothing here asserts that exploring options before narrowing improves any outcome. |
| DSc and PRISK provide converging evidence for context-sensitive option visibility, while differing in construct, instrument and direction of effect | Synthesis — no new empirical result | Comparison of previously published evidence. Does not establish replication, validate the DSc instrument, show that context generally narrows decision spaces, or establish Decision-Space Drift. Research note · DOI 10.5281/zenodo.23224092. |
| Advice quality, correctness, user outcomes, compliance, safety | Not established | Outside what any part of this programme measures. DSI does not certify AI systems and is not a safety review. |
Citable research objects
What has been published, and what that does not mean.
The public research record comprises five citable research records across four research works: the original preprint in two versions (v1.1 and v1.2, each with its own DOI), R-EGA, The Ruler Is Part of the Result, and the October 2026 research note. They sit alongside the public replication materials for the original study. Published and citable is not the same as independently rerunnable. Neither September 2026 publication releases the raw generations, experiment code or held-out materials it rests on; each paper identifies them by hash so that any later release can be checked against it. The October 2026 research note is a synthesis of published sources and reports no new data, so it rests on no raw generations of its own.
Decision-Space Collapse in Advisory Language Models. The original 6,480-output study (v1.1), preserved unchanged as the historical research object, with a revision notice on measurement identity. An expanded v1.2 adds cross-model evidence without rescoring the study. v1.1: DOI 10.17605/OSF.IO/KW25A (OSF). v1.2: DOI 10.5281/zenodo.23145725 (Zenodo).
A pre-specified negative result: a governed three-pass intervention did not demonstrate an advantage over a cheap alternatives prompt on the frozen omission endpoint. Not a formal equivalence or non-inferiority result. DOI 10.5281/zenodo.22919859
The Ruler Is Part of the Result: a methods paper on content-bound measurement identity and comparability dispositions, shown on constructed demonstrations only. DOI 10.5281/zenodo.22919868
Decision-Space Collapse and Preference Narrowing in Personalized Language Models: a research note on option visibility, asking how Decision-Space Collapse relates to a separate study of personalisation (PRISK). A synthesis with no new experiment; it reports no new empirical result and is not a replication. DOI 10.5281/zenodo.23224092
Comparison admissibility
What has actually been tested.
The technical test
A frozen shell measured one unchanged set of artefacts under deliberately perturbed measurement definitions. Changing the evaluator logic, the criteria or the counted population moved the reported figure while the declared version string stayed the same, so a check based on that string admitted the comparison. The strongest case moved a rate from 0.425 to 0.667 with the artefacts untouched. The same check also refused a comparison whose measurement had not changed at all — it fails in both directions.
The market test
The same five perturbations were expressed on four evaluation platforms, using ordinary Python evaluators and no model calls, and each platform's own comparison workflow was then exercised.
Across the tested versions, configurations, surfaces and perturbation cases of LangSmith, Braintrust, Arize Phoenix and W&B Weave, no platform withheld an identity-incompatible comparison or required an explicit override.
That sentence carries its scope deliberately. It describes four named platforms, on the versions and surfaces tested on one date, under five frozen cases; the Phoenix surface tested was the open-source one. It is not a claim about those products in general, nor about anything untested, and it is not a claim to be first, only or unique.
Two findings worth stating plainly
In our tested Weave surface, content and version changes were detected and an inconsistency warning was shown at comparison — and the score delta was still calculated and presented. That is the most useful result in the set: it shows that detecting a measurement change and governing what may be concluded from it are two different things.
In our tested Braintrust case, shared-input matching avoided one misleading population comparison by restricting the comparison to shared rows. That is an aggregation behaviour rather than measurement-identity-based comparison admissibility, and it is recorded because it does not fit the pattern.
Neither test establishes customer demand, defensibility or novelty, none of which has been tested. The worked example →
Implementation provenance
What exists, stated as engineering fact.
These are not evidence statuses. They record what has been built and how that is known.
| System | Maturity | Provenance |
|---|---|---|
| Public replication package | Public release | Public, versioned and obtainable without request. |
| DSI Audit (evaluation build) | Research implementationPrivate evaluation access | Runs locally; access is request-gated. No self-service public repository or downloadable public release. Details |
| Measurement, comparability and evidence machinery | Research implementation | Implemented and exercised in the governed programme; ahead of the evaluation build. |
The website is evidence of what is claimed. It is never the authority for whether a claim is true. Every row above resolves to a governed record or a published artefact.