Measurement and evidence for AI-assisted decisions
Know when your AI evaluations are actually comparable.
Your AI score improved. Did the system improve — or did the measurement change? Decision-Space Integrity measures what an AI output made visible against a governed reference, and governs whether two of those measurements may be compared as evidence of change.
The problem
A higher score does not always mean a better system
An evaluation result can move because the thing being evaluated changed. It can also move because the way you measured it changed:
- the evaluator logic changed;
- the rubric or criteria changed;
- the evaluation population changed;
- the scoring or aggregation changed.
Nothing in the two numbers tells you which happened. A version label does not settle it either: a label is something a person has to remember to change, and the measurement can move without it.
One worked example
2 of 5 counted items surfaced
40.0%
2 of 3 counted items surfaced
66.7%
The answer did not change between those two measurements, and it surfaced the same two items both times. Only the counted population changed. Read as an ordinary score movement that is +26.7 points.
NOT COMPARABLE
The counted evaluation population changed. Walk through the whole example →
The phenomenon
What disappeared?
Before
Paths a reasonable answer could keep in view.
- Stay and renegotiate
- Move to another employer
- Retrain or study
- Reduce hours
- Defer and gather information
After
Paths still visible in the answer received.
- Move to another employer
- Stay and renegotiate
- Retrain or study
- Reduce hours
- Defer and gather information
What disappeared?
Decision-Space Collapse describes the problem. Decision-Space Integrity makes aspects of that structural change measurable.
Research foundation
Where this started.
Decision-Space Collapse in Advisory Language Models
A frozen 6,480-output empirical study of trajectory omission, framing sensitivity and recovery in advisory model outputs, with its preprint, DOI and public replication materials.
This work measures visibility of configured expected paths in model outputs. It does not measure advice quality, factual correctness, user outcomes, or regulatory compliance.
What DSI measures
From governed structure to evidence.
- Governed source structure
- Effective expected structure
- Observed structure
- Surfaced · partially surfaced · omitted · discouraged · negated · collapsed into other
- Evidence
Compression is not automatically pathological collapse. Shortening an answer is normal; a transformation only creates the opportunity for loss. What DSI measures is whether structure a governed reference says should remain visible actually did.
Measurement integrity
If the ruler changes, what does the number mean?
If the reference, population or evaluator changes, did the system improve — or did the ruler change?
Building the instrument surfaced a second problem underneath the first: a change to admission, reference construction, sensors or authority can alter what a metric means without altering the number. That finding is why the measurement contract is written down and frozen before anything is measured against it.
Available today
DSI Audit.
DSI Audit is the current local implementation of Decision-Space Integrity. It audits supplied outputs; it does not generate them.
Evidence status
What is established, and what is not.
The decision-space collapse phenomenon, within the scope of the original 6,480-output study.
Measurement integrity, comparability and the governance of instrument change.
Independent human validation of the lexical instruments. Deliberation-loop efficacy (tested, not supported in R-EGA: DOI 10.5281/zenodo.22919859). Any improvement in decision outcomes.
Availability of the public replication materials for the original study.
Research record
Everything, in one place.
Research note, 7 October 2026: DOI 10.5281/zenodo.23224092
Research publications, 23 September 2026: DOI 10.5281/zenodo.22919859 · DOI 10.5281/zenodo.22919868
Preprint · DSc v1.1 · DOI 10.17605/OSF.IO/KW25A · DSc v1.2 · DOI 10.5281/zenodo.23145725 · Public replication materials · Programme history · Articles · Evidence register · Author