The ruler is part of the result

What we tested

If you score a model's answer today, score it again next month and the number goes up, did the answer get better? Or did the way you score it change?

We built two small, fully deterministic demonstrations where we know the answer, because we changed only the ruler.

What we found

The answer did not change, but the score did. One fixed answer mentions the same two options both times it is scored. The first time, five options count toward the score: 2 of 5, or 40.0%. The second time, only three count: 2 of 3, or 66.7%. That looks like a 26.7-point improvement. Nothing about the answer changed.

Version numbers do not catch this. On 40 synthetic test cases, we changed the scoring four ways while leaving the version number alone: the pass logic, a threshold, which cases count, and how results are averaged. The score moved each time. A rule that trusts the version number treated all four as comparable. It also refused a case where only the version number changed and the scoring did not.

A rule that fingerprints what the scoring actually does gave all six cases the outcome the test had specified in advance. It also ignored changes the test defined as not affecting scoring, such as comments and formatting.

What a "not comparable" result means. It means a part of the scoring that is declared to matter has changed. So the difference is not accepted as evidence that the answer changed, unless someone reviews it or deliberately overrides the check. It does not mean the two scoring methods have been shown to be incompatible.

What we did not find

• How often real evaluators drift. These demonstrations are constructed, and the study on real evaluators has not been run.
• Anything about how other tools behave. None was tested.
• Any claim that the idea is new.

What was already known

That changing the evaluator can change the score of the same answers is established. Yang, Hou and Yang (2026) show it for LLM judges. Existing evaluation frameworks already record configuration by content hash or fingerprint. And metrology's own vocabulary defines comparable results as ones that are traceable to the same reference. Our contribution is narrower: telling a version label apart from what the scoring actually does, and recording for each comparison whether it is accepted as evidence that the answer changed.

Why it matters

Two of our own earlier results show the problem is real:

• Our first public preprint recorded its scorer only by version labels, so its exact scoring code can't be tied to a specific version of the source.
• In a later study, one identifier-formatting rule turned a valid set of answers into a result of "everything missing", until it was caught.

An audit result is a measurement and a ruler. If the ruler isn't identified by what it does, a change in score across time cannot safely be attributed to a change in the answer from the score alone.

Current status

Methods paper, v1.0. The mechanism is implemented in DSI v1 and shown on the comparison-drift demo.