Who Decides Whether the Goalposts Moved?

A system is tested against a thousand cases and fails a hundred and twenty of them. An AI coding agent investigates, concludes that the failing cases are ambiguous and should never have been in the set, changes the software, reruns the evaluation, and the score improves substantially.

The agent may well be right. Those hundred and twenty cases may not belong. But if they do belong, the system hasn't improved. The test has become easier.

That much is familiar enough. The part I find harder is that the same thing can happen when nobody touches the test at all.

Suppose the agent changes a parser instead. A parser is plainly implementation. Its job is to prepare inputs for the evaluator, and the change might be an entirely legitimate correction, a real bug properly fixed. But the old parser admitted 918 cases for evaluation and the new one admits 871. The implementation has changed. So, possibly, has what is being measured. Both of those can be true at the same time, and the boundary between them turns out to be surprisingly hard to draw.

I've been running into this while building Decision-Space Integrity (DSI), a system for measuring whether AI responses preserve reasonable decision paths or quietly drop them. I've used coding agents heavily throughout, not only to write code but to implement tests, inspect failures, propose corrections and produce evidence that the thing behaves as intended. It works remarkably well.

But DSI is itself a measurement system, and that makes some perfectly ordinary software changes less ordinary than they look. A parser decides which evidence reaches the evaluator. An applicability rule decides which cases count. A threshold changes a classification. An evaluator prompt changes what gets recognised as evidence. Aggregation logic can move the final number without altering a single observation underneath it. None of that looks like changing the test.

Which brings the question back round. Who decides whether the goalposts moved?

The obvious answer is that the AI shouldn't be allowed to change its own evaluation criteria. It's a good answer, and it skips the difficult part, which is working out which changes are changes to the evaluation criteria. An agent might change a parser because it thinks the parser is wrong, and it might be right. It can explain the bug, produce a cleaner implementation, pass every software test and make a perfectly reasonable case that it has done nothing except fix the code. If the change also alters which observations enter the measurement, then deciding it was "only an implementation change" is itself a judgement of some weight. And the agent stands to benefit from that judgement.

None of this is new. Regulated industries have worked on material change, independent validation and revalidation for years. Software assurance has change-impact analysis. Safety cases have to account for changes that invalidate their own evidence. AI safety has specification gaming and reward tampering. More recent work looks at epistemically significant changes to AI systems, including what happens when the system, or its provider, helps decide whether a change requires reassessment. Finding that literature was useful, mostly because it stopped me mistaking something I'd met in engineering for a field I'd discovered.

What does seem different with generative agents is speed and volume. An agent can make hundreds of implementation decisions, watch the consequences, investigate the failures, modify the code and the machinery around it, rerun everything and explain why the result is better. No gaming is required for this to become a problem. The agent can be doing exactly what we asked it to do, which is find problems and fix them. In measurement software, fixing the implementation and changing the instrument are not always separate operations.

I've started taking that distinction more seriously in DSI. A failed measurement shouldn't disappear because the system changed afterwards. If an agent believes the measurement itself is wrong, it should be able to say so and argue the case as well as it can. But proposing a change and making that change authoritative are different acts. If one version of the instrument produces 71% and a legitimately revised version produces 89%, nothing has gone wrong, provided the record still says that R1 produced 71% and R2 produced 89%, rather than quietly ending up with R1 at 89%.

It didn't.

All of which sounds like an elaborate way of saying don't let the AI mark its own homework. I don't think that's the problem. Marking your own homework assumes we already know what the homework is and how it will be marked. The harder version is allowing AI to make ordinary engineering changes when some of those changes alter what the marking scheme means.

Putting a human in front of every change isn't much of an answer either. At hundreds or thousands of changes, human approval becomes a button somebody clicks.

I don't have a clean solution to this. My suspicion is that it involves being much more explicit about which parts of a measurement system carry authority, preserving the evidence produced under each version, and separating the ability to propose a change from the authority to decide that it left the measurement unchanged.

There's a more practical question I'd like to ask first, and it's the reason I'm writing this. People working in regulated software, assurance, scientific computing and model validation have been drawing versions of this line for decades. When a change looks like ordinary engineering, where do you draw the line at which it becomes material enough that the existing evidence can no longer simply be carried forward?