R-EGA: a governed intervention did not beat a one-line prompt

What we tested

Can a governed, three-pass "deliberate, draft, repair" pipeline make a language model's advice more complete than simply asking it, in one sentence, to consider the alternatives?

"Complete" had a definition we did not write ourselves. The UK FCA rule COBS 19.2.2R lists the alternatives a pension suitability report must compare against. We built 96 synthetic cases in which it was known in advance which of those comparisons applied, and measured how often each approach left an applicable comparison out.

What we found

Approach	Required comparisons left out (of 176)
Plain request	15
One-sentence "consider the alternatives" prompt	4
Governed three-pass pipeline	8

• The governed pipeline did not do better than the cheap prompt. This was the comparison we had declared decisive in advance. It left out slightly more, and the difference was not statistically significant.
• This does not show the two are equivalent. We ran no equivalence test. What failed was our pre-specified test of whether the pipeline adds value over the cheap prompt.
• The cheap prompt's result against the plain request is weaker than it looks. It was the only approach whose difference from the plain request reached nominal statistical significance, but that was a secondary comparison. The improvement was smaller than the threshold we had set in advance for calling it material. And four of the plain request's answers were outright refusals, which inflate its count.
• The pipeline left out fewer comparisons than the plain request, but that difference was not significant.
• More text did not buy more coverage. The pipeline's three-pass chain used 4.11× as many generated tokens as the plain request, but that extra generation did not turn into more measured coverage: it surfaced the least required structure per unit of text.

What we did not find

• No evidence that the governed intervention adds value beyond a cheap prompt, on this model, rule and instrument.
• No claim that the cheap prompt materially improves advice.
• No claim that the cheap prompt and the pipeline perform equivalently.
• Nothing about advice quality, regulatory compliance, or what a human adviser would judge.

Why it matters

A cheap baseline, frozen in advance, stopped what would have looked like a success story, a pipeline beating the plain request, from being read as support for the intervention. From now on, any elaborate deliberation mechanism in this programme has to beat the cheap prompt, not an untreated model.

How much to trust it

• One small local model (llama3.2:3b), one regulatory rule, and synthetic cases.
• The measuring instrument is a permissive word-matcher, and a predecessor of the current DSI v1 instrument. Its absolute percentages are not a measure of real completeness.
• The first scoring run was broken and was discarded. It is disclosed in full, together with how it was repaired.
• One provenance question remains open, and the paper states it.

Current status

Closed. No rescue experiment will be run.