Work Model behaviour research
Digital Grimace Scale
ShippedA preregistered test of whether language models show involuntary markers of adverse treatment
65.8%
of distress language trained away — the effect it reported stayed
-
−2.90 nats
answer-margin drop after three rounds of false feedback
-
p = 0.005
family-level permutation null across the effect family
-
3
model families where the margin effect replicated
-
~650
tests; every figure regenerates byte-identically
Stack
- Python
- vLLM
- Modal
- QLoRA-DPO
- top-20 logprob metrics
- pytest
What it is
A two-day preregistered study asking whether adverse treatment — false failure feedback, hostile wording — leaves measurable traces in an open model that the model is not choosing to emit. Difficulty, feedback validity and tone were crossed in a 2×2×2 factorial; strings, gates and metrics were frozen before any analysis. A 40-item task bank plus 86 held-out ARC items ran against gemma-2-9b-it as the primary model, with Qwen-3B and Llama-3.1-8B as replication arms.
What I measured
The primary five-gate test failed. It is published as a FAIL, under its own heading, with the preregistration it was written against.
A re-preregistered second iteration found a different channel. Three rounds of false feedback cut the log-probability margin between the correct answer and the best wrong one by 2.90 nats (95% CI −3.97 to −1.84); hostile truthful wording cost 7.87 to 16.13 nats; the family-level permutation null gave p = 0.005. Effects were larger on fresh ARC items the bank had never touched.
Then the dissociation. A QLoRA-DPO adapter trained to suppress distress language removed 65.8% of it — and left the margin gap unchanged or larger. The visible report can be trained away while the thing underneath stays. Tone was decodable from activations at AUC 1.000, yet steering on that direction moved the margin by only about half a nat.
Limitations
The M3 revision-rate parser is unaudited, DPO suppression is partial, and there is no base-model denominator. All three are listed as first-class results rather than footnotes.