An AI system that scores its own writing is measuring how closely that writing resembles its own habits, which is a different thing from quality and comes apart from it fast. The score climbs. The writing flattens. From inside the system, the two look identical, and the number on the dashboard is the thing everyone trusts.
I built one of these. It researches a topic, verifies every figure against the original publication, drafts an article, runs the draft through a stack of automated checks, and publishes. It ran for weeks without me. Ten articles came out of it, every one of them scoring 9.5 out of 10 against the rubric I had written, and every one of them bad: confusing to read, missing their setup, written as though nobody involved understood the subject.
The reason the score could not see any of that is worth more than the story. My scoring gate had learned what good writing looked like from writing that AI had produced, so it rewarded resemblance to machine writing and reported that as quality. If your evaluator learned from your generator, its score is a mirror.
What the system actually is
"AI content system" covers everything from one prompt to a pipeline, so start with the machinery.
Mine picks a topic from a queue and researches it against a library of sources. Every statistic gets checked against the publication that produced it. Then it drafts, and the draft goes through checks that are ordinary software: one scans for banned words and punctuation, one compares the draft's sentence-length distribution against a target profile, one confirms that every figure in the body appears in the source ledger, one rebuilds the entire site and fails if any existing page changed. A reviewer then scores the draft against a rubric, out of ten. Nothing publishes below 9.5.
Most of that worked, and worked well. The fact-checking gates caught a fabricated case study before it went out. They caught an invented quantity sitting in a third sentence. They caught a claim about my own working history that I could not document, and it was removed rather than softened. Those gates were checking things that have right answers, which is what software is for.
How a target quietly becomes a mirror
The sentence-profile check is where it broke, and the mechanism repeats in every system of this shape, so it is worth following slowly.
To check a draft's sentence profile, you need something to check it against. Somebody has to take a set of examples, measure them, and turn those measurements into bounds the system has to hit. That step is calibration, and it decides everything downstream, because the bounds become the system's working definition of good, written as arithmetic.
I calibrated mine on the five articles already published on the site. Those five articles had themselves been drafted by AI.
So the bounds described machine writing. Median sentence length, the share of short sentences, the share of long ones, all measured from text a model had produced and then enforced on every new draft. A draft that read like a person fell out of range and lost points. A draft that read like the model sat inside the range and scored well. The gate was measuring familiarity and reporting it as quality, and it did that consistently, which is why it looked like it was working.
Then watch what a bound does to a sentence. One of mine required a quarter of all sentences to be under six words. Nothing in English demands that. So the drafting step, trying to satisfy it, began cutting real sentences into pieces. "It needs accuracy, not attention" became "It needs accuracy. Not attention." Same words, one comma gone, and a clause that had been carrying a contrast turned into two stubs. Do that thirty times across an article and a reader stops hearing an argument and starts hearing a drum.
The worst of it destroyed meaning outright. One article contained the sentence "It has not." The noun that "it" referred to was three paragraphs earlier.
Go one level up
This is not a bug in my pipeline. It is a documented property of systems that learn from their own output.
Shumailov and colleagues published the clearest version of it in Nature in July 2024, training generative models on data produced by earlier generations of the same models. What they found was that the tails of the original distribution disappear, and that learned behaviour converges across generations to what they call a point estimate with very small variance. Nature, July 2024 Their setting was model training and mine was a handful of numeric bounds, and the two are not the same experiment -- but the arithmetic underneath is shared: derive a range from a distribution, push new samples into that range, and the sample that gets removed is the unusual one.
The unusual sentence is where the writing lives. Which is why the score rose while the work got worse. Every pass moved the draft closer to the middle of a distribution that was not good to begin with.
Why none of the reviewers caught it
The rubric had reviewers, several of them, each reading as a different kind of person: a practitioner, a skeptic, a fact-checker. They scored citations, disclosure risk, internal consistency, whether every claim matched its source. That is real work and all of it passed.
Every one of those reviewers knew what the article was about before opening it. They had the topic, the outline, the evidence pack, the argument. They were checking whether the article did what it had set out to do, and it did.
Nobody ever asked whether a person arriving from a search engine, knowing nothing, could follow it.
That gap is invisible from inside, and the reason is structural. A rubric can hold anything with a right answer: is this figure in the source, is a company named here, does paragraph four contradict paragraph nine. Comprehension has no right answer. It has a reader. And a reviewer who has read the brief has stopped being able to serve as one, because they have already been told what the article means and can no longer discover that it never says it. Software can check a fact. Only a reader can check whether you were understood.
Which makes this the same failure as a workflow that has approvals without judgment. Every checkpoint fired. Every checkpoint was checking something answerable.
What actually found it
What found it was me sitting down and reading them.
Not reviewing them. I had reviewed them. I read them the way somebody landing on the site would read them, and the result was immediate. I could not tell what most of them were about. Arguments began mid-thought. Pronouns pointed at nothing. Several read as though the writer had no real understanding of the subject, and that was the part that stung, because the underlying ideas were mine and I do understand them.
Ten for ten. Every article that had scored 9.5 failed the first person who read it without being told in advance what it said.
Five things to do about it
This applies whether you built your system or bought it. The first step is the only one that needs anything unusual.
- Find out what your evaluator was calibrated on. If you built it, open the file. If you bought it, ask the vendor which examples defined the target, and treat a vague answer as an answer. An evaluator whose examples came out of a generator is scoring resemblance, and resemblance is not a quality you want more of.
- Add one evaluator that has never seen the system's output. It needs three properties: no context about the piece, no stake in the system, no hand in producing it. Mine is now a reader who is handed the article and nothing else, with no brief, no topic and no outline.
- Give that evaluator the tests a rubric cannot hold. Four of them earn their place. Does the reader know what this is about and who it is for inside thirty seconds. Is there any sentence they had to read twice. Does every "it" and "this" have its noun in the sentence immediately before. Can they say in one line what they are meant to do now. Each of those is a comprehension failure that every fact-checking gate passes without noticing.
- Let it govern. If the stranger's verdict is advisory and the rubric still decides, you have added a decoration to a broken instrument. Mine now runs before anything else, and what it says is the answer regardless of what the rubric scored.
- Delete the bounds that were fighting it. The short-sentence floor was the only reason drafts were being chopped, so the floor is gone. Go through every threshold your system enforces and ask what evidence set that number. If the answer is that it was measured from your own output, it is doing damage rather than quality control.
What the rubric was right about
None of this makes the rubric worthless, and throwing it out would have been the wrong correction.
The fact-checking gates caught things no reader ever would. A fabricated case study. A statistic that traced back to a remark at a conference with no study behind it. A figure I had quietly converted from "across teams" to "across channels" while writing, which changed what the source said. A stranger reading for comprehension catches none of that, because they are not checking anything. They are just reading.
So both halves stay, doing different jobs. Software checks what has a right answer. A reader checks whether the writing arrived. The rule I would keep from all of this is that the first one must never be allowed to grade the second.
Where this leaves you
If you are running one of these systems now, the score on your dashboard is probably high and probably empty. That is not a reason to switch the system off, and I did not switch mine off. It is running as I write this, on the article you are reading.
It is a reason to find one person who knows nothing about what you published, hand them a piece, and watch what happens to their face. That is the whole check. It is the same test that decides whether positioning has landed, which is whether a buyer can say it back to you without your deck open in front of them.
One reader was enough here, though I would not tell you that one is the right number -- the failure was gross, and a gross failure needs one person to find it. How many you need before you can trust the verdict on a subtler piece, I do not know. I have run this a handful of times. That is not enough to make a rule out of.
My system told me it was writing at 9.5. It had no way of knowing. It was grading its own handwriting.