A health check on an ordinary service asks one question: is it up? Return a 200 and you are healthy. That question is meaningless for a memory system. A brain can answer every request, on time, while quietly believing things that are stale, contradictory, or never happened. Its failures are not crashes — they are wrong beliefs held with confidence. So the interesting question is not "is it running" but "is what it knows any good," and that needs a different kind of check.
An earlier note described building the cognitive half of Mnemos — consolidation, forgetting, salience, self-correcting recall. This one is about a smaller, sharper thing: how we score a brain’s health, what the score is made of, and what we learned when we finally ran it against a real one. The honest headline is that the hardest part was not designing the vitals. It was making sure they measured the set they claimed to.
What a brain’s vitals actually measure
Mnemos frames health the way active-inference frames a mind: a healthy brain is one whose model of the world keeps its prediction error low. It is not surprised by what it later observes; its confidence matches its accuracy; it does not hold beliefs that flatly contradict each other. From that idea fall a handful of vitals, each a number in the unit interval where higher is worse, each with its own warn and critical line. One rolled-up verdict — healthy, degraded, or unhealthy — takes the worst of them, so a single sick organ is never hidden by four well ones.
- Free energy — the aggregate prediction-error signal, and a pointer to which layer is most wrong. It is the headline number; the rest explain it.
- Calibration — expected calibration error: when the brain says it is 80% sure, is it right about 80% of the time? A brain that is confidently wrong is worse than one that is unsure.
- Dissonance — active, high-stakes contradictions per belief. Not every disagreement; the ones between beliefs established enough that a conflict means something has to give.
- Low-trust — the share of currently-valid beliefs whose trust has decayed below a floor. Trust falls as the evidence behind a belief ages without re-verification.
- Staleness — the share of valid beliefs not seen or re-verified within a horizon. A recency proxy for "still load-bearing, or just still present?"
Alongside the vitals sit three integrity checks — structural, not cognitive. Orphan beliefs: claims with zero evidence, which should not exist, because every belief in Mnemos is supposed to trace back to a source event. Dangling edges: relationships pointing at a belief that is gone. Stale expectations: predictions the brain made and never reconciled. These are the data-integrity smell tests; a single orphan is a warning, a dangling edge is an error.
That is the whole instrument. It is deliberately boring — sums and rates over the belief set, no model in the loop, cheap enough to run on a schedule. The subtlety is entirely in one phrase that appears in every vital: "currently-valid beliefs." Get the definition of that set wrong and every number is quietly off. Which, it turned out, is exactly what happened.
Running it turned the vitals red
We wired the brain into an agent’s daily loop and let it capture weeks of real work. For a long stretch capture barely ran — a couple of claims a day. Then a change landed that made it capture incrementally, throughout a session rather than only at the end, and it started working properly. "Working properly" meant absorbing the entire transcript. Intake went from around two claims a day to several thousand; on the brain we were watching, thirty-one percent of everything it knew had been created in a single working day.
Volume is not the problem; a brain is allowed to learn a lot in a day. The problem is what the volume was. We hand-labelled a sample and it came back roughly fifty-eight percent narration — not knowledge. Progress reports, status snapshots, completion announcements. "First test fails." "CI passed in 1m59s." "Merged as abc123." Every one true at the moment it was written, and none true for long. This is the exact thing the vitals are built to assume away: knowledge erodes slowly, so trust and staleness decay over a 90-day horizon. Narration does not erode — it flips, from true to false, the instant something happens. Feed a slow-decay model a stream of instant-flip beliefs and the vitals go red for a real reason and a wrong reason at once.
Dissonance: a conversation arguing with itself
The vital that flashed hardest was dissonance. Contradiction detection is one of the parts of Mnemos we are proudest of — surfacing a conflict instead of silently overwriting is the whole point of an evidence layer — so a brain full of contradictions reads as alarming, and we started pruning them. Then we stopped and measured what they actually were. Only about a fifth were genuine: two durable beliefs that truly disagreed. The rest had a narration endpoint — the beginning of a conversation lexically colliding with its end, across unrelated sessions.
“Both PRs are open” contradicting “PR #23 squash-merged.” “Key suspicion forming: the timeout is 90s” contradicting “Root cause confirmed: the field is unset.” A narrative that moves will always disagree with itself — its start conflicts with its finish. That is not a contradiction. It is progress.
Eighty percent of the "contradictions" were an intake problem wearing a detection costume. The detectors were mostly fine; you cannot prune your way out of it, because the next session just makes more. That reframed the work: the fix is not a better detector, it is teaching the brain which of its own utterances are knowledge and which are just narration passing through.
The check was counting the wrong set
Here is the humbling part, and the reason a whole section of this piece is about measurement rather than memory. Pruning never moved the dissonance number — and it was not because we pruned too little. The vitals were miscounting, in more than one way, each found only by distrusting them. Every one traces back to that innocent phrase: "currently-valid beliefs."
A belief in Mnemos can be retired two different ways. You can close its valid-time — forget it, so recall stops surfacing it — or you can deprecate its status — mark it stale while its valid-time stays open. Two mechanisms, one meaning: this belief is no longer current. The health code only knew about the first one. It decided "currently-valid" by valid-time alone and never looked at status. So every deprecated belief still counted as current.
That single blind spot surfaced twice, months apart. First in dissonance: it counted contradictions whose endpoints had been deprecated as active, high-stakes conflicts. Retiring a bad belief — the one remedy the tooling offered — moved the number by exactly zero, because the retired belief was still in the count. Two thousand of the flagged edges had an endpoint that no longer existed as a live belief.
Then, later and worse, in the vitals themselves. The loop that computes low-trust, staleness, and the orphan check walked every belief with an open valid-time — deprecated ones included. On a 26,000-claim brain that meant 3,379 retired beliefs were being counted as currently-valid, inflating the denominator every rate divides by. And it produced a genuinely maddening symptom: an ungrounded belief that we deprecated specifically to retire it kept raising the "orphan" integrity warning that deprecating it was supposed to clear. There was no way to clear it, because the count could not see that it was gone. The fix, both times, was one clause — a deprecated belief is not a currently-valid one — and it took the brain from degraded to healthy without touching a single piece of data.
The third miscount was subtler and about arithmetic. A cleanup pass reported that twenty-eight percent of the claims it examined were narration — low against every hand-count. It was dividing by every claim on its work list, including the ones it had run out of time to look at, counting the unexamined as though they had been judged fine. Over what was actually classified the rate was sixty-one percent: understated by more than half, and it was the exact quantity you would use to decide whether the pass was worth running.
And the threshold was never real
Fixing the counts moved the numbers to where they should have been all along. But there was one more layer, and it was the foundation. The lines between healthy, degraded, and unhealthy — 0.05 here, 0.40 there — were described in our own design doc as "conservative defaults," with a note that tuning them against real data was the entire purpose of a metrics journal we had built for it. We opened the journal. It was empty. Recording a health snapshot required a flag nothing ever set, so in months of use it had collected precisely zero data points.
We had spent a week reading a system against a red line that was a placeholder, over a set that counted the retired as the living, calibrated on data that was never collected. The badge was real. What it meant was not.
So we wired the snapshot to record itself, once a day, without anyone asking. The series is short now; in a few weeks it will be long enough to replace the guessed thresholds with ones drawn from how this brain actually behaves. Only then does "degraded" become a grounded verdict rather than a conservative default. A health metric is only ever as good as the data it was tuned on — and that data is something you collect, on purpose, over time.
The discipline underneath: distrust your instruments
The measurement bugs were instances of one habit, and the habit is the transferable part. Over and over a fix started with a plausible story and a supporting number, and the honest move was to check the number before believing the story.
- A detector looked broken; a sample of eight pairs "confirmed" it. Re-running over all of them showed the fault was in a different detector entirely. Eight is not a sample; it is an anecdote with error bars.
- A rewritten classifier prompt scored a huge jump — ninety-two percent — on the examples it was tuned against, and was indistinguishable from the original on examples it had never seen. That is not an improvement; it is overfitting with a good story attached.
- A classification pass reported "zero classified" with no error, which read as a broken model. It was a stale endpoint in a config pointing a cloud client at a local address; every call failed into a safe empty result, and the safety looked like output.
None were hard to catch, and each was cheaper to catch than the work it would have wasted. What they share is that the wrong answer arrived pre-packaged with a reason to believe it. A number that confirms your plan deserves more suspicion than one that contradicts it, not less — and a health metric is nothing but a pile of such numbers about your own system.
How the brain actually stays healthy: mark, don’t delete
Diagnosis is only half of health; the other half is what you do about it, and here Mnemos makes one load-bearing choice. Knowing most intake was narration, the tempting move is to drop it at the door. We do not. Narration is not wrong — it was true when written — and real knowledge is threaded all through it. Deleting a belief on a misclassification is an invisible loss: you never see the thing you no longer know. So instead of dropping, we mark. Each belief carries a durability — does its value outlive the conversation that produced it? Session-local beliefs stay in the store, stay queryable, and simply stop competing with knowledge: they no longer rank in recall, and they no longer generate contradictions with each other.
The safety property that makes this liveable is the same clause the health bugs violated, applied deliberately: unclassified is treated as durable. The entire back catalogue predates the durability field, so absence of a verdict never demotes a belief, and an unrecognised value fails toward keeping a belief in play. Every irreversible step bends toward not losing knowledge — which is exactly the guarantee a memory system exists to make.
Measure the model before you trust it, too
The classifier that decides durability is itself a measurement instrument, so it got the same treatment. A local model was reliable in one direction and a coin-flip in the other: when it called something session-local it was almost always right; when it called something durable it was right about half the time. That asymmetry is usable — we key the destructive action, suppressing a contradiction, on the reliable direction only, so a mistake leaves a belief exactly where it was.
A stronger, hosted model changed the economics: its session-local precision measured between ninety-two and one hundred percent on held-out samples — reliable enough to act on aggressively and safely, and that single number is what let one pass cleanly separate roughly seventeen hundred pieces of narration from knowledge. But we only trusted it because we scored it first, on examples it had never seen, more than once — having already been burned by a model that scored fifty-five percent on a run where every call had silently failed and returned nothing.
So — is the brain healthy?
Now, yes: every vital green, the integrity checks clean, dissonance sitting just under its line. And the remaining contradictions are, for the first time, worth reading — genuine conflicts between durable beliefs, plus a smaller set where a live observation bumps into something the brain holds as settled, which is precisely the signal the vital exists to raise. What was removed was noise; what is left is a question worth a human. But the badge is the least interesting part of the answer. The number moved because we taught the check to count the right set and to tell narration from knowledge — not because the brain underneath ever changed. It was sound the whole time. It was just talking a lot, and we had built a check that could not tell the talking from the knowing.
A coda, in the spirit of the thing. We wrote most of this piece the day the brain first read healthy — and then, cleaning up one last stubborn orphan warning, found the second half of the deprecated-belief bug described above, still live, still miscounting the retired as the living. The article’s own lesson, proving itself one more time on the article’s own author. That is the nature of measuring your own system: the check is another part of the system, and it is wrong in the same quiet ways everything else is. Health is not a badge you reach. It is a practice of distrusting your instruments until they earn it.