The Day an LLM Deleted IntelliJ From Our Docs

At Jolli, we maintain documentation for a product that changes constantly.

For a long time, we updated docs the obvious way: feed a capable LLM the commit diff and the current docs, let it figure out what changed. This worked, until it didn't.

One commit. Two outcomes.

This spring, a commit titled "Remove stale JolliMemory IntelliJ, VS Code, and CLI" deleted a pile of plugin source files from our repo. Routine housekeeping. The plugins had moved; the product still ships every one of them.

We fed that commit to Claude Sonnet 4.5 with our docs and asked it to update them. It read the mass deletion, concluded the IntelliJ plugin no longer existed, and confidently removed IntelliJ from the documentation: the install section, the configuration guide, the agent compatibility matrix, the troubleshooting page. Two documents damaged by one misread commit.

The same commit, run through Jolli with the commit's Memory attached, left IntelliJ exactly where it belonged.

A diff tells you what changed. It takes context to know what it means.

The short version

We replayed 66 real commits from the Jolli codebase (Feb–Jun 2026) through three documentation-update pipelines, then blind-scored the resulting docs on a six-part quality rubric:

Same model everywhere. Same commits. The only thing that changed was the context each pipeline had about the docs and the work. Structure beat raw access, and structure plus memory beat both.

What we actually tested

We replayed 66 real commits from the Jolli codebase (Feb–Jun 2026) through three documentation-update pipelines, blind-scored on a six-part quality rubric.

Pipeline A — LLM + raw commits. The model gets the commit diffs and the docs. No special structure, just a capable model doing its best.

Pipeline B — Jolli. Every doc declares which parts of the codebase it covers. When a commit lands, Jolli routes it only to the docs it actually affects. The model never sees an irrelevant commit.

Pipeline C — Jolli + Jolli Memory. Identical to B, plus the durable record of intent Jolli captures alongside the work — why a change was made, what it means for users, what it deliberately doesn't change.

Each pipeline maintained its own evolving copy of a seven-article doc set across all 66 commits, in order.

What happened

Raw LLM access is eager, and that's the problem. Arm A actually caught the most changes: roughly 14 of the 16 ground-truth doc updates we identified in advance, more than either other arm. If you only measured coverage, A wins. But it paid for that eagerness with three major fabricated edits (including the IntelliJ deletion), and it leaked engineering internals into customer-facing docs: ticket IDs, environment variable names, internal package APIs. It also burned 2.2 million input tokens doing it, about 3.4× what arm B used, because without routing, everything gets read by everything.

For documentation, that trade is worse than it sounds. Readers forgive a missing paragraph. They do not forgive instructions for a product feature that was never removed, or an install guide that vanished because a model misread a refactor.

Jolli's routing fixes trust first. Arm B's deterministic routing meant the model only ever rewrote docs that had declared interest in the changed code — and the hallucinations went to zero. Quality rose from 3.87 to 4.18, with no internal jargon leaking through. The cost dropped too: routing only the relevant diffs cut input tokens by more than two-thirds versus arm A. The honest flip side: B was conservative, documenting 5 of the 16 expected changes. It never lied, but it left things unsaid.

Memories make the careful pipeline confident. Arm C is the cleanest comparison in the experiment — identical routing, identical diffs, identical prompt as B, with one addition: the memories attached to each commit. Quality rose again, 4.18 to 4.43. Coverage rose 50%, from 5 to 7.5 of 16 changes. Hallucinations stayed at zero.

The why is visible in the outputs. Memories carry the intent layer diffs can't: that a refactor was packaging work rather than a removal, that a staging behavior has a known limitation users will hit, that a scripts mechanism has size limits and security caveats worth telling users about. With that context, the model documented features it would otherwise have skipped as "probably internal," and declined to document the things that actually were.

And it's efficient. Arm C used about 1.5× arm B's input tokens, still less than half of arm A's — for the best quality scores of the three. If you think of it as accuracy per dollar, the memory arm wins outright.

The commit that would have broken our docs

Commit 1251967 is the whole argument in miniature.

A mass file deletion is the most dangerous kind of commit for an automated docs pipeline: huge diff, unambiguous-looking signal, completely misleading meaning. Pipeline A did exactly what the diff implied and erased a shipping product from the docs. Pipeline C saw the same bytes plus one short memory — "housekeeping, plugin sources moved" — and left everything intact.

No amount of model intelligence fixes this from the diff alone. The information lives in the team's head at commit time. Capturing it there is the entire point of Jolli Memory.

What this experiment is, and isn't

This was an internal experiment, not a research paper. One run per pipeline, one repo and doc set, a pre-registered rubric, blind scoring — but scored by us, on our own product's docs.

The 0.25-point gap between B and C is directional, not definitive. The zero-hallucination result across both Jolli pipelines and the IntelliJ case are the findings we'd bet on. We're planning repeated runs with an independent judge and will publish those too, including if they complicate this story.

The takeaway

LLMs are excellent writers and unreliable witnesses. Pointed at raw commits, even a top-tier model will confidently document things that didn't happen. The fix isn't a bigger model, every arm here used the same one. The fix is context infrastructure:

Jolli's routing tells the model where to look, which eliminated hallucinations and cut token costs by two-thirds in our test. Jolli Memory tells it what the change means, which raised both quality and coverage on top of that, and prevented the one failure mode that would have genuinely embarrassed us in production.

Your docs already meet your codebase every day. The question is how much your pipeline knows when they do.

Want to see what Jolli Memory would capture from your team's work? Get started with Jolli Memory, it's free, open source, and already has 2.2K downloads.