When Documentation Lies: Detecting Drift Between Code and Reality

\ For the last 15 years I’ve worked on dozens of software projects, and almost every time the documentation was terrible. There are several reasons for this:

• Developers underestimate the need for documentation because they think they already understand everything.
• Managers underestimate it because they assume developers can simply read the code and understand the project in a couple of hours.
• Even when people don’t underestimate documentation, they often lack the time and capacity for it. The accurate, up-to-date overview usually sits in the heads of senior developers, who already carry many responsibilities.
• There is no shared culture or intuition about what must be documented and what is obvious. Developers often add comments for trivial functions or describe parameters when clearer variable names or types would have solved the problem. Meanwhile, the truly complex or “hacky” parts remain undocumented.

Maybe documentation is just obsolete?

only 4% of companies always document their processes, indicating many teams treat documentation as an “optional formality” https://www.bptrends.com/bpt/wp-content/uploads/2015-BPT-Survey-Report.pdf/

At the same time we have strong evidences that without good documentation business just lose huge amount of time-money:

• “insufficient documentation” cited by 41% of developers as a top cause of wasted time, with 69% of devs losing 8+ hours per week to such inefficiencies (amounting to ~$18.5M in annual lost productivity per 1,000 developers) https://newsletter.getdx.com/p/state-of-developer-experience-2024
• Onboarding a New Engineer Takes 3–9 Months, Heavily Dependent on Documentation https://stripe.com/files/reports/the-developer-coefficient.pdf
• Lack of Documentation is a Major Component of Technical Debt (https://www.informit.com/store/economics-of-software-quality-9780132582209)
• An experiment with 30+ programmers showed that lack of documentation caused about 21% more time spent understanding code during maintenance tasks https://hci.com.au/get-documentation-budget/

So what is the actual barrier behind such a widespread lack of documentation?

In the brilliant work from Andrew Forward “Software Documentation — Building and Maintaining Artefacts of Communication” have made a good research into the actual reasons.

External Factors Influencing Documentation Quality

\

As you can see, the lack of time and budget correlates with the frustration that documentation effort may become useless and quickly outdated. \n  So if we reduce the manual effort and solve the problem of documentation becoming outdated, we can make a significant step forward.

The developer community has already introduced approaches like Javadoc and similar tools that generate documentation automatically from code signatures. This helps, but it covers only a small fraction of what actually needs to be documented.

So we still need well-written, up-to-date, human-readable documentation.

The real challenge is that documentation is not a one-time deliverable but a living artifact. Even if a team writes good docs at the beginning, code evolves faster than documentation habits. Without continuous synchronization, documentation becomes misleading — which is even worse than having none.

This means the bottleneck is not writing documentation, but keeping it correct.

Solutions

AI

AI is actually good at matching simple patterns between code and docs. There are some nice solutions where you can automate documentation drift via AI:

• Merge request rules, pointing agent to existing documentation and changed code base
• Claude Code documentation agent that keeps project docs up to date with Docusaurus. \n

npx claude-code-templates@latest --agent=documentation/docusaurus-expert --yes

This approach works well for:

• Updating high-level descriptions
• Summarizing code intent
• Rewriting outdated phrasing
• Restoring missing narrative context

However, AI has clear limitations:

• It struggles to detect structural inconsistencies (e.g., undocumented env vars, ports, CLI flags, feature flags).
• It can hallucinate, especially when docs are incomplete or ambiguous.
• It does not produce traceable, deterministic checks — every run may generate a slightly different result.
• It cannot reliably tell what is important and what is noise without domain knowledge.

In other words, AI can help rewrite and improve text, but it is not yet reliable as a source of truth validator.

Static checks

Tools like Ducku addresses this problem by treating documentation as something that must be monitored continuously, just like code quality or infrastructure drift. It works purely algorythmically so not affected by hallucinations, although false positives are of course still possible. But since it’s configurable, you can mute certain use cases.

Ducku works by extracting structural signals from your repository — environment variables, API routes, service entry points, module imports, directory structure, configuration keys — and comparing them with what is referenced in your READMEs and wiki. When something diverges, it flags it.

Current Capabilities

• Entity presence checking: Detects when environment variables, config keys, ports, API routes, or script names appear in documentation but not in code (and the other way around).
• Parallel entity coverage: Identifies groups of similar items (services, ETL jobs, lambda handlers, CLI commands) and flags undocumented additions.
• Dead module analysis: Detects files that are not imported/used anywhere — either entry points that deserve explanation or obsolete artifacts.
• URL integrity check: Detects broken or outdated links to internal or external resources.
• Spell and style consistency: Basic hygiene that normally gets ignored.

This is already enough to reduce a large portion of silent documentation drift in real projects.

Conclusion

Documentation does not fail because engineers are careless. It fails because there is no feedback loop that keeps it aligned with the system it describes. Code has tests. Infrastructure has drift detection. CI has policy gates. Documentation, in most teams, has nothing.

AI can improve phrasing and restore context, but it cannot reliably tell whether documentation is correct. Static checks, on the other hand, can validate factual alignment, but they cannot explain intent or domain logic.

Both together they can bring what you need.