AI Tech Debt: What Breaks After the Demo Works
5 min read Deutsch
Every startup is shipping AI features right now.
Most are also building a second kind of tech debt. It does not look dangerous at first, because the demo works. The prompt returns something useful. The RAG pipeline finds the right document. The customer says: nice, this feels like magic.
Six months later, nobody wants to touch it.
Classic tech debt is visible: messy code, missing tests, services that should have been split years ago. AI tech debt is different. It lives across prompts, models, data, evaluation, product behavior, and team trust.
You cannot refactor it away with one motivated weekend and a large coffee.
Here is what it looks like in practice.
1. Prompt debt
Your product behavior lives inside a 2,000-token prompt.
Nobody has version-controlled it properly. Nobody has tested it. Nobody understands it end-to-end. One engineer added a sentence last October because it fixed one edge case. Now that sentence is load-bearing and nobody knows why.
This is how prompts become folklore.
Changing them feels harmless because it is just text. So people add instructions, exceptions, examples, counter-examples, and one odd sentence that contradicts the previous one because it improved staging.
The system still works. Until it does not.
Fix: treat prompts like code. Version them. Review them. Test them. Keep a changelog. Write down why a prompt changed, not just what changed.
And please do not let production behavior depend on a prompt that only one person understands.
2. Model debt
You built the first version on GPT-4. Then Claude was cheaper. Then Gemini had the better context window. Then one team added a local model for privacy reasons.
Now retries hit three providers, costs are hard to attribute, and nobody knows what happens when one model changes behavior.
This is not vendor flexibility. This is dependency sprawl.
Model choice affects latency, cost, output style, failure modes, evaluation results, and product behavior. Switching models is not like switching a database driver. The same prompt can behave differently. The same user input can fail differently. Safety logic can become too strict or too loose.
Fix: put a thin abstraction between your product and the model provider. Not a fantasy “we can swap any model any time” abstraction. A boring, practical one.
Track model usage, cost, latency, prompts, and provider choice per feature. If you cannot answer “what did this AI feature cost us last week?”, you have a surprise invoice generator.
3. Evaluation debt
You shipped a RAG feature.
Users ask questions. The system answers. Sometimes users complain. Someone tweaks the prompt. Complaints stop. Everyone moves on.
But what improved?
Did retrieval get better? Did the model become more cautious? Did the prompt hide uncertainty better? Did you fix one customer problem and break five silent ones?
Without evaluation, you do not know. You are steering by vibes.
Teams ship LLM features without a repeatable way to tell whether the system is getting better or worse. That is not engineering. That is superstition with nicer tooling.
Fix: build a golden dataset.
Start small. 50 to 200 real examples are enough at first. Include real user questions, real documents, boring cases, edge cases, and cases where the right answer is “I do not know.”
Run this dataset on every meaningful prompt, model, or retrieval change. You do not need a perfect benchmark. You need a baseline that tells you when you made the system worse.
4. Data debt
Your AI feature depends on data that was never designed for AI.
Sales notes in Salesforce. Freeform tags in Airtable. Old wiki pages. User-entered strings with typos. Product categories that marketing renames every quarter.
Then someone connects an LLM to it, and messy operational data becomes part of the product.
A human can understand that “enterprise”, “Enterprise Plan”, “ENT”, and “large customer” might refer to the same thing. Your pipeline might not. Your embeddings might not. Your retrieval filters definitely might not.
Every undocumented schema change becomes a possible regression. Every inconsistent label becomes model context. Every outdated document becomes a confident wrong answer waiting to happen.
Fix: document the contract between your AI features and your data sources.
Which fields matter? Which labels are stable? Which documents are allowed into retrieval? Which systems are the source of truth? What happens when a field changes?
Treat data changes as product changes when AI depends on them.
Because they are.
5. Trust debt
This is the one people underestimate.
Every time your LLM hallucinates in front of a customer, trust goes down. Not just customer trust. Team trust.
Engineers start distrusting the system. Product avoids ambitious use cases. Support writes manual workarounds. Leadership asks for more “guardrails”, without knowing what that means.
Then the system gets wrapped in validation layers, fallback logic, manual approvals, and hardcoded exceptions. Some of that is necessary. Too much kills the thing that made the feature useful.
The team no longer improves the AI system. It defends itself from it.
Fix: decide upfront what the AI owns and what it assists.
There is a big difference between:
- AI suggests a reply
- AI sends the reply
- AI classifies a support ticket
- AI closes the support ticket
- AI summarizes a contract
- AI decides whether the contract is acceptable
These have different risk profiles. They need different product design, evaluation, and fallback paths.
Never ship an AI feature without a non-AI fallback. Production systems fail, and this one fails in ways that can look correct.
Why this debt is worse than normal tech debt
Normal tech debt compounds inside your system.
AI tech debt compounds inside your system and outside of it.
The model changes. The provider changes pricing. The API has an outage. The same prompt behaves differently after a model update. Your data changes shape. Your users discover inputs you did not test. Your evaluation set becomes outdated.
You cannot freeze the version and feel safe.
The ground moves under you.
“We added AI” is not a strategy. It is barely even a feature description.
The startups that handle this well are not the ones with the cleverest prompts. They are the ones with technical leadership that treats AI as a system.
Prompts, models, data, evaluation, and trust are all part of that system.
Ignore one long enough and it will show up on your roadmap later. Usually at the worst possible time.
If your team is shipping AI features without these guardrails, a fractional CTO engagement is usually cheaper than cleaning up the mess later.
Ask me how I know.