AI Vendor Due Diligence: What Founders Should Check Before They Sign

Every AI vendor pitch sounds strangely similar right now.

Enterprise-grade. Fine-tuned on your data. 90% accuracy. SOC 2 in progress. Seamless integration. Production-ready.

Nice words. Not useless, but not proof.

Most founders are asked to buy technical confidence from people who are very good at selling technical confidence. Without someone in the room who has shipped systems like this, a polished demo can look close to a real product.

That is how bad vendor decisions happen.

Not because founders are stupid. Because the sales process hides the parts that matter.

Here is the checklist I run with founders before they sign anything meaningful with an AI vendor. Especially anything over €10K.

1. Ask them to show you a failure

Any serious AI vendor knows where their system breaks.

They know which inputs create bad outputs, which use cases are risky, where accuracy drops, where latency gets ugly, and where humans still need to review the result.

If their pitch is 100% success, that is not confidence. That is inexperience or theater.

Ask:

What does your product do badly?

Then be quiet.

A good vendor answers directly. Maybe the system struggles with sparse data, multilingual edge cases, noisy PDFs, domain terminology, long-tail exceptions, or ambiguous user intent.

That is fine. Every real system has limits.

A bad vendor pivots back to benefits.

Great question. What we usually see is that customers get a lot of value from…

No. That was not the question.

You are not looking for perfection. You are looking for proof that they understand their own product outside the demo.

2. Ask for the eval method, not the eval number

“90% accuracy” means almost nothing by itself.

90% on what? Measured how? Against which baseline? With which data? Reviewed by whom? Under what failure cost?

An AI system that is 90% accurate at summarizing support tickets might be useful. One that is 90% accurate at approving insurance claims might be a liability machine.

Ask them to send the evaluation protocol.

Not the slide. The protocol.

You want to see what data they tested on, how they define success, how they handle ambiguous cases, what baseline they compare against, and whether the evaluation looks like your real use case.

A serious vendor will be proud of this. A weak vendor gives you a number and hopes you stop asking.

Bonus question:

Can we run the eval on our own data during the pilot?

Most serious vendors can accommodate this, even with constraints. The ones that cannot are often selling you a demo, not a product.

3. Check if they are a product or a wrapper

There is nothing wrong with building on OpenAI, Anthropic, Google, Mistral, or any other model provider.

There is something wrong with pretending an API call plus a dashboard is deep technical infrastructure.

Ask:

If your main model provider raised prices 3x tomorrow, what happens to your product?

If the answer is, “We would have to pass the cost through,” you are probably buying a thin wrapper. That might still be fine. Sometimes a wrapper is enough. But price it like a wrapper.

If the answer is, “We can route across providers, we already benchmark alternatives, and switching would take us a week,” that is different.

Then ask what would break during the switch: embeddings, prompt behavior, latency, output format, compliance approvals, customer-specific fine-tunes.

A vendor that has thought about this will answer with specifics. One that has not will give you strategy words.

4. Check the technical team, not the advisory board

Advisory boards do not ship your integration.

Look up the top three technical people at the vendor. You are looking for two things.

Have they shipped AI or data systems into production before?

Have they been at this company long enough to matter?

An impressive researcher who joined three months ago might be a strong signal for future capability. They are not proof that the current product is solid.

A staff engineer who shipped two ML products before and has been there for 18 months is often the stronger signal.

You are not judging prestige. You are judging operational reality.

Research experience is useful. Production experience is different. You are buying the ability to deal with bad data, strange users, broken pipelines, customer escalations, security reviews, cost constraints, and the thousand ugly details between a model and a working system.

5. Ask the boring data questions early

This is the part everyone wants to rush.

Do not.

Data handling is where a harmless-looking AI vendor can become your compliance problem.

Ask, in writing:

• Where does our data live?
• Which region?
• Which cloud provider?
• Is our data used to train your models?
• Is our data used to improve any shared system?
• What is your retention policy for prompts, outputs, files, embeddings, logs, and metadata?
• Who has access to our data inside your company?
• Do you support deletion requests?
• Do you have a DPA?
• Can I have it today?

Not next quarter. Today.

Vendors who cannot answer these questions in writing have not thought enough about compliance.

That is fine for a prototype. It is not fine for customer data.

Their missing process becomes your problem later, usually during an enterprise audit, a security review, or a board meeting nobody enjoys.

Boring questions save expensive problems.

6. Do not sign annual before reality has touched the system

Never sign an annual contract for a new AI vendor before a real pilot.

Not because vendors are evil.

Because AI systems are sensitive to your data, workflows, users, edge cases, and quality bar.

A demo does not test that.

Demand a 60-day paid pilot with a hard exit. Paid is fine. Free pilots often attract unserious behavior on both sides. But the pilot needs clear terms.

Include two things:

• A measurable success criterion
• A clean exit clause

The success criterion cannot be “we feel good about it.”

It should be something like:

• Reduce manual processing time by 30%
• Reach 95% extraction accuracy on defined fields
• Keep human correction rate below 10%
• Handle 80% of support requests without escalation
• Stay below a defined cost per processed item

The exit clause matters just as much.

If you leave, can you export your data, embeddings, prompts, configurations, annotations, feedback history, workflows, and logs?

If not, you are not running a pilot. You are walking into lock-in with nicer language.

7. Watch how they behave when you slow the deal down

Good vendors tolerate scrutiny.

Bad vendors punish it.

When you ask technical questions, do they bring in someone technical? When you ask for security documents, do they send them? When you ask for references, do they offer relevant ones? When you ask for a pilot, do they help you define success?

Or do they keep pushing urgency?

This pricing is only valid until Friday.

Our implementation team has limited capacity.

Most customers just start with the annual plan.

Maybe true. Also maybe pressure.

You are not buying a landing page. You are buying dependency.

A vendor that gets difficult before you sign usually does not get easier after you sign.

8. Call someone technical before you sign

If you do not have a CTO on staff, call someone who is one.

Show them the pitch deck, contract, architecture diagram if there is one, security answers, and pilot proposal.

They will catch in 15 minutes what you might miss in four sales calls.

The cost of that conversation is usually small. The cost of skipping it can be the full contract, six months of distraction, and a vendor nobody properly checked.

This is one of the fastest ways a fractional CTO pays for itself.

Not by making the founder feel technical. By catching the expensive mistake before it becomes a roadmap item.

The simple rule

AI vendor due diligence is not about being cynical.

It is about forcing the conversation away from promises and toward evidence.

Show me the failure modes.

Show me the eval method.

Show me the data handling.

Show me the exit path.

Show me the team that can ship this.

The real vendors can handle those questions. The weak ones will try to escape them.

That is the checklist.

If you are about to sign an AI vendor and want a second pair of technical eyes on it, that is a call I take for free before any engagement starts.