AI Vendor Selection: The 12 Questions You Must Ask Before Signing

Your inbox is full of them. Polished slide decks from AI vendors promising to transform your business with machine learning, generative AI, and intelligent automation. The demos are impressive. The case studies are compelling. The pricing page is suspiciously vague.

Here is the uncomfortable truth: most enterprise AI vendor relationships end in disappointment. A 2025 Gartner survey found that 54% of organizations that purchased AI solutions from external vendors reported that the delivered product did not meet the expectations set during the sales process. More than a third experienced unexpected costs that exceeded the original quote by 40% or more.

The problem is not that AI vendors are dishonest. Most are building genuinely useful products. The problem is that buyers ask the wrong questions. They focus on features and pricing while ignoring data ownership, model transparency, lock-in risk, and what happens when things go wrong. By the time these issues surface, the contract is signed, the integration is deep, and switching costs are enormous.

This guide gives you the 12 questions that separate informed buyers from regretful ones. Each question includes the answer you want to hear, the red flag answers that should make you pause, and the business context for why it matters. Whether you are evaluating an LLM API provider, a vertical AI SaaS product, or an ML platform, these questions apply.

The AI Vendor Landscape: What Are They Actually Selling?

Before diving into questions, you need a mental model for the AI vendor landscape. Not all AI vendors are the same, and the risks vary dramatically by category.

Each category creates different dependencies. An LLM API provider locks you into their model behavior and pricing. A vertical AI SaaS product locks you into their data model and workflow. An ML platform locks you into a cloud ecosystem. Understanding which category you are buying from determines which questions matter most.

The 12 Questions You Must Ask

These questions are ordered by impact. The first four are deal-breakers. Questions five through eight are negotiation leverage. Questions nine through twelve are long-term protection. Skip none of them.

Question 1: Who owns the data I put into your system?

This is the single most important question in any AI vendor relationship. It sounds simple, but the answer is almost never straightforward.

What you want to hear: "You retain full ownership of all input data and output data. We do not use your data to train our models. We do not retain your data beyond the processing window. We can provide a Data Processing Agreement (DPA) that specifies this in legally binding terms."

Red flag answers:

• "Your data helps improve the model for everyone." This means your proprietary data is being used to train a shared model. Your competitive intelligence becomes a training signal for your competitors.
• "We anonymize and aggregate." Anonymization is weaker than most vendors claim. Research consistently demonstrates that anonymized datasets can be re-identified, particularly when combined with external data sources.
• "It's in the Terms of Service." If the sales team cannot explain the data ownership terms in plain language, the terms are not in your favor.

Question 2: What happens to my data if I cancel the contract?

Data portability is the escape hatch. Without it, you are not a customer. You are a hostage.

What you want to hear: "Upon cancellation, we provide a full data export in standard formats (CSV, JSON, Parquet) within 30 days. After export confirmation, we permanently delete all your data from our systems, including backups, within 90 days. We provide a certificate of deletion."

Red flag answers:

• "We can provide a data dump." A dump is not an export. Dumps are often in proprietary formats that require the vendor's tools to read.
• "Your data is retained for compliance purposes." This is sometimes legitimate (financial services, healthcare), but the retention period should be explicit and minimized.
• No answer at all. If a vendor has not thought about data portability, they have not thought about your long-term interests.

Question 3: What is your actual uptime SLA, and what happens when you miss it?

AI services have fundamentally different reliability characteristics than traditional SaaS. A database can offer 99.99% uptime because the failure modes are well-understood. An LLM inference endpoint depends on GPU availability, model loading times, and computational complexity that varies by request.

What you want to hear: "Our SLA is 99.9% for API availability, measured monthly. Latency SLA: p95 response time under 2 seconds for standard requests. If we miss either target, you receive service credits of 10% per 0.1% below the SLA, up to 30% of monthly spend. Credits are applied automatically without requiring a support ticket."

Red flag answers:

• "99.99% uptime." For AI inference services, this is almost certainly aspirational, not contractual. Ask for their actual trailing 12-month uptime.
• "Best-effort latency." If latency is not in the SLA, you have no recourse when inference calls take 30 seconds during peak load.
• "Credits require a support ticket." This means you will never file for credits, and the vendor knows it.

Question 4: Can I see the model architecture, training data sources, and evaluation benchmarks?

Model transparency matters for two reasons: regulatory compliance and debugging. If an AI model makes a wrong decision that costs your company money, you need to understand why. If a regulator asks how your AI works, "we use a vendor's black box" is not an acceptable answer.

What you want to hear: "We publish model cards for every model version, including training data composition, known biases, evaluation benchmarks on standard datasets, and performance degradation curves. For enterprise customers, we provide access to model interpretability tools and can walk your team through specific predictions."

Red flag answers:

• "That's proprietary." Some level of IP protection is understandable, but a complete refusal to discuss model behavior is a risk you are accepting blindly.
• "We use the latest GPT/Claude/Gemini." This means the vendor is a thin wrapper around a foundation model. You could build this yourself, often more cheaply and with more control.
• "Our AI is 97% accurate." Accuracy without context is meaningless. Accurate on what dataset? Measured how? Against what baseline?

Question 5: How do you handle model updates, and can I pin to a specific version?

AI models change. Vendors retrain, fine-tune, and update their models regularly. Each update can change behavior in ways that break your downstream workflows. A model that correctly classifies customer complaints today might misclassify them tomorrow after an update.

What you want to hear: "We version all models and provide at least 90 days notice before deprecating any version. You can pin to a specific model version for up to 12 months. We provide a staging environment where you can test new model versions against your data before migrating."

Red flag answers:

• "We continuously improve the model." Continuous improvement without version control means your production system can change behavior at any time without your knowledge.
• "You'll always get the latest version." This is a feature for consumers. It is a risk for enterprises.

Question 6: What is the total cost at 10X our current volume?

AI pricing is designed to look cheap at proof-of-concept scale and become expensive at production scale. Token-based pricing, per-seat licensing, and usage tiers all have inflection points where costs accelerate.

What you want to hear: "Here is a pricing calculator with volume tiers. At 10X your current volume, the per-unit cost drops by 35%. We cap annual price increases at 5%. Here is a breakdown of all costs: API calls, storage, support, and any overage charges."

Red flag answers:

• "Contact sales for enterprise pricing." This means the price is whatever they think you will pay.
• "We'll work with you as you scale." Translation: we will renegotiate from a position of strength once you are dependent on us.
• No published pricing at all. Opacity in pricing correlates strongly with unexpected costs.

Question 7: What is your security and compliance posture?

AI systems process some of the most sensitive data in your organization: customer records, financial transactions, strategic documents, proprietary code. The security bar must be higher than for traditional SaaS.

What you want to hear: "SOC 2 Type II certified, audited annually by a Big Four firm. Data encrypted at rest (AES-256) and in transit (TLS 1.3). We support customer-managed encryption keys. We offer dedicated tenancy for enterprise customers. Here is our penetration test summary from the last 12 months. We are GDPR compliant and can sign a DPA. For HIPAA-regulated customers, we offer a BAA."

Red flag answers:

• "SOC 2 Type I." Type I is a point-in-time assessment. Type II covers a sustained period, usually 6 to 12 months. Type I means they passed a snapshot. Type II means they maintain controls consistently.
• "We're working on it." If a vendor processing your sensitive data does not have SOC 2 Type II, they are behind the minimum bar for enterprise procurement.
• "We use AWS/GCP/Azure, so we inherit their certifications." Cloud provider certifications cover infrastructure. They do not cover the vendor's application, data handling, or access controls.

Question 8: What does your exit clause look like?

Every vendor relationship should have a clean exit path. The exit clause determines whether you are entering a partnership or a prison.

What you want to hear: "Annual contracts with 90-day termination notice. No early termination penalties after the first year. Full data export within 30 days of notice. We provide a migration assistance period of 60 days at no additional cost. All integrations use standard APIs, so replacement vendors can connect to the same endpoints."

Red flag answers:

• "Three-year minimum commitment." Long commitments reduce the vendor's incentive to keep you happy and increase your switching costs.
• "Early termination fee of remaining contract value." This means you pay for the service whether you use it or not.
• "We can discuss exit terms later." If they will not discuss it now, they will not make it easy later.

Question 9: How do you handle model drift and performance degradation?

AI models degrade over time. The world changes, data distributions shift, and what was accurate six months ago becomes unreliable. This is not a bug. It is a fundamental characteristic of machine learning systems.

What you want to hear: "We monitor model performance continuously against customer-specific benchmarks. We provide a dashboard showing accuracy, latency, and confidence score distributions over time. When performance drops below agreed thresholds, we proactively alert you and provide a remediation timeline. We retrain models quarterly at minimum, using updated data that you approve."

Red flag answers:

• "Our models don't drift." Every model drifts. A vendor who claims otherwise does not understand their own product.
• "We monitor internally." If you cannot see the monitoring data, you cannot verify their claims.

Question 10: Can I run your solution in my own environment?

For many enterprises, sending data to a vendor's cloud is a non-starter. Regulated industries, defense contractors, and organizations with strict data residency requirements need on-premises or VPC deployment options.

What you want to hear: "Yes, we offer on-premises deployment, private cloud deployment, and VPC-hosted options. The on-premises version has feature parity with our cloud product, with updates released within 30 days of cloud releases. We provide deployment automation (Helm charts, Terraform modules) and ongoing operational support."

Red flag answers:

• "Cloud-only." Depending on your industry, this may be a hard blocker. Even if it is not today, regulatory changes could make it one tomorrow.
• "On-prem is available but limited." Feature gaps between cloud and on-premises deployments tend to grow over time, not shrink.

Question 11: What is your company's financial position and runway?

The AI industry is in a funding boom. Many vendors are burning cash at rates that are not sustainable. If your AI vendor runs out of money, your production system goes down with them.

What you want to hear: "We are profitable (or have 24+ months of runway at current burn rate). Here are our most recent revenue growth numbers. We have [specific number] of enterprise customers on multi-year contracts. Our largest customer represents less than 15% of revenue (concentration risk is low)."

Red flag answers:

• Refusal to discuss financials. For a vendor who will become a critical dependency, financial opacity is a risk factor.
• "We just raised our Series A/B." Early-stage funding is not stability. The mortality rate for AI startups between Series A and Series C is significant.
• Rapid team turnover on LinkedIn. High attrition at an AI vendor often precedes product instability.

Question 12: Can I talk to three customers who look like me?

Reference checks are the most underused tool in vendor evaluation. Every vendor can provide cherry-picked case studies. Customer references let you hear the unscripted version.

What you want to hear: "Absolutely. Here are three customers in your industry, at a similar scale, who have been using our product for at least 12 months. We will make warm introductions."

Red flag answers:

• "We can share case studies." Case studies are marketing. References are diligence.
• "Our customers prefer not to be contacted." Some customers do have policies against reference calls, but if a vendor cannot produce any references, they may not have customers who are happy enough to recommend them.
• All references are from the last three months. You want customers with at least 12 months of experience. That is when the honeymoon period ends and real issues surface.

Understanding Lock-In: The Four Dimensions

Vendor lock-in in AI is more complex than in traditional software. There are four distinct dimensions of lock-in, each requiring different mitigation strategies.

Data lock-in is the most dangerous. If your vector embeddings are generated by a vendor's proprietary model, switching vendors means re-embedding your entire corpus. If your training data is stored in a vendor's format, migration requires transformation. If your data is enriched by vendor-specific features (entity extraction, classification labels), those enrichments may not transfer.

API lock-in happens when you build against vendor-specific APIs rather than industry standards. OpenAI's function-calling format, Anthropic's tool-use schema, and Google's grounding API all work differently. Code written for one does not work with another without an abstraction layer.

Workflow lock-in is subtler. When your customer support team is trained on a specific AI copilot's interface, switching tools means retraining the team. When your data pipeline is built around a vendor's feature store, replacing the store means rebuilding the pipeline.

Knowledge lock-in is the most overlooked. When your engineering team has spent 18 months becoming experts in a specific ML platform, they have an institutional incentive to recommend staying with that platform, even when alternatives are objectively better.

Mitigating Lock-In

The mitigation strategy depends on the dimension:

• Data: Insist on standard formats. Store raw data in your own systems. Treat vendor-generated enrichments as derived data that can be regenerated.
• API: Build an abstraction layer between your application and the vendor's API. LiteLLM, Portkey, and similar tools provide a unified interface across multiple LLM providers. The cost of the abstraction layer is a fraction of the switching cost without one.
• Workflow: Document your processes in vendor-neutral terms. Ensure that the workflow logic lives in your code, not in the vendor's configuration.
• Knowledge: Require cross-training. If your team builds on Databricks, ensure at least one engineer maintains proficiency with Snowflake or BigQuery. This is an insurance policy.

Build vs. Buy: When In-House Wins

Not every AI capability needs to be purchased. In some cases, building in-house is cheaper, faster, and lower-risk. The decision depends on four factors.

Build when:

• The AI capability is a core differentiator for your business. If AI is your product, do not outsource the core.
• You have the engineering talent. A team of three experienced ML engineers can build and maintain most classification, NLP, and recommendation systems.
• Your data is too sensitive for any third party. Defense, intelligence, and certain healthcare applications have data that simply cannot leave your environment.
• The vendor's solution is a thin wrapper around an open-source model. If the vendor's value-add is primarily UX and hosting, you can often replicate it for less.

Buy when:

• The capability is commoditized. Speech-to-text, OCR, and basic sentiment analysis are solved problems. Building from scratch is reinventing the wheel.
• Time-to-market matters more than cost. A vendor can get you to production in weeks. Building takes months.
• The vendor has proprietary data or models that you cannot replicate. Some vertical AI companies have years of domain-specific training data that no amount of engineering can reproduce.
• You lack the talent and cannot hire it. ML engineering talent is scarce and expensive. A vendor amortizes that cost across many customers.

Structuring the Contract: Terms That Protect You

Once you have asked the 12 questions and decided to proceed, the contract is your last line of defense. Here are the non-negotiable terms that should be in every AI vendor contract.

Data Rights

• Explicit data ownership clause: "Customer retains all rights, title, and interest in Customer Data. Vendor shall not use Customer Data for any purpose other than providing the Services."
• No training on customer data: "Vendor shall not use Customer Data to train, improve, or fine-tune any model, whether used by Customer or any third party, without prior written consent."
• Data deletion upon termination: "Within 30 days of contract termination, Vendor shall delete all Customer Data from its systems, including backups, and provide a written certificate of deletion."

Performance Guarantees

• SLA with teeth: Uptime and latency targets with automatic service credits. Credits should be meaningful (10% to 30% of monthly spend), not token amounts.
• Model performance baselines: Define specific accuracy, precision, or recall thresholds at contract signing. If the model degrades below these thresholds, the vendor must remediate within a defined timeline.
• Regression testing rights: You should have the right to run your own benchmark suite against any model update before it is promoted to your production environment.

Exit Protection

• Termination for convenience: The ability to exit the contract with 90 days notice, without penalty, after the initial term.
• Data portability: Full data export in standard, open formats within 30 days of termination notice.
• Transition assistance: A defined period (60 to 90 days) where the vendor provides migration support at no additional cost.
• Escrow for critical dependencies: If the vendor's product is critical to your operations, consider requiring source code escrow that triggers in the event of vendor insolvency.

Pricing Protection

• Price caps: Maximum annual price increase of 5% to 7%. Without this, the vendor can price you into forced migration.
• Volume commitments: If you commit to a volume floor, insist on a corresponding price ceiling. Commitments should be bilateral.
• Overage clarity: Define exactly what happens when you exceed committed volumes. Per-unit overage rates should be specified in the contract, not determined "at the time of overage."

The Evaluation Process: A 30-Day Framework

Rushing vendor evaluation is how organizations end up with regretful contracts. A structured 30-day evaluation process gives you enough time to be thorough without creating analysis paralysis.

1. Week 1: Long List to Short List
   Identify 8 to 12 vendors through analyst reports, peer recommendations, and your own research. Apply the 12 questions as a screening filter. Eliminate any vendor that cannot answer questions 1 through 4 satisfactorily. You should be down to 3 to 4 vendors.
2. Week 2: Technical Evaluation
   Run a proof of concept with each short-listed vendor using your actual data (not their demo dataset). Measure accuracy, latency, and cost on your workload, not their benchmarks. Involve your engineering team in the evaluation, not just procurement.
3. Week 3: Commercial and Legal Review
   Request draft contracts from the top 2 vendors. Have legal review data ownership, exit clauses, and liability terms. Negotiate the non-negotiable terms listed above. Call customer references.
4. Week 4: Decision and Negotiation
   Select the winning vendor. Use the competing proposal as negotiation leverage. Finalize contract terms. Define success metrics for the first 90 days. Establish a quarterly business review cadence.

Nine Red Flags That Should End the Conversation

Some signals should stop the evaluation immediately. Do not waste time negotiating with a vendor that exhibits any of these behaviors.

1. They cannot explain their pricing model clearly. Complexity in pricing is not sophistication. It is a tax on your procurement team's attention.
2. They resist a production pilot. A vendor confident in their product welcomes testing. Resistance suggests they know the product underperforms outside controlled conditions.
3. Their security certifications are "in progress." SOC 2 and ISO 27001 take 6 to 12 months. If they started recently, they are at least a year away from maturity.
4. They pressure you with artificial urgency. "This pricing expires Friday" is a sales tactic, not a business reality. Reputable vendors do not manufacture urgency.
5. Their customer references are all from the last quarter. New customers are in the honeymoon phase. You need references who have lived with the product through at least one contract renewal.
6. They cannot produce a service-level agreement. An SLA is table stakes for enterprise software. Its absence signals either immaturity or an unwillingness to be held accountable.
7. Key personnel leave during your evaluation. Executive departures during a sales cycle often signal internal problems. Check LinkedIn for attrition patterns.
8. Their product roadmap is "exciting but confidential." A vendor that cannot share a high-level roadmap either has no roadmap or is pivoting and does not want you to know.
9. They dismiss your questions about lock-in. "Our customers love us, so lock-in isn't a concern" is the AI equivalent of "trust me." Your fiduciary duty is to plan for contingencies, not to rely on vendor goodwill.

The Bottom Line

AI vendor selection is not a technology decision. It is a business decision with technology implications. The model's accuracy matters, but so does the contract's exit clause. The API's latency matters, but so does the vendor's financial runway. The features matter, but so does the data ownership clause buried on page 47 of the terms of service.

The 12 questions in this guide are not a checklist to rush through. They are a framework for understanding the full scope of a vendor relationship before you enter it. The vendors who answer them well are the ones worth working with. The ones who deflect, obfuscate, or pressure you to skip due diligence are telling you something important about how the relationship will work once the contract is signed.

Ask the hard questions now. Your future self, the one who has to live with this vendor for the next two to three years, will thank you.