Evaluating Vendor Solutions for Digital Transformation

We’ve entered the era of “AI vaporware”. Big claims, fragile tech, and minimal insight into the data that powers it. If you're a B2B buyer, read this 👇 before you invest $50,000/yr on fancy new AI tech: We all know how quickly the tech landscape can shift. Just a few weeks ago, Xandr (a $1B DSP used by some martech platforms) suddenly shut down. Not because it wasn’t working. Microsoft simply sunset it to focus on its own advertising ecosystem and first-party data strategy. Now we’re seeing a new wave of risk: this time, dressed up as AI innovation. Fast launches. Flashy claims. Shaky foundations. But with AI, it's 10x faster. "AI-powered!" everyone screams. Sure. But powered by what? Trained on what? Is it built to last, or built to raise a Series F? If you're evaluating new AI vendors, here are the questions I'd ask before signing on the dotted line (shout out to Chad Holdorf): 1. Model & Intelligence - Can I trace how the model makes decisions? - What training data was used? Is it proprietary or public? - How is model performance tracked and improved? - Can models be tuned or retrained for our use cases? 2. Infrastructure & Ownership - Who owns the infrastructure and hosting? - What happens if the provider changes cloud vendors or LLMs? - Is it multi-cloud or locked to one ecosystem? 3. Security & Compliance - How is data handled? Is it encrypted at rest and in transit? - Does it meet our compliance standards (SOC 2, GDPR, etc)? - Can I audit or delete my data? 4. Integration & Extensibility - Can it connect to my tools (CRM, MAP, CDP)? - Does it expose APIs for other systems to use? - Is there a roadmap for more ecosystem support? 5. UX & Governance - How do users interact with it—chat, UI, workflow? - Are there guardrails for bad outputs or hallucinations? - Who controls permissions, access, and audit trails? 6. Business Impact - What metrics or outcomes has it improved for others? - Can it reduce cost, increase speed, or drive revenue? - Does it scale across teams or stay in a silo? Remember... “AI-first” without infrastructure is just AI branding. If the tech is built on weak systems, the smartest model in the world can’t save it.

Once companies have identified their AI opportunities, the first question they ask is: "Should we build custom or buy off-the-shelf?" Most large companies I speak with think they need a custom build. But in reality, there may already be a tool that fits their needs. Here's the framework I use to evaluate off-the-shelf solutions: (Across 5 key dimensions) 1. Costs A $30 per user per month fee seems reasonable now. But how affordable is it at scale? I recently worked with a startup that would have paid $7,000/month just in API calls for a basic recommendation engine. Always calculate projected costs at scale before committing. 2. Business risks You must understand how the underlying algorithms of specific AI tools make decisions. - What variables is it using in making decisions? - Is it using demographic data that could perpetuate specific types of biases?  - Or is it strictly using task-relevant variables? For example, many companies now use AI-powered hiring tools to streamline recruitment. However, these systems can unintentionally inherit biases from historical hiring data, reflecting and amplifying past patterns that favor certain demographics over others. A serious risk for all sorts of hiring decisions. 3. Model quality Key questions I ask vendors: - How often is the model retrained? - How do you monitor for degradation? - What testing ensures consistent performance? Which us leads to the next criteria. 4. Usability and performance Always test vendor solutions on YOUR data before purchasing. Their published accuracy rates may look impressive, but performance can vary significantly when applied to your specific use case. Take it for a test drive to make sure it satisfies your features and accuracy requirements. 5. System compatibility Will it integrate smoothly with your existing systems? Some solutions require specialized infrastructure or complex integrations that can create additional costs and technical debt. Bottom line: Don't skip proper evaluation because a tool seems convenient or popular. Testing different vendors gives you an apples-to-apples comparison. But you must know what to look for.

How do I recommend you evaluate a potential technology provider? Over the years, I've focused on six key dimensions when completing my due diligence of any tech provider: 1. Requirements vs. Vendor Capabilities - I typically recommend a customized RFP based on customer / user requirements, followed by some limited scope POC. If the vendor is unwilling to support a POC, then I recommend disqualifying them. - Engagement with end users during RFP process is critical, which includes involving them in some form of vendor scoring exercise. - Ensure you also engage all the necessary IT stakeholders from an architecture and security perspective as engaging them too late could throw a big wrench into your process. - Criticality of this dimension: 30% of total vendor score 2. Business Value and Pricing/License Options - Will the vendor help you build a business case to justify a spend? If not, this is a huge red flag. - Ensure your financial evaluation method aligns to your CFO's preferred method (TCO vs. ROI vs. NPV vs. IRR, etc.) - License restrictions, exit options, pricing levers - Is the vendor pricing roughly aligned to how others price? - Does the pricing / licensing model support flexibility for future growth? - Criticality of this dimension: 25% of total vendor score 3. Roadmap & Strategic Alignment - Does the vendor have a well-articulated roadmap, and does it align to how you see your requirements evolving in the future? - Does the vendor roadmap align to where you see the market heading? - Does the vendor solution, and their roadmap align to your long-term data and IT strategies? - Criticality of this dimension: 10% of total vendor score 4. Market Feedback - customer testimonials and references - analyst reviews - peer insights, reviews, social media - Criticality of this dimension: 15% of total vendor score 5. Ongoing Support - what is the vendor support model? - do you have an assigned customer success manager? - how dedicated is the vendor to your success? - Criticality of this dimension: 10% of total vendor score 5. The overall vendor 'vibe' - difficult to quantify and beware of biases, but also listen to your gut - does your experience with the vendor feel like a partnership, or a transaction? - how important is your success to the vendor? - Criticality of this dimension: 10% of total vendor score What have I missed? What else would you add? #cdo #chiefdataofficer #rfp

Choosing the wrong AI vendor can harm your business. Overhyped tech. Hidden costs. With hundreds of AI vendors pitching solutions, you need to know which ones are truly built to deliver value. Choosing the wrong vendor can result in wasted budgets, security risks, and stalled momentum. That’s why a structured, strategic evaluation process is essential. Here are 20 critical questions, grouped into five key categories, to help you assess AI vendors with clarity and confidence: 1. Technology & Capabilities ↳ What AI models and frameworks power your platform, and how often are they updated? ↳ How does your solution handle unstructured data like images or audio? ↳ Can the AI system be customized for our specific use cases? ↳ What level of transparency do you offer regarding model decisions? ↳ How do you measure and maintain accuracy over time? 2. Data Privacy & Security ↳ What data privacy standards do you comply with (e.g., GDPR, CCPA)? ↳ How is customer data stored, encrypted, and accessed within your platform? ↳ Can we retain ownership and control over our data and outputs? ↳ What protocols are in place to handle data breaches or AI misuse? ↳ Is customer data ever used to train or improve your models? 3. Integration & Usability ↳ What systems does your platform integrate with out of the box? ↳ What does the onboarding and training process look like? ↳ Is your solution usable by non-technical team members? ↳ How do you support cross-functional workflows or multi-department collaboration? ↳ What is the typical timeline to see value after implementation? 4. Support & Service ↳ What kind of technical support is available (e.g., live chat, dedicated rep)? ↳ Are there SLAs in place for uptime and issue resolution? ↳ Do you provide onboarding, documentation, and continued training? ↳ How frequently do you update the platform, and how are users informed? ↳ Is there a user community or partner ecosystem to tap into? 5. Pricing & Scalability ↳ What is your pricing structure, and how does it scale with usage or seats? ↳ Are there hidden fees for features like API access or integrations? ↳ Can your platform scale with our business needs over the next 3–5 years? ↳ What is the minimum contract length, and are there options for pilot programs? ↳ How do you measure ROI for clients? The AI landscape is evolving rapidly. Use these questions as a framework to cut through marketing gloss, clarify value, and build AI partnerships that serve your business. Which of these questions do you find most valuable? Share below 👇   ♻️ Repost if your network needs this mindset shift. Follow Carolyn Healey for more AI content.