The AI vendor selection playbook
How to choose high quality AI vendors
Adrian Parlow
July 10, 2025
Welcome to Attorney Intelligence, where we break down the biggest advancements in AI for legal professionals every week.
Artificial Lawyer recently covered how three major firms navigated the minefield of AI vendor selection:
CRS & Harvey: A full-on bake-off with 5 vendors, 45 AI superusers, and 6,000+ pieces of feedback across 9 criteria.
Dentons & Legora: No details on testing, just a quiet pivot to a vendor that promised speed and efficiency for routine legal tasks.
DLA & Newcode: No bake-off, just deep integration - Newcode embedded a team member directly into DLA’s offices to co-build custom agent workflows.
Each took a different route, with varying levels of end-user involvement, technical rigor, and long-term commitment. The benefit of something like the CRS bake-off is clear: you get hands-on insights across use cases and user types.
However there’s a trade-off to all of it: getting your lawyers to dedicate meaningful time to multiple tools is no small ask, and usually a costly one.
So what’s a better approach?
In this week’s Attorney Intelligence, we’ll explore:
Why bake-offs might reveal today’s best tool but not tomorrow’s winner
How to factor in team quality, roadmap, and velocity, alongside features
A hybrid approach to picking AI vendors that keeps you covered short and long term
What firms can learn from tech investing when placing their AI bets
Let’s get into it.
The Traditional Playbook…and Why It’s Not Enough
Historically, firms rely on a handful of familiar steps when evaluating legal tech: watch a demo, call a reference, do some internal testing, and maybe run a 6- to 12-month pilot with a small user group.
It’s structured, but limited.
Most decisions come down to whether the product can solve a clear problem today, not whether it will solve the problems of tomorrow.
That’s where the bake-off approach comes in.
It’s thorough. But it’s also time and resource-intensive, especially in a legal environment where attention is already at a premium.
Bake-Offs vs. Betting on the Future
In theory, bake-offs sound perfect, you:
Test vendors head-to-head.
Collect real usage data.
Pick the winner based on live performance.
But in practice, this can be misleading.
The AI tools you evaluate today may not resemble what they’ll look like in 12 months, let alone 3 years.
That’s particularly true in AI, where the product is often evolving weekly. Judging vendors only by today’s performance ignores one key truth: this race is long.
And what looks like a slight edge now could disappear fast if a competitor is shipping faster or building smarter.
The Underrated Factor of Founders, Teams, and Trajectory
The smartest firms are starting to treat vendor selection more like venture capital.
Alongside evaluating products, they’re also evaluating the people and vision behind them.
Who's building this?
What’s the product velocity?
How often do they ship?
Are they building with real user feedback or just chasing buzzwords?
Are they backed by funding and runway or are they duct-taping a demo for the next sale?
These questions don’t show up on a demo, but they’re often more predictive of long-term success than any feature comparison.
A Balanced Approach
It’s tempting to go all-in on the flashiest demo or the most polished UX. But legal tech buyers should think in horizons.
Short-term: Can we use this tool right now and get value from it?
Mid-term: Do we trust the team to keep improving this rapidly?
Long-term: If we embed this deeply, are we confident this company still exists and thrives five years from now?
The firms getting it right are taking the time to meet the teams, understand the culture, and track how often they ship. Their buying process is focused long-term bets.
Buy Like an Investor, Not a Shopper
Choosing a legal AI vendor today is a lot like investing in a startup: you're not just buying what’s in front of you, you’re betting on what it could become.
That’s why process matters. Bring in your users, but don’t over-index on short-term usability. Watch the product roadmap, but don’t ignore who's steering it. And above all, treat selection as a strategic investment, as opposed to just a procurement process.
When you get it right, yes you’re not just getting a tool. But you’re also getting a partner.
Legal Bytes
Here are the latest updates in legal tech and AI:
Bloomberg published inaugural Leading Law Firms survey, revealing what defines success among the top firms in the world
Freshfields partnered with King’s College to offer future trainees legal tech masters plus £20k grant
American Association of Law Libraries published their 2025 State of the Profession report, documenting how law libraries are navigating AI
