Welcome to Attorney Intelligence, where we break down the biggest advancements in AI for legal professionals.

This week, we consider how law firms are demonstrating the return on investment (ROI) associated with new AI programs—a challenging metric to quantify, as it turns out.

Law firms are under pressure to demonstrate ROI from AI

I recently participated in a discussion with law firm leaders from Mintz, Paul Weiss, and Foley Hoag. The discussion centered on how larger law firms, under pressure to demonstrate the true value of recently deployed AI programs, are tracking ROI. Turns out, tracking ROI is easier said than done.

Adoption is a poor metric

Currently, most firms have landed on adoption as the key metric for tracking ROI. But without additional parameters—for example, conducting ongoing monitoring, differentiating users based on teams or seniority, or tracking usage of specific program features—adoption rates alone provide little insight. If everyone at a firm has logged into a program but no one uses it, arguably, the ROI on that program is nil.

Client feedback is more useful, but links to AI are less clear

Some firms have begun collecting client feedback on the timeliness and quality of service of attorneys armed with AI. Relative to adoption, this metric is likely more useful in gauging how an individual attorney’s workflow has changed since they began using AI effectively—and how that change manifests for the client. However, from the subjective perspective of the client, it’s difficult to isolate the productivity change from AI from the myriad other dynamics that influence attorney-client interactions. If attorneys are anything like us, the quality of their responses could be conditional on whether they made it to lunch that day.

Collecting data at the case-matter level is the most promising

Likely the most robust model for determining the ROI on AI products involves collecting very granular data on a case-matter level. That, in turn, requires that firms develop new codes that are specific to each practice area: For example, creating a specific label for a stock sale managed by a firm’s M&A practice. Over time, the hope is that collecting data at the case-matter level will enable firms to leverage that data to better forecast rates and alternative fee arrangements.

However, this model demands significant work by a firm’s in-house technology and analytics teams, especially on the front-end—resources not available to all law firms. When a similar model was presented at the FutureLawyer USA conference a couple of weeks ago, panelists acknowledged the resource-intensive nature of their proposal, implying that it might not be replicable in firms with leaner administrative resources.

Clients will challenge firms to quantify ROI

Managing partners and innovation leaders are not the only agents pushing for a more accurate quantification of the ROI on AI products. Clients are increasingly asking for more context—in other words, data—around attorneys’ rates, as well as continuous updates on time spent on relevant matters. That means demonstrating ROI on AI programs will continue to be priority for tech-focused law firms, not only as a means to streamline internal operations, but as a matter of customer service.

Legal bytes

Here are the latest updates in legal tech and AI:

OpenAI updates its user policy - OpenAI updated its user manual to state that individuals should not rely on ChatGPT for legal or medical advice, although the behavior of the chatbot remains unchanged (The Verge).

Amazon signs $38 billion deal with OpenAI for Nvidia Chips - OpenAI will pay Amazon Web Services for access to hundreds of thousands of Nvidia GPUs as part of the seven-year deal (Bloomberg Law).

DeepJudge closes $41 million in Series A funding - Legal tech tool DeepJudge, which builds customized indices of law firms’ data, closed a $41 million Series A round (Forbes).

Looking to improve your firm's efficiency?

One of the best ways to quantify ROI is by analyzing time entry data. Fortunately, PointOne is designed to facilitate meaningful analyses of time sheet data for leading law firms.

Book a demo to see how PointOne can stop revenue leakage and drive insights across your practice.

Until next week,

Katon