Robert Ambrogi Blog
Yesterday, Allen & Overy, one of the world’s largest law firms, announced that it had integrated the legal artificial intelligence product Harvey into its global practice, where it will by used by more than 3,500 lawyers across 43 offices operating in multiple languages.
I wrote about Harvey in November, when the previously stealth startup emerged with news that it had raised $5 million in funding led by the startup fund of OpenAI, the company that is the developer of the GPT AI technology. Harvey uses the GPT-3 technology (not ChatGPT) to enable lawyers to create legal documents or perform legal research by providing simple instructions using natural language.
Allen & Overy (A&O) now describes itself as “the first law firm to use generative AI that’s based on OpenAI’s GPT models.” While that appears to be true, it also true that others are soon to follow.
In a conversation yesterday with Harvey’s two founders, Winston Weinberg, formerly an associate at law firm O’Melveny & Myers, and Gabriel Pereyra, formerly a research scientist at DeepMind and a machine learning engineer at Meta AI, they told me that they are working with other firms that are similarly preparing to deploy Harvey.
They are not ready to name names and will leave it to the firms to make their own announcements to clients and the public. But they said that some of the firms will deploy it for specific practices, while others, as A&O did, will deploy it firmwide.
“I think other firms will adopt it in piecemeal, while others might also follow A&O’s path, so we’re letting these firms lead it and just working closely with them to develop this in parallel with them,” Pereyra said.
‘Game Changer’
A&O said it began a trial of Harvey in November. By the end of the trial, some 3,500 A&O lawyers had asked Harvey around 40,000 queries for their day-to-day client work.
Pereyra and Weinberg added that A&O had used Harvey in 50 different languages and 250 practice areas across all of its offices. “It’s almost a fourth of the firm daily and about 80% of the firm monthly,” Pereyra said.
David Wakeling, A&O partner and head of its Markets Innovation Group, which led the Harvey trial, said the technology is a game-changer.
“I have been at the forefront of legal tech for 15 years but I have never seen anything like Harvey,” Wakeling said. “It is a game-changer that can unleash the power of generative AI to transform the legal industry. Harvey can work in multiple languages and across diverse practice areas, delivering unprecedented efficiency and intelligence. In our trial, we saw some amazing results.”
Fine Tuning by Firm
Pereyra and Weinberg said that they will continue to roll out Harvey on a firm-by-firm basis, as the model requires them to fine tune it for each firm.
“We develop these different systems for specific use cases and for specific firms, so you’ll have the specific model for A&O which becomes fine tuned for them,” Pereyra said. “You can even specialize it more than that, where you can get specific models for cases — you can have a case where you can have a specific client matter or specific litigation and the model is fine tuned for that litigation or transaction.”
Pereyra and Weinberg said that Harvey is trained over at least three types of data. It starts with the general internet data that underlies the GPT model. Harvey is then further trained against general legal data, such as case law and reference materials. Finally, it is fine tuned against the law firm’s own data, such as its historical work product, templates, and the like.
In addition, it can be further fine tuned using data for a specific matter or client.
To respect confidentiality, once Harvey is trained for a specific firm, that model stays specific to that firm and is not used as a base model when deploying Harvey at a different firm.
“We firewall it based on the firm,” Pereyra said. “Any training that A&O does on their model just makes their model better. And that’s kind of the point. Right? It’s what makes A&O a unique law firm — the feedback from their attorneys is going to make their model look significantly better, or at least different, than another firm.”
Even within a firm, the two founders said, there will be different models protected by different sets of permissions and firewalls.
“You can’t just take all of A&O’s client matters and use the same model because you don’t want leakage, even within the same firm, across client matters, so we’re being extremely careful of firewalling off all of these different things,” Weinberg said.
Pereyra analogizes the training process to that of a law firm associate. Think of OpenAI’s training of the base model as K-12 schooling. Then Harvey puts the model through law school using general legal data. But even a law school graduate does not know how a particular firm works. Harvey’s firm-specific training is like teaching an associate the firm’s unique practice.
AI without Hallucinations
Some users of ChatGPT for legal purposes have noticed its tendency to “hallucinate” — to make up answers from whole cloth. Pereyra and Weinberg say that Harvey’s method of fine tuning the AI dramatically reduces occurrences of hallucinations and, in highly context-specific applications, eliminates them almost entirely.
For contract review, for example, Harvey is able to reduce hallucinations “basically to zero.” In fact, Pereyra said, the error rate is lower than for review by a contract attorney.
Meanwhile, Pereyra and Weinberg say they are still not demonstrating the product publicly. But they expect to have several new announcements of partnerships and capabilities coming within the next few months.
“We’re trying to not scale too fast just because we want to make sure that we’re giving enough time to each firm.”