Notes from LEGALWEEK 2024  “Better Call GPT” – comparing Large Language Models against lawyers and legaltech providers

12 February 2024 — As we noted in our first post-event analysis, AI dominated LEGALWEEK 2024.

But, as we also noted there was one good thing: the “AI hype buzz” was muted, at least in the better sessions. The deployment of generative AI is full of hazards with the usual suspects … “hallucinations” and other misleading or erroneous outputs generated by LLMs … getting a lot of air time.

As one of eDiscovery’s leading lights/Master Sensei told us:

This stuff is far, far away from the safe prime time usage stage. Good for some backroom admin, and research, and consumption stuff, and some straight mechanical uses. But for the important cognitive stuff – no. When you spend more of your conference time talking about AI errors, risks and guardrails you have something not ready for the main stage. Not the legal industry stage anyway.

One of those “mechanical uses” is contract lifecycle management (CLM) and there were plenty of vendors selling all types of AI tools to improve contract drafting and negotiation, and contract review. It was demonstrated in vendor booth, after vendor booth, after vendor booth that generative AI could, in particular, speed up adoption momentum for drafting and review tools.

Note to readers: later this week we’ll have a video interview from LEGALWEEK 2024 with Beom Jin Kim of BHSN Legal who will give you an overview of CLM, and explain how his company’s use of genAI offers a streamlined solution for creating, negotiating, approving and reviewing contracts to save businesses time and reducing their risk of lost or mismanaged agreements.

One of our favorite arXiv pre-prints which was widely referenced at LEGALWEEK 2024 (there were many pre-prints/white papers at the event and we’ll try to comment on the key ones over the next few weeks, as our time allows) was “Better Call GPT, Comparing Large Language Models Against Lawyers” (there is a full link below) which was a comparative study between Large Language Models (LLMs) and traditional legal contract reviewers — senior and junior lawyers, and Legal Process Outsourcers (LPOs). The researchers examined:

– whether LLMs can outperform humans in accuracy, speed, and cost-efficiency during contract review

– benchmarks to use in evaluating LLMs against a “ground truth” set by senior lawyers

– uncovering advanced models that match or exceed human accuracy in determining legal issues

The general conclusion of the study:

Advanced models match or exceed human accuracy in determining legal issues. In terms of speed, LLMs complete reviews in mere seconds, eclipsing the hours required by their human counterparts. Cost-wise, LLMs operate at a fraction of the price, offering a staggering 99.97% reduction in cost over traditional methods [aka humans].

Needless to say, this could herald a seismic shift in the legal industry. The AI models were wildly cheaper and more than ten times faster. A fully-blown piece of legal tech could incorporate multiple layers of LLMs with traditional techniques and likely reduce the error rate to below the junior lawyer level — while retaining a two-order-of-magnitude cost advantage.

In speed, LLMs completed reviews in mere seconds, eclipsing the hours required by their human counterparts.

Cost-wise, the results were astounding. LLMs operate at a fraction of the price, offering a staggering 99.97 percent reduction in cost over traditional methods.

Note to readers: the hourly rates for lawyers were based on inside counsel rates determined through industry benchmark reports such as ACC’s 2023 Law Department Compensation Survey and external counsel rates determined by market data held by Onit Inc. The costs for LLMs were determined through commercial pricing provided by the service suppliers.

We’ll end with a quote from the study:

These results are not just statistics – they signal a seismic shift in legal practice. LLMs stand poised to disrupt the legal industry, enhancing accessibility and efficiency of legal services. Our research asserts that the era of LLM dominance in legal contract review is upon us, challenging the status quo and calling for a reimagined future of legal workflows.

The study deserves your full read and can be accessed by clicking here.