Notes from LEGALWEEK 2024 and the Berlin symposiums: “Prompt Engineering” – the monster emerging from the cybersecurity attack den

10 February 2024 — Last week we split the media team to cover events in two places: the cybersecurity symposiums and military intelligence conference in Berlin, Germany and LEGALWEEK 2024 in New York City.

We have attended LEGALWEEK for 18 years. It started in 1982 under the moniker “LegalTech” until the name change in 2017. It is the smallest technology event/conference we attend (by number of attendees and exhibitors) so it has provided a good training ground for our new media team members to get their feet wet on how to cover an event. Everything else we cover … the Consumer Electronics Show, the Frankfurt Book Fair, International Forum on Cybersecurity, the Mobile World Congress and the RSA Security Conferewnce … have attendees and exhibitors that number 120,000+/4,500+ and involves 3-4 member teams so we like to start our newbies out on smaller events.

But the LEGALWEEK event had so deviated from its original mission and had become so stale (“same old, same old”) we stopped attending in 2019. But as artificial intelligence has taken hold of so many industries over the last 2 years, we thought we’d revisit.

And just as AI dominated the Berlin events, AI dominated LEGALWEEK. There were 94 sessions/panels listed on the 4-day agenda, and 75 focused on/noted AI – which is 79% of the program. And this does not include the off-the-main-floor presentations and sessions by legal technology vendors.

But as we have noted, the legal technology / eDiscovery world always runs on cycles, on hype loops. We have had:

• “early case assessment”
• “managed services”
• “predictive coding”
• “cloud computing”

And now, artificial intelligence.

There were several “themes” or “hot topics” at last week’s event. One of them was cybersecurity, and cyber-threats.

More complex than ever, but unfortunately the event treated cybersecurity in a cursory manner, focusing on the organizational mantras: “awareness and communication”. And all the new cyber regulations out there. The focus was on staying aware of the mass of cybersecurity and privacy rules and regulations, particularly for organizations that operate on a global scale. Not about how the tubes and wires and networks and infrastructure of the web and platforms work and how cyber attackers take advantage of that, and how you can defend against/mitigate the effect of of attack.

But to be fair LEGALWEEK is a trade show, with vendors merely hawking their products and services. It’s not an educational forum. It’s not the International Forum on Cybersecurity (in Europe) or the RSA Security Conference (in the U.S.) where you can get into the nooks and crannies of cybersecurity and where many eDiscovery product and service providers (and lawyers and IT professionals) flock to get a real cyber education.

Which brings us to …

One good thing about LEGALWEEK this year was that the “AI buzz” was muted, at least in the better sessions. The deployment of generative AI is full of hazards with the usual suspects … “hallucinations” and other misleading or erroneous outputs generated by LLMs … getting a lot of air time. As one of eDiscovery’s leading lights/Master Sensei told us:

This stuff is far, far away from the safe prime time usage stage. Good for some backroom admin/research/consumption stuff, some straight mechanical uses. But for the important cognitive stuff – no. When you spend more of your conference time talking about AI errors, risks and guardrails you have something not ready for the main stage. Not the legal industry stage anyway.

We were somewhat amused at one session when a presenter said there is a clear parallel between the digital spreadsheet and generative AI: both are computer apps that collapse time. A task that might have taken hours or days can suddenly be completed in seconds. The right technology in the right place can take over very quickly indeed. In the time it takes to qualify as a chartered accountant, digital spreadsheets laid waste to a substantial industry of cognitive labor, of filling in rows and columns, pulling out electronic calculators and punching in the numbers. Accounting clerks became surplus to requirements, and the ability of a single worker to perform arithmetic was multiplied a thousandfold – and soon a millionfold – almost overnight.

His point was that generative AI does something similar on a grander scale, letting the humans deal with the big creative questions while the machine handles the nagging details. When a tool is ubiquitous, and convenient, we kludge our way through without really understanding what the tool is doing or why. And that, he said, as a parallel for generative AI, is alarmingly on the nose.

Except … very few people are capable of checking the outputs of a spreadsheet with the input data to see if the outputs make sense if the calculations are complex. It requires a lot of discipline to keep a spreadsheet honest. God help you if you use AI. It’s an inappropriate analogy. Spreadsheets are easily tested and corrected and have no ambition nor means to generate their own content. AI is materially different in that how it comes to its results isn’t easily verified by humans and that the whole machine learning process leads to results defying straightforward verification, even before musing about the underlying weighing of different input variables. In other words: there is no sumcheck function and the formulae governing individual results cannot easily be checked nor tested.

Taking all of this in, the potential for manipulating individual results or for that matter whole populations is vast, far transcending the potential damage from errors in simple spreadsheets.

But, hey, a solution!! A new skill set will be born, we were told … in session after session at LEGALWEEK … and that is “prompt engineer”. We were told that legal practitioners need to acquaint themselves with AI’s inner workings and “prompt engineering” will become a coveted skill – emerging as the nexus of machine intelligence and legal acumen. One session presenter told us “you just need to learn the right questions to ask ChatGPT. It’s simple, really“. Really? It is?

Already the debates have started on “how to best incorporate generative AI into law school curriculum and practice”. Debates swirl, from outright bans of generative AI tools to revisiting pedagogical approaches to embrace the technology’s potential.

And it gets … better or worse, just pick your appropriate word. Last month Seton Hall law professor David Kemp published an article in the Seton Hall Journal of Legislation and Public Policy which champions integrative approaches, drawing parallels to essential legal research tools such as Westlaw. His argument: we must institute AI competence. It is not only preemptive – but also ethically mandated.

NOTE TO READERS: as noted in a recent post by the American Bar Association, law schools like the University of Michigan and Suffolk University are already adapting, taking divergent yet equally proactive stances. While the former requests applicants to swear off AI assistance, the latter advocates for grounding future legal professionals in generative AI literacy from day one.

We get it, we get it. Lawyers will be “data wranglers”, and learn to be “prompt engineers” and wrestle that ornery beast – genAI – to the ground for the greater legal good.

But there are a lot of (bad) folks waaaaay ahead of you.

Some notes from Berlin:

Despite the growing concerns around deepfakes and phishing attacks powered by LLMs, let’s realize that the hype might be diverting attention from more significant vulnerabilities associated with GenAI at a higher, and perhaps unsolvable level.

As we discussed in Berlin, the cybersecurity community might want to shift their focus from the threats posed by GenAI to the threats against it, specifically from attackers aiming to exploit design weaknesses and flaws in these systems in corporations and law firms.

One of the most critical threats is prompt injection, a technique where attackers feed text prompts into LLM systems to elicit unintended or unauthorized responses. There are some badass “prompt engineers” who have this stuff down to a science.

Prompt injection operates similarly to prompt engineering but with malicious intent, tricking GenAI systems into revealing sensitive information or performing harmful actions. Attackers manipulate the system with repeated prompts, akin to social engineering, but targeting AI.

NIST (the National Institute of Standards and Technology, an agency of the United States Department of Commerce whose mission is to promote American innovation and industrial competitiveness) categorizes prompt injection into direct and indirect attacks, where attackers either input malicious prompts directly into the LLM or manipulate external information sources the LLM relies on. This can lead to various malicious outcomes, including denial-of-service, misinformation spread, or credential disclosure.

The real thriller is that multimodal GenAI systems, which respond to image prompts, are also vulnerable. Attackers can now use images embedded with text prompts to mislead these systems, blurring the lines between genuine instructions and malicious injections.

Prompt injection offers attackers diverse methods to exploit LLM vulnerabilities, from exposing system internals to bypassing content filters or extracting data. It’s likened to finding a backdoor into the AI’s “brain,” accessing proprietary training information or customer data.

This is not trivial stuff, as LLMs are increasingly integrated with sensitive data and critical systems in corporations, law firms, energy power grids, etc., etc. And tools like ReAct pattern, Auto-GPT, and ChatGPT plugins, which facilitate automation through API requests and code execution, as always, amplify the potential for damage.

The challenge with LLMs is their reliance on natural language, complicating the differentiation between legitimate and malicious prompts. We saw at several red-team events how easy LLMs can be manipulated.

The cybersecurity industry is exploring solutions, including input scrubbing and output guardrails, to prevent data exposure or inappropriate content generation. However, these measures are in their infancy and face inherent limitations. But they will be discussed in detail this year at LeFic in France and at RSA in the U.S. so plan to attend.

For law firms, the urgency for effective defense mechanisms against these truly sophisticated attacks is clear, underscoring the need for swift action in response, given their overall cybersecurity defense weakness (among the worst of any industry, a topic we’ll discuss in the coming weeks after we combine all of our Berlin notes).

And yes, we know. It is becoming impossible. In one corner, we have the criminally insane nations state actors, led by their criminally insane totalitarian leaders seeking total global hegemony, whose ends always justify whatever means necessary, cyber attacks a close second to kinetic attacks.

In another corner toil the greedy little hackers, driven by crass but simple monetary motivations, who spray across industry and environment seeking unpatched vulnerabilities to exploit, so they can earn a handful of coin which they can launder through untraceable exchanges that no one will shut down.

And in the middle stands the beleaguered CISO, trying hard to make sense of any of it, while losing enthusiasm for his or her role by the hour. Another day, another breach. Another cyber attack tool nobody thought about.

The future ain’t bright.