Pentesting Large Language Model Apps using the OWASP Top 10 for LLM Apps

Embarrassing news accounts show the risks of prematurely giving an LLM-powered app its freedom and exposing it to the Internet. On the other hand, effective pentesting of your app can save you and your customers from some painful results.

But how do you pentest an AI app? Thankfully a lot of smart people have been working on this and producing tools like the OWASP Top 10 for LLM Apps. Pentesting generative AI begins with prompt engineering.

Prompts are to an LLM as shell commands are to an operating system. Tons of work has been done to harden shells against command injection, enforcement of least privilege, etc, to prevent malicious users from escalating their privileges or otherwise tricking the system into doing things that particularly aren’t allowed. The same is going on today with LLMs.

In this webinar, we’ll discuss the difference between prompt injection and jailbreaking LLMs. We’ll also explore what system prompts are and how they are crucial to implementing guardrails and limits over user prompts.

But an interesting thing about prompts is that you probably need to give as much attention to unintended prompt injections as to direct malicious ones. While you can typically make some assumptions about the competency of a sysadmin the same cannot be said about users of an LLM. In a recent conversation with Luke Doherty and Gisela Hinojosa’s AI pentesting team at Cobalt, they shared a perfect example of what I mean. An education software company bringing an AI-powered app to market for elementary school students has a responsibility to protect its young users and limit content returned by innocent queries about human reproduction to age-appropriate results – not the unfiltered mashup of content that could potentially be referenced by an AI searching the web.

 Another interesting aspect of prompt engineering is indirect prompt engineering. OWASP says “Indirect prompt injections occur when an LLM accepts input from external sources, such as websites or files. The content may have in the external content data that when interpreted by the model, alters the behavior of the model in unintended or unexpected ways.”

For instance, imagine one of your employees asking an LLM to summarize a webpage about a new product released by one of your competitors. Nothing’s wrong with that. But what if the competitor’s website contains hidden instructions to the LLM leading to exfiltration of the entire conversation up to that point. And what if earlier in that conversation your employee had asked the LLM to “find recent product announcements by other companies that could have a competitive advantage against our product X”?

In my next real training for free session, I will be introducing the concept of AI pentesting with an overview of the OWASP Top 10 for LLM Apps and I’m excited to have a panel of subject matter experts from Cobalt to provide tales from the trenches in this fast-moving area of cyber security. Cobalt conducted over 4,000 pentests last year alone and they have developed a practice for pentesting LLM Applications based on their work with the OWASP working group on this topic.

We will focus on real-world AI pentesting stories, sharing detailed examples of security failures the team has encountered in AI implementations and how they identified them through testing. We’ll walk through specific cases, demonstrating how they develop test cases - such as by manipulating language inputs (both human and machine) - and iterating through different attack scenarios to uncover vulnerabilities. The session will also explore the distinction between AI safety and AI security, how Cobalt scope AI pentests based on customer needs, and how their approach has evolved from early development issues like prompt injection to pre-release assessments aligned with the OWASP Top 10.

Please join us for this real training for free session.