AI is rife in the current digital landscape; both business and pleasure have been infiltrated by AI agents, LLMs, and chatbots. Absolutely no one can escape it. The rate of adoption of AI as a new digital tool is the fastest ever recorded in technological history. This makes it the most rapidly adopted innovation worldwide. ChatGPT alone gained over 100 million users in the first 2 months of its launch and is currently reaching over 700 million weekly users in 2026. To put that into perspective, Facebook took roughly 7 years to reach 700m users. The sheer speed of its global takeover is enough to give anyone the shivers.
Cyber security professionals around the globe are now dealing with a double-edged sword. Extremely insecure AI tools, shadow AI and, of course, a rapid increase in AI-powered cyber attacks. Attackers can launch novel, sophisticated attacks at scale in just a couple of minutes, something that would have usually taken days to pull off.
Similarly, reliance on AI agents specifically has become a huge problem in the world of cyber security. Around 92% of industry leaders are concerned about the use of AI agents across the workforce and their impact on security. This blog pulls together the challenges, techniques and resources a cyber security expert will find invaluable for testing and securing AI agents in their company.
What is Penetration Testing for AI?
Pentesting AI is, in short, an attack on an AI tool to uncover vulnerabilities before a real attacker can exploit them. It follows the same principles as any other penetration test, except the target is different. A tester would typically probe the agent for ways it can be tricked or manipulated beyond its intended use. This could include, but is not limited to, producing content it should refuse, leaking prompts and internal instructions and executing unintended actions.
AI penetration testing requires slightly different techniques than traditional testing; this matters because AI doesn’t fail the way traditional software does. An agent can be working as intended yet still be manipulated into leaking data or performing unintended actions with persistent attacker effort.
Challenges when Pentesting AI Agents
As with any new technology, there come challenges. Here are a few we, as a pentesting company, see often with AI agents. Challenges that traditional penetration testing doesn’t really deal with:
- Non-deterministic Behaviour: The same input produces different outputs each time. A prompt that fails to break into the model once might succeed on the 12th attempt. This makes both testing and reporting harder than a traditional test where a vulnerability either exists, or it doesn’t.
- No Fixed Attack Surface: Unlike a web app or API, which have defined endpoints. An agent’s “surface” is effectively unlimited because any input phrasing can be an attack vector.
- Fast-moving Unstandardised Risk Landscape – Guidance for testing on AI specifically is sparse. Pentesters are working with less-established methodologies, as much of it is still under development. OWASP recently updated its GenAI LLM Top 10 in August 2026. It shows significant movement in the rankings since 2025.
Methodology for Penetration Testing AI Systems
The foundation for most AI testing is prompt injection. Essentially, tricking the agent through natural language and syntax. Because of the non-deterministic nature of LLMs, it’s not uncommon for testers to run 8-12 different prompt injection attempts before drawing conclusions.
One of the pioneers in research on pentesting and redteaming AI agents is Jason Haddix, founder of Arcanum Information Security. His methodology for attacking AI breaks the process into the following seven stages:
- Identify system inputs by mapping every way that data or instructions can reach the model.
- Attack the ecosystem by testing the surrounding infrastructure the AI operates within.
- Attack the model itself by probing it directly for manipulation.
- Attack the prompt engineering by targeting how the system prompt and its instructions are built. Each one of these systems runs on an underlying system prompt that can potentially be overridden.
- Attack the data by testing which data the model can access or is trained on.
- Attack the application by testing the software layer wrapped around the model.
- Pivot by using a successful compromise in one part of the system to move further into others.
In practice, this might look like uploading a document containing malicious elements that use cryptographic techniques to be processed as ordinary content. This approach could essentially leak the underlying prompt engineering and cause malicious code to execute.
Resources for Attacking AI
Arcanum Information Security have a tonne of resources for red teaming and penetration testing AI. These actively maintained resources are a great starting point and definitely worth bookmarking:
- The Arcanum Prompt Injection Taxonomy – A classification of prompt injection attacks organised into four pillars: Intents, Techniques, Evasions and Inputs. Each entry cross-references how other frameworks such as OWASP, MITRE ATLAS, and NIST name the same attack.
- P4RS3LT0NGV3 – An open-source payload crafter tool that encodes text into a multitude of formats and scripts (Base64, leetspeak, ROT13, Unicode styles, invisible text, and more). Very useful for quickly generating variants of a test payload.
- The AI Security Resource Hub – A well-curated directory of over 100 free labs, CTFs, competitions, bug bounty programs, and tools for AI pentesting and red teaming.
Most importantly, the OWASP Top 10 for Gen AI LLMs is the gold-standard guidance that any security professional should align their testing with. It lists the ten most common and impactful risk categories, including prompt injection, insecure output handling, training data poisoning, excessive agency, and sensitive information disclosure.
Do I need my AI system tested?
Arguably the most important question. If your organisation has deployed or is about to deploy an AI system that has any of the following characteristics, the answer is probably yes:
- It can read content you don’t actively control (documents uploaded by clients, scraping web pages, support tickets, etc.)
- It has write access to databases, tools, or systems.
- It can execute code or trigger other automated workflows.
- It relies on the model itself to enforce data-handling or safety rules.
- It’s built from multiple agents, models, or vendors stitched together (especially if any component is open-source).
- It has access to sensitive data.
Attackers have used these techniques to manipulate AI in many documented, disclosed incidents. The pattern is that teams secure the chat interface, while the weak point lies somewhere in the ecosystem around the model. Meaning the data it ingests, the tools it can interact with or simply the trust placed in its own filtering.
Pen testers’ closing thoughts
From our own experience at Sencode, the security of AI-enabled applications is still developing rapidly. We have now tested dozens of chatbots, AI-augmented applications and agentic systems, and across those assessments we have identified vulnerabilities spanning a wide range of security classifications — including several critical-risk issues.
One of the fundamental challenges is that LLMs are inherently probabilistic. An attack may fail repeatedly and then succeed on a later attempt with only minor changes to the input. A traditional application might be expected to produce the same result given the same conditions; an LLM-driven system cannot always be treated in the same way. Security testing therefore has to account for repeated attempts, variations in phrasing, changes in context and interactions across multiple components of the system.
This is particularly important when AI systems are given tools, credentials or access to backend resources.
The principle of least privilege must be treated as a fundamental security requirement for AI systems.
An agent designed to answer a user’s question may, behind the scenes, have access to databases, internal APIs, file stores, administrative functionality or third-party services. If the underlying agent or orchestration layer has significantly more access than is required to perform its intended task, successful manipulation of the model can quickly become much more than a prompt injection issue.
During testing, we have identified vulnerabilities in which the AI layer provides a route into the wider application ecosystem. This can involve manipulating prompts, influencing tool calls, attacking connected systems, accessing data outside the intended user context and using the agent itself as a mechanism for further attacks.
For that reason, testing the model in isolation is not enough.
It is also important to remember that behind every polished conversational interface there is still a conventional application architecture. There are system prompts, APIs, WebSockets, streaming responses, authentication tokens, databases and backend services. In many cases, developers place too much trust in the LLM to decide what information should be returned, which instructions should be followed, or which actions a user should be authorised to perform.
An LLM is not an access-control mechanism.
Developers should assume that system prompts will eventually be influenced, model safeguards will sometimes fail, and unexpected inputs will reach parts of the system they were never intended to reach. Security controls therefore need to exist outside the model: enforce permissions at the API and data layer, strictly constrain available tools, validate every action server-side and minimise the resources available to the agent.
AI will undoubtedly continue to make penetration testing faster and more capable. But it is also creating an entirely new attack surface.
For penetration testers, the challenge is no longer simply “can I break the model?” It is: “If I can influence the model, what can the model reach — and how far can I go from there?”
That is where some of the most serious AI security vulnerabilities are currently being found.
Treat AI agents the same way you would treat any other system with access to sensitive data or internal infrastructure. Tested by a professional. Sencode offers free cyber security advice and has tested a multitude of LLM-based agents and vibe-coded applications. Book a call with a consultant today to discuss your requirements for a penetration test and explore our range of services.