PRT (Primary Refresh Token): How it Works and Exploitation Techniques
With the widespread adoption of hybrid environments and the Zero Trust model, Microsoft Entra ID (formerly Azure AD) has established itself as a centr...
What is LLM Penetration Testing?
An LLM penetration test involves analysing the security of an application that incorporates language models (LLMs), autonomous agents or generative AI pipelines, by simulating targeted attacks. Our auditors analyse not only vulnerabilities specific to the models themselves but also application flaws, third-party integrations (APIs, RAG, MCP servers) and the security configuration of the AI environment. The hosting infrastructure is also included within the scope of the analysis, whether it is based on a cloud-based model provider, a self-hosted model or a hybrid architecture.
At the end of the testing phase, our auditors provide a detailed report. This report includes a description of the vulnerabilities identified, exploitation scenarios and recommendations for remediation. A revalidation phase can then be carried out to verify that the fixes have been implemented correctly and thus ensure the security of your AI system.
Our Technical Expertise in LLM Penetration Testing
Our auditors carry out penetration tests on AI systems built using all types of models, frameworks and architectures — proprietary models (OpenAI, Anthropic, Google) or open-source models, RAG applications, autonomous agents, multi-agent systems and business co-pilots. Whether it’s an internal chatbot, a customer assistant, a development co-pilot or an agent orchestration platform, we tailor our approach to your technical stack and your use case.
Tests Performed During an LLM Penetration Test
Analysis of the model’s defence mechanisms
- Resistance to direct and indirect prompt injections
- Bypassing filters using fake markup and mimicked system tags
- Bypassing via the injection of fake dialogue turns (fake user/assistant responses)
- Bypassing safeguards (system prompts, guardrails, content filters)
- Jailbreaking and misuse of the model
- Extraction of system prompts and safeguard instructions
- Extraction of training data via sentence completion and reminder-based prompting
- Model inversion and membership inference tests
Access controls and isolation checks
- Assessment of user role and privilege management
- Detection of information leaks between sessions or between tenants
- Identification of missing or misconfigured access controls within the model
- Privilege escalation tests using tools exposed within the model
RAG integration and data source security
- Analysis of retrieval pipelines (vector database, embeddings, re-ranking)
- Data poisoning and indirect injection tests via indexed documents
- Verification of the validation and sanitisation of returned content
- Exposure of sensitive data in the context passed to the model
AI-specific business logic testing
- Detection of exploitable inconsistencies in agent workflows
- Misuse of the model’s intended use case
- Excessive agency: unauthorised actions triggered via the model
- Circumvention of critical validations through natural language manipulation
- Triggering malicious actions against other users or systems via the model (application-level CSRF, unsolicited API calls, sending unauthorised messages)
API and associated application security
- Audit of APIs exposed by or around the LLM
- Server-side controls (rate limiting, quotas, cost control)
- Conventional injection testing of exposed parameters and functions
- Leaks of sensitive data in model responses
Model output processing security
- Injection of malicious payloads into model responses (stored XSS, un sanitised HTML tags)
- Remote code execution via LLM outputs consumed without validation (shell commands, SQL queries, file paths)
- CSRF tests triggered by model-generated responses or actions
- Audit of front-end post-processing and rendering pipelines
Security Analysis of Agents and the Model Context Protocol (MCP)
Audit of MCP servers and exposed tools
- Inventory of connected MCP servers and tools exposed to agents
- Detection of tool poisoning and malicious tool descriptions
- Analysis of authentication and authorisation flows (OAuth, tokens, scopes)
Tests for tool abuse by agents
- Unauthorised execution or misuse of tools by agents
- Circumvention of permission policies and intent detection
- Unauthorised action chains exploiting multiple tools
Security of agent/tool interactions
- Indirect injection via third-party tool responses
- Exfiltration of sensitive data via call arguments
- Fuzzing of exposed parameters and JSON schemas
Audit of agent-based pipelines
- Excessive agency and privilege escalation via the model
- Tests for adversarial behaviour in multi-stage workflows
- Detection of anomalous behaviour in multi-agent orchestration
Tests Performed on the AI and MLOps Infrastructure
Fine-tuning and MLOps pipelines
- Audit of training and validation datasets
- Data poisoning tests on ingestion pipelines
- Management of secrets and API keys (LLM providers, vector stores)
- Audit of model-related CI/CD workflows
Cloud environments and vector databases
- Audit of cloud configurations (LLM providers, vector stores, orchestrators)
- Analysis of identity and access management for AI services
- Audit of security groups and network rules relating to the model
- Isolation of environments (development, testing, production)
Types of LLM Penetration Tests
An LLM penetration test can be carried out using three approaches, each offering a unique perspective on the security of your AI system.
Black box LLM penetration testing
Our auditors adopt the perspective of an external attacker, with no prior knowledge of the application, model or exposed tools.
Grey box LLM penetration testing
Our auditors have access to partial information: user accounts, functional documentation, a description of the model’s capabilities, and an inventory of MCP tools.
White box LLM penetration testing
Our auditors assess the security of your system with full access: system prompts, source code, model configuration, exposed tools and RAG datasets.
LLM Penetration Testing Methodology
AI Attack Surface Discovery and Mapping
Our auditors map your AI attack surface by identifying the models in use, exposed tools and functions, agents, RAG sources and MCP servers. They also look for leaks of sensitive information (disclosed system prompts, exposed API keys, accessible training data).
Modelling AI-Specific Threats
The analysis draws on industry benchmark frameworks: OWASP LLM Top 10, OWASP Top 10 for Agentic Applications, MITRE ATLAS, NIST AI RMF and PortSwigger Web LLM Attacks. This stage enables us to identify attack scenarios relevant to your use case.
Identification and analysis of vulnerabilities
Our auditors detect and validate vulnerabilities through in-depth manual testing, supplemented by specialised tools. Each vulnerability is analysed according to its severity and its actual impact on your application and business.
Exploitation and impact assessment
The identified vulnerabilities are exploited in a controlled manner to measure their actual impact — data exfiltration, model sabotage, tool abuse, uncontrolled costs — and to identify any potential attack chains involving multiple components.
Penetration test report, debriefing and follow-up audit
A detailed report sets out the vulnerabilities identified, their severity level, proof-of-concept exploits and prioritised remediation recommendations. A debriefing with your AI and development teams allows us to explain the findings and support the remediation process, followed by a follow-up audit to validate the effectiveness of the measures implemented.
CUSTOMER TESTIMONIALS
About vaadata
LLM Penetration Testing with Vaadata, a Trusted Offensive Security Partner
Carrying out an LLM penetration test with Vaadata means choosing a company certified to the industry’s most stringent standards.
Our company is PASSI, CREST, ISO 27001 and ISO 27701 certified, which attests to the quality of our services and our compliance with international standards for cybersecurity and personal data protection. These certifications bolster our clients’ confidence by guaranteeing a high standard of methodological and organisational rigour.
Our LLM penetration tests are carried out by certified auditors who are proficient in both traditional application attack techniques and the specific characteristics of language models, agents and the Model Context Protocol. This dual expertise (AppSec and offensive AI) enables them to accurately identify, exploit and document the vulnerabilities specific to each AI environment.
Resources
Our Latest Resources
EDR (Endpoint Detection and Response): How It Works and Detection Mechanisms
Traditional antivirus solutions are now facing increasingly effective evasion techniques. From payload obfuscation and shellcode encryption to the use...
Cyber Threat Intelligence: Objectives and Methodology
Cyber Threat Intelligence now plays a central role in cybersecurity strategies. Faced with increasingly targeted threats, organisations must move beyo...
