New Autonomous re-testing now validates fixes in under an hour. See how

AI Agent Security Risks: What Could Go Wrong When Agents Test Your Systems

AI Agent Security Risks: What Could Go Wrong When Agents Test Your Systems

When a security team evaluates agentic penetration testing, one concern surfaces consistently: the testing agent has credentials, tool access, and exploit capabilities inside your environment. What happens if the agent behaves unexpectedly? What data does it access? Who audits what it does? What is the blast radius if something goes wrong?

These are legitimate security questions. An agentic testing system that operates without satisfactory answers to them should not be trusted with access to production systems. Any vendor who dismisses these concerns rather than addressing them is giving you a reason not to buy.

This post addresses each concern directly: what the risk is, where it comes from, what design controls are in place to bound it, and what you should verify before deploying any agentic testing platform.

Why agentic testing systems have a different risk profile than traditional tools

A vulnerability scanner has a limited risk profile: it sends HTTP requests, reads responses, and reports. Even if the scanner misbehaves, the damage it can cause through scanning is constrained.

An agentic penetration testing system is different by design. It conducts active exploitation to confirm vulnerabilities. It operates as multiple authenticated users. It executes payloads, extracts data to demonstrate exfiltration is possible, and navigates application flows in ways that mirror real attacker behaviour. This is what makes it more effective than scanners, and it is precisely what raises the security questions worth asking.

The goal of this guide is not to reassure you that nothing can go wrong. It is to map the specific risk categories, explain what controls address each, and give you the questions to ask before deployment.

Risk category 1: Credential handling and exposure

The risk: An agentic testing system operating as authenticated users requires test credentials or service accounts. If those credentials are mishandled by the testing platform: stored insecurely, transmitted without encryption, logged in plaintext, or retained after the engagement closes: they represent a credential exposure risk.

Where the risk concentrates: Credentials provided for testing (dedicated test accounts, service account tokens, API keys with test-scope permissions) are sensitive because they provide access to the target system. If the testing platform's own infrastructure is breached, those credentials could be used to access the systems they were created to test.

Design controls to verify: Credentials should be stored encrypted at rest within the testing platform, using key management that isolates credential storage from the testing execution environment. Credentials should be transmitted only over encrypted channels. The platform should log credential use (which credential, which test, what action) without logging credential values in plaintext. Credentials should be automatically invalidated at engagement close, or the platform should provide explicit notification that credentials should be rotated after each engagement.

Questions to ask: How does the platform store test credentials? Are credentials encrypted at rest, and what key management system protects them? Are credential values logged anywhere in the audit trail? What is the credential lifecycle: when are credentials purged after an engagement closes?

What you control: Use dedicated test accounts with the minimum permissions required for the defined test scope. Do not provide production service account credentials for testing. Create test-specific accounts that can be invalidated immediately if any concern arises. Rotate credentials after every engagement as a standard practice. Penetration testing rules of engagement covers credential handling as a standard pre-engagement requirement.

Risk category 2: Data access during testing

The risk: A penetration testing agent that demonstrates data exfiltration must access data to prove the vulnerability is real. What data does the agent access? Is it accessing production customer data? Does it retain what it reads?

Where the risk concentrates: Application penetration testing frequently requires accessing application data to demonstrate that vulnerabilities are exploitable. A demonstration that an IDOR allows accessing another user's profile requires actually reading that profile. The question is whether the testing system handles this access in a way that respects data confidentiality.

Design controls to verify: A well-designed agentic testing system should not retain the content of data accessed during testing. The finding documentation should record that data was accessible and the category of data (personal records, financial data, credentials) without retaining the actual content. Testing should use synthetic test data where possible, and where real application data is accessed to prove exploitability, that access should be logged and the data not retained.

The engagement scope should define data handling explicitly: what data categories the testing system may access, whether real production data may be accessed or only synthetic test data, and what happens to any data accessed during the engagement. Penetration testing scope: how to define it before you start covers data scope as a pre-engagement requirement.

Questions to ask: Does the platform retain content of data accessed during testing, or only metadata (that data was accessible, the category, the access path)? How is access to real production data handled versus synthetic test data? What are the data retention policies for engagement records?

What you control: Specify in the scope document that testing should use synthetic test data where possible and that any access to real production data should be documented but not retained by the testing platform. Define acceptable data access categories explicitly in the rules of engagement.

Risk category 3: Audit trail and observability

The risk: If an agentic system takes actions in your environment and something goes wrong (an unexpected system interaction, a test that causes unintended side effects), you need to know exactly what happened, when, and why. An audit trail that is incomplete, difficult to query, or absent creates a forensic blind spot in your own environment.

Where the risk concentrates: Unlike a human penetration tester whose actions can be reconstructed through network logs and application logs, an agentic system's internal decision-making (why it chose to execute a specific test, what it observed that led to that decision) may not appear in your own logs at all. Your SIEM sees the network traffic; it does not see the agent's reasoning chain.

Design controls to verify: The testing platform should produce a complete, chronological log of all actions taken during a test cycle: what inputs were sent to the application, what responses were received, what the agent decided to test next and why, what payloads were used, and what findings were produced. This log should be exportable in a format that can be reviewed independently.

The platform should also support real-time visibility: a live activity feed that shows what the agent is currently testing, so that the security team can monitor agent behaviour during test execution rather than only reviewing results afterward.

Questions to ask: Can you export a complete action log from the testing platform? Does the log include the agent's reasoning at each step, or only the network-level actions? Is there a real-time activity view during test execution? How long are test logs retained, and in what format?

What you control: Review the action log for each test cycle, not only the findings report. Monitor the first several test cycles actively to understand what the agent does, in what sequence, and how it behaves in your specific environment before treating it as a background process.

Risk category 4: Blast radius: what happens if the agent behaves unexpectedly

The risk: An agentic testing system that misbehaves (due to a bug, an unexpected application state it encounters, or a prompt injection attack against the agent itself (a risk covered in LLM penetration testing: testing the models behind your AI features) could take actions beyond its intended scope.

Where the risk concentrates: The blast radius of an agentic testing system is bounded by its tool access and by the scope boundaries the platform enforces. An agent with read-only test account access and no write capabilities has a much smaller blast radius than one with broad application permissions. An agent that halts and notifies when it encounters an unexpected application state has a smaller blast radius than one that continues autonomously.

Design controls to verify:

Scope enforcement at the platform level, not just the instruction level. The agent should be architecturally constrained to the defined scope: unable to make requests to systems outside the defined scope regardless of what its reasoning produces. This is different from instructing the agent to stay within scope; it requires the platform to enforce scope boundaries as technical constraints.

Destructive action prohibition by design. The agent should not be able to delete, modify, or corrupt data: only read and test. Exploits that demonstrate write-access vulnerability should do so through controlled writes (a clearly labelled test record with a defined prefix) rather than modification of real data.

Automatic halt conditions. Define the conditions under which the agent pauses and notifies rather than continuing: encountering a system not in the defined scope, a response suggesting the application is experiencing genuine distress, or an action sequence that does not match expected testing patterns.

Questions to ask: Are scope boundaries enforced technically at the platform level, or only through agent instructions? What is the platform's behaviour when the agent encounters an unexpected application state? Are destructive operations architecturally prevented, or only instructed against? What is the halt and notification procedure for unexpected behaviour?

What you control: Define a narrow initial scope for the first engagement cycle and expand it as you observe agent behaviour and build confidence. Start with a staging environment before testing production. Ensure you have a clearly defined halt procedure with the testing vendor. Penetration testing rules of engagement covers halt procedures as a required pre-engagement element.

Risk category 5: Supply chain risk of the testing platform itself

The risk: The testing platform is software that runs in your environment (or accesses it remotely). Like any third-party software, it introduces supply chain risk: vulnerabilities in the platform itself, in its dependencies, or in the model it uses could create an attack surface within your security infrastructure.

Where the risk concentrates: A security testing platform by definition has privileged access to the environment it tests. If the platform itself is compromised (through a vulnerability in its code, a compromised dependency, or an attack against the platform vendor's infrastructure), an attacker gains a position with elevated access to the systems being tested.

This is the same risk that exists for any security tool with privileged access: SIEM platforms, endpoint agents, vulnerability scanners. The question is not whether the risk exists (it does for all privileged tools) but whether the vendor has appropriate controls to reduce it.

Design controls to verify: The vendor should be able to articulate their own security programme: SOC 2 certification, penetration testing of the platform itself by independent third parties, vulnerability disclosure programme, and dependency management practices. Ask for their most recent penetration test report executive summary.

The platform's access model should follow least-privilege. The test credentials provided to the platform should have the minimum permissions required for the defined testing scope, so that even if the platform were compromised, the attacker's access is bounded by the test account permissions.

Questions to ask: What security certifications does the platform vendor hold? When was the platform last independently penetration tested, and by whom? What is the vendor's vulnerability disclosure programme? How does the platform authenticate to the target environment, and what happens to those credentials after the engagement?

Agent orchestration: how autonomous systems coordinate covers the supply chain risk of AI system components. MCP security: what model context protocol means for AI agent safety covers how tool connectivity protocols introduce their own supply chain considerations.

The risk asymmetry argument

Having mapped five real risk categories with genuine controls, the remaining question is how these bounded risks compare to the alternative.

The alternative to agentic continuous penetration testing is periodic manual testing, typically once or twice annually. The risks of that model are:

Vulnerabilities introduced in the deployments between annual engagements remain undetected for months. The 2024 breach disclosure record shows the median time from vulnerability introduction to exploitation is measured in weeks, not months. Annual testing leaves a window measured in months for vulnerabilities that take weeks to be found and exploited.

A fixed-window engagement produces a point-in-time report. The code deployed the day after the engagement closes is not in the report. In continuous delivery environments, this means a significant fraction of the production codebase is never tested.

The risk of agentic testing is: a well-designed system with appropriate controls (scoped test accounts, read-only where possible, complete audit trail, technical scope enforcement) may have residual risk in the five categories above, all of which are bounded and manageable.

The risk of not testing continuously: unknown vulnerabilities accumulate in production, discovered eventually either by your testing programme or by an attacker. Only one of those outcomes is bounded.

Continuous penetration testing and how it differs from annual pentests covers the cadence comparison. Agentic pentesting and continuous security validation covers the testing architecture. Generative AI security risks: what every CISO should understand covers the broader AI risk context within which agentic testing sits.

For penetration testing services in the US, agentic penetration testing with transparent audit trails and scope-enforced access, and PTaaS for the ongoing model, the 10x Pentest platform covers security programme requirements including the credential handling, data access, and audit trail questions raised above. See pricing or get in touch to discuss how these controls are implemented in a specific deployment and what verification you should require before onboarding any agentic testing platform.

Frequently asked questions

Q1. What are the security risks of using AI agents for penetration testing?

The primary security risk categories when deploying agentic penetration testing are: credential handling (test credentials provided to the platform are sensitive and must be stored encrypted, transmitted securely, and purged at engagement close); data access during testing (agents must access application data to prove vulnerabilities, raising questions about data retention and confidentiality handling); audit trail completeness (the agent's actions must be fully logged so that any unexpected behaviour can be reconstructed); blast radius of unexpected agent behaviour (scope boundaries must be enforced technically, not just by instruction, and destructive actions must be architecturally prevented); and supply chain risk of the testing platform (the platform itself is privileged software that introduces vendor security risk). Each of these risks is bounded by specific design controls and should be verified before deploying any agentic testing platform.

Q2. What data does an AI penetration testing agent access?

An agentic penetration testing agent accesses application data proportional to what is required to confirm the vulnerabilities it tests. Testing for IDOR vulnerabilities requires accessing data through the vulnerable endpoint to confirm it is real. Testing for SQL injection may require reading database records to confirm extraction is possible. A well-designed testing system documents what data was accessible and the category of that data without retaining the actual data content. The engagement scope should specify whether testing uses synthetic test data or real production data, and what happens to any data accessed during the engagement. Organisations that require strict data residency or prohibition on accessing real production data should configure testing accordingly and verify the platform's data handling practices before deployment.

Q3. How do you audit what an AI agent does during a penetration test?

A well-designed agentic testing platform produces a complete, chronological action log covering every input sent to the application, every response received, the agent's reasoning at each decision point, every payload used, and every finding produced. This log should be exportable and independently reviewable. The platform should also provide real-time activity visibility during test execution so the security team can monitor agent behaviour rather than only reviewing results. When evaluating agentic testing platforms, request a sample action log from a completed engagement to verify the completeness and readability of the audit trail before deployment.

Q4. What happens if an AI agent misbehaves during a penetration test?

Blast radius from unexpected agent behaviour is bounded by the platform's scope enforcement and permission model. A well-designed platform enforces scope boundaries as technical constraints rather than agent instructions: the agent is architecturally prevented from accessing systems outside the defined scope regardless of its reasoning. Destructive operations should be architecturally prohibited: the agent should not be able to delete or modify real data, only read and test through controlled means. Define explicit halt conditions before testing begins: circumstances under which the agent pauses and notifies rather than continuing autonomously. The rules of engagement should include a 24/7 emergency halt contact and a defined halt signal mechanism that the testing vendor acknowledges within a specified time window.

Q5. Is the agentic testing platform itself a security risk?

Yes, like any privileged third-party software: a SIEM, an endpoint agent, a vulnerability scanner. The platform has access to the environment it tests, making it a potential attack surface. Controls that reduce this risk include: vendor SOC 2 certification and independent penetration testing of the platform itself; least-privilege test accounts that bound what an attacker could access if the platform were compromised; transparent supply chain practices including dependency management and vulnerability disclosure programme; and scoped, short-lived credentials rather than persistent access. Ask the vendor for their most recent independent penetration test executive summary and verify their approach to platform security before deployment.

Stop playing defense.
Automate your offense.

Schedule a free consultation and see how teams like yours are strengthening their security posture — continuously.