SCHOI1337

Designing an AI-Orchestrated Framework for Penetration Testing

October 17, 2025

Introduction

Automation has always been part of offensive security.
What has changed is how much reasoning we can embed into that automation.

This project examines how large language models (LLMs), when paired with controlled execution environments, can coordinate penetration testing workflows safely and reproducibly.

The goal is not to remove human decision-making, but to design a framework where AI systems handle repeatable analysis while human testers focus on higher-order reasoning.

Motivation

Traditional pentesting pipelines rely on human operators to plan, run, and interpret every step of an engagement. While this ensures precision, it also consumes time on tasks that could be standardized: reconnaissance, service fingerprinting, and report summarization.

The central question became:

Can we use an LLM to plan, prioritize, and summarize pentest activities, without compromising safety or control?

The answer required combining two complementary layers:

  1. Cognitive automation — an LLM acting as a planner and analyst.
  2. Execution isolation — a disposable environment performing approved actions under strict policy.

Framework Overview

Planner and Triage Agents (Azure OpenAI)

⬇️

Orchestrator (Validation and Policy Enforcement)

⬇️

Executor (Docker / AWS Kali EC2)

⬇️

Artifacts and Reports (S3 / Confluence)

Each layer has a defined responsibility:

  • Planner Agent : generates structured job definitions (job.json) from reconnaissance data.
  • Orchestrator : validates targets, applies allowlists and approval rules, and dispatches safe jobs.
  • Executor : runs approved scans within a short-lived container or AWS Kali instance.
  • Triage Agent : consolidates raw results, drafts summaries, and recommends next steps.

Every action is logged, traceable, and can be reviewed before escalation.

Implementation Highlights

1. Safe Planning through Azure OpenAI

The planner agent receives contextual reconnaissance data and produces a machine-readable job definition.

A simple example:

{
  "type": "nmap",
  "params": {
    "scan_profile": "safe",
    "args": "-Pn -sT --top-ports 100 -T3"
  },
  "approval_required": false
}

If a deeper scan is required (-sC -sV), the agent automatically marks
"approval_required": true, ensuring that intrusive actions always pass human review.

2. Controlled Execution with AWS Kali

Executors operate inside isolated networks — either as Docker containers or short-lived AWS Kali EC2 instances.

They contain the necessary toolsets (nmap, nuclei, jq, etc) and no persistent credentials.

After execution, artifacts are uploaded to encrypted storage (S3) and the instance is terminated.

This separation ensures that testing logic (AI reasoning) and testing activity (network interaction) never overlap in a way that risks scope violations.

3. Automated Triage and Reporting

Once artifacts are collected, a secondary LLM agent summarizes findings, prioritizes risk, and produces a markdown report suitable for internal publication.

The agent focuses on clarity and traceability, not verdicts — all conclusions are reviewed by a human analyst before distribution.

Governance and Safety

Automation was designed around explicit safety boundaries:

  • Allowlist enforcement – no execution outside preapproved assets.
  • Approval matrix – credentialed or exploit-level scans require authorization.
  • Network isolation – executors have no external internet connectivity.
  • Secrets management – all keys stored in Azure Key Vault or AWS KMS.
  • Auditability – every prompt, response, and command is logged with timestamps.

These controls make the system suitable for internal pentesting team operations under existing corporate compliance frameworks.

Reflection

Building this system clarified an important distinction between automation and autonomy.

True value lies not in removing humans from the process, but in letting machines handle the mechanical layers of reasoning so testers can focus on creative, adversarial thinking.

This work sits at the intersection of AI architecture and offensive security practice.

It suggests a direction where large models become collaborators rather than tools — augmenting the precision, speed, and accountability of future red team operations.

→ GitHub: Agent-First Pentest