PentAGI can run on any OpenAI-compatible API through its custom LLM provider: set LLM_SERVER_URL, LLM_SERVER_KEY and LLM_SERVER_MODEL in the .env file, and every agent in a flow talks to that endpoint instead of OpenAI. That one PentAGI custom endpoint setting is how teams put vLLM, a LiteLLM proxy or a hosted provider behind the framework. This guide maps those settings to RedRouter's OpenAI-compatible API, shows how to test the connection before a flow starts, and covers the gaps you are most likely to hit.
PentAGI is VXControl's self-hosted, multi-agent penetration testing system. You describe a target and a goal, and a team of agents (primary agent, pentester, coder, searcher and others) plans the work and runs tools such as nmap, Metasploit and sqlmap in an isolated sandbox container built from an image such as Kali Linux. It's MIT-licensed, and its README says it's for authorized testing only.
Every PentAGI setting below comes from the current LLM providers docs (last updated October 9, 2026), the project README and the custom provider's source, checked against PentAGI v2.2.0, the latest release (October 5, 2026). RedRouter facts come from docs.redrouter.org and the live model catalog. All were read on November 11, 2026. We haven't run a PentAGI flow against RedRouter for this post, so treat the configuration as a starting point to test, not a verified result.
Authorized work only. Point PentAGI at systems you own, isolated labs, or bug bounty and vulnerability disclosure targets inside the program's published scope. PentAGI's own EULA and RedRouter's terms both prohibit unauthorized testing. An autonomous agent does not shrink your scope; it makes staying inside it your job.
Why PentAGI is in the news
On October 30, 2026, Google Threat Intelligence Group (GTIG) reported that since late August it had seen threat actors operating open-source multi-agent pentesting platforms, naming ARTEX, PentAGI and CAIRN, in Mandiant incident response engagements. GTIG wrote that most observed PentAGI activity stayed at WHOIS asset mapping or vulnerability scanning, though some operations progressed to intrusions and persistence. On November 9, 2026, Help Net Security summarized the same research, listing PentAGI alongside Hexstrike and CAIRN.
That attention cuts both ways. Defenders want to understand how these frameworks work, and authorized red teams want to run them on their own infrastructure with a model provider they choose. GTIG notes PentAGI "supports over a dozen cloud and local LLM backends." The custom provider is the one that takes any OpenAI-compatible base URL. If you are working with ARTEX instead, see ARTEX with RedRouter.
What the PentAGI LLM provider needs from an API
PentAGI drives its agents with tool calls. The README says any custom OpenAI-compatible backend "must return valid tool-call JSON in the format the OpenAI Chat Completions API defines," and that a backend that can't do this fails right at the start of a flow, because the first action is a tool call that picks the sandbox image. So the model you choose needs reliable tool calling on Chat Completions.
RedRouter's compatibility guide covers what that endpoint offers:
- Base URL:
https://redrouter.org/v1. The client appends/chat/completions. - Client-executed function tools: yes, on models whose catalog entry lists tool calling. PentAGI runs the tools in its sandbox; RedRouter never touches your target.
- Stateless requests: every call carries the full history, which matches how PentAGI already works.
- No embeddings API. PentAGI's memory and knowledge search need embeddings, so those go to a separate provider (see below).
- Unknown request parameters return 400. This matters for one PentAGI default, covered in the gaps section.
PentAGI custom endpoint settings
These are the documented variables for the custom provider, from the README's configuration table and the .env.example file:
| Variable | What it does | Value for RedRouter |
|---|---|---|
LLM_SERVER_URL | Base URL of the OpenAI-compatible API | https://redrouter.org/v1 |
LLM_SERVER_KEY | API key. Required: an empty key disables the custom provider | Your RedRouter API key |
LLM_SERVER_MODEL | Default model for all agents; a provider config file can override it | A catalog ID, such as glm-5.3-flash-red |
LLM_SERVER_CONFIG_PATH | Optional YAML file that sets a model and parameters per agent type | /opt/pentagi/conf/custom.provider.yml if you use one |
LLM_SERVER_PROVIDER | Adds a prefix to every model name, meant for LiteLLM proxies | Leave empty |
LLM_SERVER_PRESERVE_REASONING | Sends earlier reasoning back in multi-turn history (default false) | Leave false |
LLM_SERVER_API_TYPE | azure or azure_ad for Azure OpenAI; empty for a plain endpoint | Leave empty |
Two of those defaults are worth a sentence each. LLM_SERVER_PROVIDER would turn glm-5.3-flash-red into something like prefix/glm-5.3-flash-red, which isn't a RedRouter model ID and returns a 404. LLM_SERVER_PRESERVE_REASONING exists for providers that reject tool-call history without reasoning attached. RedRouter's reasoning docs accept replayed reasoning on Chat Completions but don't require it, and replayed reasoning counts as input tokens, so false is the cheaper choice.
Point PentAGI at an OpenAI-compatible API
If you use PentAGI's interactive installer, pick the custom endpoint in its LLM providers step and enter the same three values. For a manual install, add this to .env:
# PentAGI custom LLM provider -> RedRouter (OpenAI-compatible API)
LLM_SERVER_URL=https://redrouter.org/v1
LLM_SERVER_KEY=your-redrouter-api-key
LLM_SERVER_MODEL=glm-5.3-flash-red
LLM_SERVER_PROVIDER=
LLM_SERVER_PRESERVE_REASONING=false
Then restart the stack with docker compose up -d. The base URL must end in /v1 exactly once. If you paste https://redrouter.org/v1/chat/completions, the request path doubles and you get a 404; the same happens with a misspelled model ID. Copy IDs from GET https://redrouter.org/v1/models, which is public and needs no key. PentAGI's custom provider also tries to read that model list at startup and logs a warning if it can't.
Keep the key out of any repo. If several people share the PentAGI host, give PentAGI its own RedRouter key so you can revoke or limit it without touching anything else. For a first request outside PentAGI, the API quickstart has a curl example, and the OpenAI-compatible API guide covers the SDK and proxy clients.
Optional: a model per agent type
LLM_SERVER_MODEL alone puts every agent on one model. PentAGI's docs suggest the strongest model for primary_agent, pentester and adviser, and lighter models for searcher, enricher and summarization. To split them, copy the repository's examples/configs/custom-openai.provider.yml, mount it, and point LLM_SERVER_CONFIG_PATH at it. The example file names OpenAI models, including o3-mini, which PentAGI's docs say OpenAI was retiring on October 23, 2026. Replace every model: value with a RedRouter catalog ID. A fragment:
primary_agent:
model: "glm-5.3-flash-red"
max_tokens: 8192
price:
input: 0.2
output: 0.8
pentester:
model: "glm-5.3-flash-red"
max_tokens: 8192
price:
input: 0.2
output: 0.8
The file has an entry for each agent type: simple, simple_json, primary_agent, assistant, generator, refiner, adviser, reflector, searcher, enricher, coder, installer and pentester. The price block only feeds PentAGI's own cost estimates, in USD per million tokens. RedRouter bills from its own usage records, so check your Usage page for real spend. Set max_tokens to 8,192 or less: RedRouter caps output at 8,192 tokens per request on most models and caps larger requests to that.
Test the PentAGI custom endpoint with ctester
Don't find out in the middle of a flow that the provider doesn't work. PentAGI ships ctester, which runs basic completion, JSON, function-calling and pentest-knowledge checks per agent type. After docker compose up -d, run it inside the container:
docker exec -it pentagi /opt/pentagi/bin/ctester -verbose
Without -config, it tests the file LLM_SERVER_CONFIG_PATH names. Check both sides if anything fails: docker compose logs -f pentagi on the PentAGI host, and the Usage page in RedRouter for the matching requests and error codes.
Known gaps between PentAGI and RedRouter
These come from reading both projects' docs and PentAGI's source. Check them in your ctester run.
The n parameter. PentAGI's custom provider sets n=1 as a default call option (custom provider source), and the OpenAI client library it uses includes any non-zero n in the request body. RedRouter's Chat Completions docs list "multiple completions via n" as outside the endpoint's contract and say unknown top-level fields return 400. If ctester fails with a 400 that names n, put a small LiteLLM proxy between PentAGI and RedRouter and drop the field with LiteLLM's documented additional_drop_params setting:
model_list:
- model_name: glm-5.3-flash-red
litellm_params:
model: openai/glm-5.3-flash-red
api_base: https://redrouter.org/v1
api_key: os.environ/REDROUTER_API_KEY
additional_drop_params: ["n"]
Then set LLM_SERVER_URL to the proxy's address and LLM_SERVER_KEY to the proxy's key. Keep LLM_SERVER_PROVIDER empty, because the alias above already matches the model ID. If a 400 names a different field, drop that one the same way. Never expose the proxy to the internet without authentication.
Embeddings. PentAGI's embeddings docs default to OpenAI's embedding API, set through EMBEDDING_PROVIDER, EMBEDDING_URL, EMBEDDING_KEY and EMBEDDING_MODEL. RedRouter has no embeddings endpoint, so don't point EMBEDDING_URL at it. Use a provider PentAGI supports for embeddings, such as a local Ollama model. If you change the embedding model later, PentAGI's docs say to reindex with etester reindex, not etester flush, which deletes the knowledge base.
Output limits. PentAGI's custom provider defaults to max_tokens 16,384. RedRouter caps that at the model's maximum (8,192 tokens on most models), so long agent outputs can stop at finish_reason: length. Keep per-agent limits at 8,192 or below and keep tool output short where you can.
Rate limits. A multi-agent flow sends many requests in parallel. RedRouter's rate limits page lists a starting tier of 60 requests per minute and 4 concurrent requests, with a daily spend ceiling. Check Settings for your account's actual limits. When you hit one, you get a 429 with Retry-After. Run one flow at a time while you size things up.
Failover goes to a second provider. LLM_FALLBACK_PROVIDER repeats a failed call once on another configured provider. That's useful for reliability, but the same prompts and tool output then go to that provider too, so only configure one you're cleared to send engagement data to.
Choosing a model for PentAGI
PentAGI's flows are long, tool-heavy and resend context on every step, so input price and context length matter more than in a chatbot. GLM 5.3 Flash Red (glm-5.3-flash-red) costs $0.20 per 1M input tokens, $0.08 per 1M cached input tokens and $0.80 per 1M output tokens, with a 1,048,576-token context window, an 8,192-token output cap, tool calling and JSON output, per the model page on November 11, 2026. It's text-only. The catalog describes it as modified to reduce refusals, which changes how often it declines, not how often it's right.
The catalog also lists red-cyber-1, a featured beta model RedRouter describes as built for authorized penetration testing and red-team work. See pricing for the full list and the inference API page for what the API is for. PentAGI's docs add that open models under 32B parameters work better with execution monitoring and task planning turned on (EXECUTION_MONITOR_ENABLED, AGENT_PLANNING_STEP_ENABLED).
Keep PentAGI inside your scope
The model is the easy part. Most of the risk is in what the PentAGI host can reach:
- Callback ports. PentAGI's docs warn that each flow's sandbox publishes two callback ports from the 28000–29999 range on every interface of the Docker host, and
ufwdoesn't filter them. Restrict that range with a firewall in front of the host before you run anything on a reachable machine. - Docker group access. The installer needs root or the
dockergroup, and PentAGI's docs point out thatdockergroup membership is root-equivalent. - Default login. Change the default
admin@pentagi.compassword on first sign-in. - Sandbox images. By default the model picks the sandbox image.
DOCKER_ALLOWED_IMAGESrestricts that choice to an allow-list. - Egress. Limit where the sandbox containers can connect to: your target range and nothing else. Help Net Security's report includes a case where a researcher's ARTEX agent left its test environment and reached a third party's systems, which is the failure mode to plan for with any autonomous framework.
Put the engagement scope in the flow's first message, as PentAGI's README recommends, and review findings with your own evidence before they go in a report. For the reporting side of bounty work, see the AI bug bounty workflow.
What leaves the box
PentAGI sends prompts, scan output and anything its tools collect to the model provider on every call. Strip credentials and client data you don't need from flows where you can.
On RedRouter, each API key has a store_content setting, and new keys default to storage off; check the current key with GET /v1/key. Account-wide zero data retention is a separate opt-in switch in Settings. It's off by default, and eligibility and pricing are on the data handling docs. Usage metadata such as model, token counts, cost and errors is still recorded. What RedRouter keeps explains the details, and the security page covers how RedRouter approaches authorized security work.
Your PentAGI host keeps its own copies: flow history in PostgreSQL, reports, and Langfuse traces if you run that stack. PentAGI's architecture docs also say it checks update.pentagi.com within two minutes of starting and every three hours after that, sending a summary of the installation; its docs explain how to turn that off.
FAQ
Does PentAGI support an OpenAI-compatible API?
Yes. PentAGI's custom LLM provider connects to any OpenAI-compatible API through LLM_SERVER_URL, LLM_SERVER_KEY and LLM_SERVER_MODEL. Its docs list vLLM, LiteLLM and several hosted services as examples. The backend must return valid Chat Completions tool calls.
What is the PentAGI custom endpoint URL for RedRouter?
LLM_SERVER_URL=https://redrouter.org/v1. PentAGI appends /chat/completions. Don't include the full path in the URL, or the request path doubles and returns a 404.
Which PentAGI LLM provider settings are required?
LLM_SERVER_URL and a non-empty LLM_SERVER_KEY, plus either LLM_SERVER_MODEL or a provider config file at LLM_SERVER_CONFIG_PATH that names a model for each agent type. An empty key disables the custom provider.
Can PentAGI use RedRouter for embeddings?
No. RedRouter has no embeddings API. Configure PentAGI's EMBEDDING_* settings with a provider that offers embeddings, such as a local Ollama model, and use RedRouter for chat and tool calls.
How do I test a PentAGI custom endpoint before running a flow?
Run docker exec -it pentagi /opt/pentagi/bin/ctester -verbose. It checks completion, JSON output and function calling for each agent type. Fix any failures before you start a flow, because a broken tool call stops the flow at its first step.
Is it legal to run PentAGI against a website?
Only with authorization: your own systems, an isolated lab, or a bug bounty or VDP target inside the program's published scope. PentAGI's EULA and RedRouter's terms both prohibit unauthorized testing.
Sources
All read on November 11, 2026 (PT).
- PentAGI docs: LLM providers, Embeddings, Quick start, Manual installation, Architecture
- VXControl on GitHub: PentAGI README, .env.example, custom-openai.provider.yml, custom provider source, EULA
- Google Threat Intelligence Group: ARTEX and Friends: Enter Agentic Offensive Frameworks (October 30, 2026)
- Help Net Security: ARTEX goes from pentesting tool to attacker staple in three months (November 9, 2026)
- LiteLLM: Drop unsupported params
- RedRouter Docs: API compatibility, Chat Completions, Models and pricing, Reasoning, Rate limits, Troubleshooting, Data handling
- RedRouter model catalog API (revision 2026-11-10.3-red)