Laptop terminal during a cyber competition, seen over a competitor's shoulder, with the article title about using an LLM for CTF challenges
Cover based on a photo by U.S. Air Force photo by Tech. Sgt. Robert Biermann, via Wikimedia Commons, Public domain

Using an LLM for CTF challenges works best when the model is a fast second brain, not the player: it reads unfamiliar code, suggests what a cipher or binary might be doing, and drafts the solve script, while you run the tools, check the output and decide what is true. That LLM CTF workflow is also how the strongest teams use AI today. It turns a long night of guessing into a shorter loop of hypothesis, test and keep or drop.

This guide covers the whole loop: whether AI is even allowed at your event, how to set up a sandbox, where an LLM helps in each CTF category (crypto, reversing, web, pwn, forensics and misc), the prompts that work, and how to avoid the most embarrassing failure, a confident model handing you a flag that doesn't exist.

CTFs are authorized targets by design, but only the targets the organizers name. Attack the challenge instances you were given and nothing else: not the scoreboard, not other teams, not the platform. Everything below assumes that scope.

Is using an LLM for CTF allowed?

It depends on the event, so read the rules before you paste a challenge into any model. Policies in 2026 run from permissive to strict:

  • The picoCTF 2026 competition rules say there are no limitations on the resources or tools participants can use, but only eligible team members may solve challenges, and asking people on the internet for help is cheating.
  • The National Cyber League rules allow AI tools to assist players, with limits. Players may not use AI agents to interact directly with the competition platform on their behalf at any time, such as connecting to platform terminals or web instances.
  • Some events ban generative AI outright, and others run a separate division for AI agents. If the rules don't mention AI, ask the organizers before the clock starts.

The distinction that matters is usually assistance versus autonomy. Asking a model to explain an RSA padding mistake is one thing. Pointing an agent at the platform and letting it submit flags is another, and several events now treat that as a rules violation.

What the evidence says LLMs can do in CTFs

Capable models now solve many easy and medium CTF challenges, which is why rules are tightening. A few data points, all from primary or near-primary sources:

  • Cybench, from Stanford researchers, collects 40 professional-level CTF tasks from four competitions. In the paper, agents without subtask guidance solved tasks that took the first human team up to 11 minutes, while the hardest task had taken humans 24 hours and 54 minutes.
  • NYU CTF Bench is an open benchmark of CTF challenges with a framework that lets models call external tools, which is the setup that matters for real solves.
  • In the AI vs. Humans CTF that Hack The Box ran with Palisade Research in March 2025, Hack The Box reports that five of the eight AI teams solved 19 of 20 crypto and reversing challenges. Palisade's report adds that on the larger Cyber Apocalypse event, where many challenges required interacting with remote machines, the best agent solved 20 of 62.
  • At Hack The Box's NeuroGrid CTF in November 2025, Help Net Security reports that AI-augmented teams worked faster, but the only team to finish all 36 challenges was human-only; the best AI team stopped at 32. In the 2026 Global Cyber Skills Benchmark, agents appeared most often on the strongest teams.

The pattern across these sources: models are fast on self-contained puzzles, and weaker on long, stateful challenges that need live interaction, debugging and judgment. That's where you come in.

Set up a sandbox first

Treat every challenge file as hostile. Reversing and forensics files can be real malware samples or simply break your machine.

  1. Work in a disposable VM or container with no access to your personal files, browser sessions or work credentials.
  2. Keep the challenge network separate. If the event gives you a VPN, connect only the sandbox.
  3. If you use a coding agent such as Claude Code, run it inside that sandbox, and require approval before it runs commands against remote hosts. The agent should only ever touch the instance you were assigned.
  4. Log what you run. You'll want it for the write-up.

If you use RedRouter, the Claude Code setup docs show the environment variables, and our Claude Code, OpenCode and Cline guide covers the other clients. Any OpenAI-compatible tool can use https://redrouter.org/v1 as its base URL; the OpenAI-compatible API guide has client-by-client settings.

Diagram of an LLM CTF workflow: check the event rules, triage the challenge files in a sandbox, ask the model for hypotheses by category, test each hypothesis with real tools such as pwntools, CyberChef or a debugger, and submit only a flag the tools actually produced, then write up what you learned
Diagram of an LLM CTF workflow: check the event rules, triage the challenge files in a sandbox, ask the model for hypotheses by category, test each hypothesis with real tools such as pwntools, CyberChef or a debugger, and submit only a flag the tools actually produced, then write up what you learned Image: RedRouter, Original

The AI CTF solver workflow, step by step

1. Triage before you prompt

Run file, strings, checksec, binwalk or exiftool yourself and look at the challenge description. Give the model that evidence, not just the raw file. A model reasoning from tool output makes far fewer wild guesses than one reasoning from a filename.

2. Ask for hypotheses, ranked

A prompt that works across categories:

CTF challenge (authorized, from <event>). Category: crypto.
Description: <paste>
Evidence: <file output, source snippet, sample ciphertext>
List the 3 most likely weaknesses, ranked. For each: why you think so,
the single test that would confirm or kill it, and the tool to run it.
Do not guess a flag.

The last line matters. You want testable ideas, not an answer.

3. Test with real tools

Run the test the model proposed. Use pwntools for exploit scripts and remote interaction, CyberChef for encodings and quick transforms, and a debugger or decompiler for binaries. Paste the actual output back. If the test kills the hypothesis, say so and ask for the next one.

4. Let the model draft the solve script, then read it

Models are good at the tedious part: a padding-oracle loop, a z3 constraint model or a pwntools ROP chain skeleton. Read every line before running it, and especially before pointing it at a remote instance.

5. Submit only a flag your tools produced

A flag must come out of your script, decoder or debugger, never from model text alone. Check that it matches the event's flag format before you submit. Models can and do invent plausible flags, and a hallucinated flag is exactly what organizers look for when they review suspicious solves.

6. Write it up yourself

Explain the bug, the solve and what you'd do differently. If you can't explain the solve without the model, you haven't learned it yet, and some events ask teams to explain suspicious solves.

Where an LLM helps in each CTF category

Crypto. Strong at recognizing classic mistakes such as reused nonces, small RSA exponents, weak padding or homemade ciphers, and at writing the math in Python or SageMath. Weak at long computations, so make it write code instead of doing arithmetic in the chat.

Reverse engineering. Paste decompiler output for one function at a time and ask what it does and what input reaches the success path. For bigger binaries, an MCP bridge to a decompiler helps; our GhidraMCP guide shows that setup.

Web. Good at reading source for injection points, auth logic and deserialization sinks. Our XSS analysis guide covers the reasoning patterns, which carry straight over to web CTF challenges. Keep requests aimed at your assigned instance only.

Pwn. Helpful for explaining the vulnerability class, computing offsets from your debugger output and drafting exploit skeletons. Expect to debug the exploit yourself; memory layout details are where models slip.

Forensics. Useful for suggesting where data hides in a format and which tool extracts it. Let the tools do the extraction.

Misc and OSINT. Treat these with care. OSINT challenges should stay within what the challenge asks for, and never extend to real people outside the puzzle.

Why refusals show up in CTF work

Mainstream assistants sometimes explain a buffer overflow in the abstract and then decline to write the exploit, even when the target is a CTF binary built to be exploited. That gap is common in pwn and reversing work. Our refusal archive collects public examples from security workflows, and the security page explains how RedRouter approaches authorized security work.

RedRouter is an OpenAI- and Anthropic-compatible API with models modified to reduce refusals for authorized work. A typical setup for CTF practice:

  • GLM 5.3 Flash Red (glm-5.3-flash-red): $0.25 per 1M input tokens, $0.10 per 1M cached input tokens and $1.00 per 1M output tokens, with a 1,048,576-token context window and tool calling. It suits fast, high-volume loops like hypothesis lists and script drafts. See its model page and pricing.
  • red-cyber-1 (Beta): $0.28 input, $0.10 cached input and $1.10 output per 1M tokens, also with a 1M-token context window and tool calling. The models docs describe it as RedRouter's featured model for authorized penetration testing and red-team work. See the red-cyber-1 page.

Prices can change, so check those pages before budgeting. Free credits explains the current credit offers.

On data handling: each RedRouter API key has a store_content setting, and account-wide zero data retention is an opt-in setting that is off by default. The data handling docs cover eligibility and details. Don't paste unreleased challenges into any hosted model if the organizers ask you not to.

Practice where AI use is clearly fine

  • Retired challenges and permanent practice platforms, after checking each platform's terms.
  • Your own lab: deliberately vulnerable apps and VMs on your own machine.
  • Past events listed on CTFtime, many of which publish challenge files after the event ends.

A good learning pattern: solve unaided for a fixed time, then use the model to explain what you missed. You'll keep the speed benefit without skipping the skill. When you're ready to turn CTF skills into real findings, the AI bug bounty workflow covers scoped programs, and the recon guide covers attack-surface mapping.

FAQ

Is using ChatGPT or another LLM for CTF cheating?

Only if the event's rules prohibit it. picoCTF 2026 places no limits on tools but requires team members to solve the challenges themselves. The National Cyber League allows AI assistance but forbids AI agents interacting with its platform. Other events ban AI or run separate AI divisions.

Can an LLM solve CTF challenges on its own?

Many easy and medium ones, according to Cybench, NYU CTF Bench and Hack The Box's AI events. Long, stateful challenges that need live interaction still trip agents up, and a human-only team was the only one to finish every challenge at NeuroGrid.

What's the best way to use Claude Code for CTF?

Run it inside a disposable sandbox, give it tool output rather than raw guesses, require approval for remote commands, and only accept flags that your scripts actually produce. Check the event's AI rules first.

Why does my AI assistant refuse CTF exploit requests?

General-purpose safety filters often can't tell a CTF binary from a real target, especially for pwn and reversing. Stating the event and scope helps sometimes. A model built for authorized security work refuses less often.

Which LLM is good for CTF practice?

One with tool calling, a long context for decompiler output, and few refusals on exploit code. On RedRouter, GLM 5.3 Flash Red covers fast loops and red-cyber-1 is the featured model for authorized pentesting.

Sources

All read on November 11, 2026 (PT).