Security Tests Are Becoming Part of the Generation Specification
A coding agent is asked to build a URL validator. It receives functional requirements and security tests, generates an implementation, and runs it in a sandbox.
The tests turn green.
For an engineering team deciding whether that code can move forward, the important distinction is whether the agent implemented the broader security requirement or only learned enough to satisfy the examples it was given.
An August 2026 experiment suggests those are not always the same thing.
Researchers tested security requirements as executable inputs to AI code generation across 2,705 generation trajectories. Supplying visible functional and security tests before generation improved hidden functional-and-security success in seven of nine model-and-benchmark comparisons, with a macro-average gain of 19.3 percentage points.
The intervention generally improved performance, but it also increased the gap between passing visible tests and succeeding against unseen security cases. That tradeoff matters if test results are going to determine when AI-generated code is allowed to progress.
A 2024 study found that supplying functional tests alongside programming requirements could improve code-generation results. The August 2026 study, Security Tests as Executable Specifications for LLM Code Generation, extends that approach to security, evaluating 31 Python task instances representing 16 CWE categories across three secure-code benchmarks and five models.
How the Mechanism Works
In the SecTDD workflow, visible functional and security tests can be placed in the model's context alongside the programming requirement before code is generated.
Candidate code then runs in a fresh sandbox. Test execution and pass-or-fail decisions are deterministic. When a visible test fails, the result can be returned to the model to guide a revised implementation.
The final candidate is evaluated against hidden functional and security tests that the model never saw. Those hidden tests introduce different behavior families, including alternative encodings, delimiters, nesting, boundary values and attack constructions. They are designed to distinguish an implementation that captured a broader security property from one that fitted the visible examples.
The resulting architecture has a clear division of labor. Generation, interpretation and repair remain probabilistic. Test execution, pass-or-fail decisions and stopping rules can be deterministic.
But when the deterministic layer says "pass," the generation loop stops. The quality of that decision depends on what the tests were capable of detecting.
Analysis
Across nine model-and-benchmark comparisons, supplying functional and security tests upfront improved hidden joint success in seven, with a macro-average gain of 19.3 percentage points, while performance declined in two. None of the reported pairwise comparisons reached corrected statistical significance at p < .05, limiting how broadly those improvements can be generalized.
The more consequential finding appears after the visible tests turn green.
Among candidates that passed all visible tests, 5.6 percent in the requirement-only baseline subsequently failed hidden joint tests. When tests were supplied upfront, that rate increased to 18.8 percent. Passing the visible suite therefore became a less reliable indicator of performance against unseen security behavior.
One URL-validation example shows the gap. Generated code passed the visible checks but failed hidden cases involving a deep subdomain, authority-userinfo syntax and encoded delimiters. The model had produced an implementation that satisfied the behaviors represented in the visible suite without fully capturing the underlying validation policy.
For a coding agent, green tests can be a stopping condition. For the team designing the harness, they have to mean something more precise: the generated implementation passed the security behaviors this suite knows how to ask about.
That distinction moves the problem from model performance into engineering design.
Implications for Enterprises
Consider the team deciding what happens after the agent receives that green result. If passing the suite automatically advances code toward a pull request or another pipeline stage, someone has effectively decided that those tests constitute sufficient evidence for that transition.
That makes security-test design part of the control architecture around the coding agent.
The study includes tasks involving injection flaws, path traversal, cryptographic weaknesses and validation behavior, showing how some localized security properties can be represented through executable tests.
Other controls, including static analysis, dependency checks, fuzzing, property-based testing and deeper CI/CD validation, can provide independent signals for failure surfaces that generation-time tests do not cover.
July 2026 research on specification grounding adds another consideration. Under comparable testing and repair budgets, tests grounded in an explicit specification performed substantially better than simply increasing the quantity of ungrounded tests. More tests are therefore not necessarily equivalent to a better specification.
The platform decision is not simply whether to give a coding agent security tests. It is which security properties should be executable, which tools should enforce them, and what combination of evidence is sufficient to let generated code move forward.
Risks and Open Questions
The hardest issue is the specification gap itself. Localized behaviors such as input validation may be relatively straightforward to encode, while multi-tenant authorization, distributed state, concurrency and business-logic invariants can depend on identities, resources, request sequences and side effects. Teams therefore need to decide which behaviors the suite represents, which remain outside it, and who owns that boundary as the application changes.
The acceptance environment also has to remain trustworthy. Hidden tests need to remain isolated from generation and stopping decisions; if the generation process can inspect the hidden oracle, it no longer provides an independent measure of generalization. The evidence for this approach is also still narrow: SecTDD evaluated relatively small Python components and does not establish equivalent results for repository-scale systems, memory-unsafe languages, concurrent applications or complex enterprise authorization architectures.
The practical question is therefore not whether security tests can make probabilistic code generation deterministic. They cannot. The more useful question is where enterprises can place deterministic boundaries around generation, and what evidence those boundaries should require before the system is allowed to say: this code can move forward.
Further Reading
- Security Tests as Executable Specifications for LLM Code Generation: Benefits, Trade-offs, and Coverage Limits
- Specification Grounding Drives Test Effectiveness for LLM Code
- Test-Driven Development for Code Generation
- Constrained Decoding for Secure Code Generation
- Rethinking the Evaluation of Secure Code Generation
- Use Property-Based Testing to Bridge LLM Code Generation and Validation