How well do agents use test/verification techniques?

Testing makes this especially clear: “use fuzzing” turns into generating random bytes, and “use formal verification” into proving a property that isn’t particularly useful. Formally, the technique is applied, but its actual purpose is lost.

This may also explain why different testing techniques produce similar results: the problem may be less about knowing a particular technique and more about the agent’s ability to keep its actual goal as the primary constraint.

The way you test code cannot (and should not) be decoupled by the way in which you architect the code itself.

80%+ of effective testing is not in the testing framework but in the code architecture.

The author doesn't mention how the code is being architected and managed.

For what it is worth, I found that forcing agents on DI/hexagonal architecture and forcing a trivial coverage check is quite useful and produce overall good enough code with relative little effort

How can I inspect exactly how agents were prompted? How can I reproduce this setup and try out my own AI setup? Am I missing a link to a repo somewhere?

These evals probably match how many people are using AI, but its not the full package of how we know software development needs to be done, and not how I do it with AI. The closest would be "Audit" which scored the highest when using xhigh- that actually incorporates a review cycle- something we know is the most important part of the software development process for code correctness (design/specification is not as much of an issue in this problem since the task is to write code against an existing spec). However, we don't know what instructions they have in their "Audit".

I would love to benchmark my own flow [1] if I can be given their exact problem. What it does is (assuming there is already a solid spec)

  * plan with expensive model. Review the plan.
  * implement with cheap model. Review for spec compliance and code quality.
  * Reviews are done adversarially from the expensive model with a fresh context.
  * ensure that verifications (automated or manual) are performed.

The workflow does use TDD. I do find useless tests being written and I need to dig into this part of the workflow a lot more, so its great to see that aspect of this article. My experience writing software has taught me that code must be written to be easy to test, but not necessarily done TDD style.

[1] https://github.com/gregwebs/skills-sdlc/

For example, when I use the Superpowers skill set, the agent proactively adopts TDD for every new feature. But its understanding of testing often stays superficial: if the user asks for a screen with a “Send” button, it first writes a test checking whether the button exists. The test fails, so it adds the button to make it pass.

As a result, the test suite fills up with low-value cases that check whether a property exists or a string matches exactly. The agent follows the “write a failing test, then implement the feature” workflow, but never really tests the business behavior: When should sending be allowed? What should happen after success or failure? How should duplicate submissions be handled?

The problem isn’t that agents can’t write tests. It’s that they seem prone to reducing TDD to a rigid sequence of steps, struggling to independently derive meaningful test cases from business requirements and use them to drive development.

But once you got there, Claude had no trouble proving the code satisfied the spec, and I uncovered a few interesting bugs this way.

Also, had a funny experience where AI implemented an architectural change completely backwards. The implementation was pointless and made things worse rather than better. But I still got "all tests green" lol, because it just proved that the incorrect thing worked properly.

I noted with some amusement that formal verification wouldn't have helped there either, it would just have been an even stronger proof of the "correctness" of the thing that shouldn't exist to begin with.

The agent really, really struggled to bridge the gap between the code and the actual business rules it was meant to be modelling. It also struggled to work out which functions at which layer were appropriate to write tests for. So, its tests tended to be very brittle to changes to domain logic.

Essentially, agents have always seemed to struggle with modularity and problem decomposition. Good testing is about finding the right things to test, which means working out how to subdivide the input state into an appropriate product state, and checking each behaviour independently. IMO, this is the most difficult and complicated thing about programming, so I won't say it struggled _more_ than a human would --- but humans have the advantage of being able to sleep on it?

One minor thing I observed was that it tended to get really hung up on floating point edge cases (NaNs, infinities). Maybe floating point edge cases are over-represented in the property-testing training data, but it's essentially irrelevant for my usecase, at least as far as the business rules go.

The article claims that the agents didn’t actually use TDD or mutation testing. While it’s possible for an agent to ignore the TDD procedure even when instructed to follow it, it can’t ignore a build failure.

   try to make illegal states unrepresentable

So it's understandable that the agents wont be able to go beyond proving trival things, given how much more difficult it is to write such code.

A (more) interesting experiment (to me) would be to write a high level spec manually for a non-trivial system (liveness etc.) and see if the agent can produce an implementation using guided refinements that satisfies this specification.

A big, big part of this space is "traceability" through the SDLC. If something goes wrong, there is a long chain of liability from the final effect (a subject with an adverse event) to the root cause (system recorded the wrong data in a clinical trial) all the ways to a 3rd party vendor. We call software "validated" if it has gone through the GAMP 5 V-model of software development and produced the necessary artifacts that correlate verified behavior to specification.

While this is way too much rigor for many scenarios, I think more folks should be familiar with the GAMP 5 V-model in the agentic era. The left side of the V defines the requirements moving from high level business requirements to low level technical requirements, the right side of the V defines the verification artifacts corresponding to the technical and business requirements (in that order). In this structure, testing covers both the functional requirements as well as the technical requirements.

I think this mental model is extremely useful and reflects the interaction model with agents: humans now focus on the requirements and less on the implementation details. It is useful to separate the business from the technical and therefore, the two types of artifacts that need to be produced to satisfy verification thresholds.

This style of structured software development (pre-agents) is very expensive. With a 2-3 weeks planning/specification phase on the front-end (since the test specification has a dependency on the approved requirements), a 4 week implementation phase (this is iterative with the test team, but informal; changes from the approved plan are documented as deviations), and then a 2-3 week formal test/verification phase. But in a post-agent world, this model feels like it is 1) more reasonable, 2) perhaps more effective, 3) more manageable.

While the teams I've now been on in startups have focused heavily on fast iteration with AI, a big downside I've noted is that it seems that we keep building the wrong things or things that are not useful because we are no longer verifying the business requirements before building. We assume the cost of building is low so we build and the iterate; testing in this reality becomes ad-hoc and full of gaps because the functional behavior of the software is no longer scoped before building; it's "vibes" based on each iteration because the iterations are cheap. The result? The teams I've been on feel like we're going nowhere in many cases, just spinning faster and producing more throwaway code (not a comment on whether this is good or bad; it may vary by domain and nature of the company).

If one is looking to build a software factory or pipeline, I think it behooves an architect to examine the GAMP 5 V-model and consider how to adapt ideas from this model to agent systems.

The whole premise of the question is hilarious. They change text in an existing one-line comment and the best models in the world think, gee, maybe I'll lint everything AND run 4000 units. Maybe 5000 integration tests too, just to ensure we collide with any other work in progress. So you write the obligatory but often-ignored obvious things into agent memory or project steering markdown or periodic nudges: You must have a hypothesis when you run expensive tests, you must spot check changes first, then start with the most relevant tests only, then move outwards only as necessary to broader labels and only then suites and only then ALL suites in a widening gyre.

But like a falcon ignoring the falconer, the models want to run the everything for anything. So you sigh, you get the model to write a deterministic hook to catch the wrong invocation of the test suite, and you spend weeks refining the rules every time you hit a edge-case, and so it goes. At least you don't have to write the involved regexes by hand, and maybe one day it will be finished..

> Perhaps the limiting factor is just that knowledge of effective test techniques isn't very widespread

That linked to: https://danluu.com/testing/

But of course... the problem of testing is a lot harder than performance optimization... I'm surprised this comes as a surprise. Performance optimization has plenty of evaluation metrics by its very nature. Testing? -- I wish there was anything tangible at all... Because we have metrics for optimization, we have theories of optimization, i.e. we have a way of explaining how or what optimization should do. With testing? -- we are nowhere close to this point.

Another aspect of this disparity is that we also know how to sell performance optimizations. It's easy to write into an ad pamphlet that the version 2.0 of gobbledygook does 185% more gobbledygook than the 1.0! (The number faithfully copied from my cereal box!) With testing? -- How can you even tell the customer that the product was tested better? Swear on your life and cross your heart (twice, as opposed to the last time when you only did it once?)

In general, in the field, I've only have so far met with extreme pessimism about feasibility of "theory of testing" existing. Even though the need for testing goes without saying, the actual testing task is reserved for the least competent and there's little no no effort made to improve anything in this department as it's perceived to be a black hole in the budget: no matter how much you could spend on testing, the effect is likely to be the same.

It's been interesting investing into different classic verification & testing methods for gfql (CPU/GPU graph queries on dataframes) and Louie (agentic investigation harness) over the last couple of years with different model & harness generations. The ones below are fewer but more longitudinal efforts compared to the shallower ones Dan worked through, and seem largely consistent:

Overall, my takeaway on standard software has been LLMs favor smart, guided fuzzing, natural language specifications deployed as iterative parallel adverserial AI review (e.g., context-reset subagents) after the functional prototyping. For formal methods / static analysis, cheaper/faster/lighter ones, though for the next level of quality, I can see this changing. The concolic testing world's heuristic results carried through a lot for me vs most others, which is unsurprising as they went deep on engineering ROI curves for fuzzing large code bases and specializing for different bug classes.

Highlights:

- alloy for gfql fell on its face relative to fuzzing, even with guidance. This was disappointing as it was intended as a cheap experiment to justify doing more expensive formal methods.

- prompts & skills need tuning: auto-memory doesn't transfer across harnesses by default, and one-shot auto-authored skills evals badly. More about iterating. Manual version of iterating would be editing skills when we hit new/repeat classes of bugs, though not guaranteed faithfulness: this is what we do the most. Positive experiments automating here, but not enough to invest deeper when we have to do manual anyways.

- natural language specifications are now a thing. Our significant new features now come with a security.md, policy.md, concurrency.md, etc., and important for them to be close to the code and taught to the review skill. Coding and review agents then can triangulate between tests, specs, code, and their own skills & general knowledge. However, we find we prefer not to do localized invariants in method comments as that gets verbose and drifts, and instead, do those as tests. This gets a bit into the global cost question of better code gen iterations and/or better review phases.

- staging coding vs reviewing. Overtesting early kills progress, so we stage heavier quality engineering at the end, and architectural research at the beginning.

- Lessons from the concolic execution era: staging static LLM analysis early with dynamic LLM testing later. Test amplify findings by area, kind, etc to convergence.

- We love community suites. GFQL builds against known Cypher language standards correctness conformance and benchmark suites, and Louie tools often start with community agent evals/benchmarks before we add our own specializations.

I don't know how big and deep the experiment Dan ran was. Something like formal methods is generally a big & invasive investment, and the target of each kind is often a much higher quality rating for a narrow set of properties. This complicates benchmarking setup. Imagine building entire compilers, and experimenting with different combos of methods & having those methods come in at different times & places.

I expect the AI security vulnerability analysis world to have similar findings. We end up baking that into our review harness as well without changing our overall methods above.