Adversarial verifier (LLM review)

Single-pass LLM code review has a well-documented failure mode: the model produces plausible-sounding findings that don't survive contact with the actual code. A finding might reference a variable that doesn't exist, claim a race condition that isn't reachable, or flag a security issue in a code path that never runs.

You can push false-positive rates down with better prompting, structured output, and evidence-quoting — but you can't get them to zero. What you can do is have a second model check the first one's work.

That's the adversarial verifier. It's not a duplicate reviewer; it's a critic. Its prompt says "here is a finding another agent produced; find the strongest reason it might be wrong." If the verifier can articulate a plausible refutation, the finding is dropped.

After the 6 review agents (bugs, security, performance, readability, best-practices, documentation) complete, LGTM assembles the deduplicated list of findings. Each finding — with its evidenceQuote, category, severity, and reasoning — is sent to the verifier LLM with a fixed prompt: "Argue this finding is a false positive. Consider whether the code path is reachable, whether the flagged pattern is a real vulnerability in this framework, whether the evidence quote actually shows the described problem, whether the finding contradicts the repository's stated conventions."

The verifier returns a structured object with a keep / drop verdict and a refutation string. Dropped findings are logged (so we can measure the verifier's precision over time) but don't post to the PR.

Concurrency is capped at 6 to stay within provider rate limits. In practice the verifier adds 3-8 seconds to the tail latency of the review — well within the overall 30-90s budget.

The critical design choice is the prompt orientation. A duplicate reviewer with "find the bugs" produces its own findings — often confabulating alongside the primary agent. A verifier with "argue this is wrong" is oriented toward scepticism, which is the underrepresented direction in a naïve pipeline.

The verifier also has more information than the primary agent had: it can see the finding, the diff, the surrounding file context, and the repository's conventions. That extra context is exactly what enables it to spot "this finding is technically true in vanilla PHP but Laravel's default middleware already handles it."

Combined with the evidenceQuote gate (every finding must quote the offending code verbatim), the verifier catches roughly the top 15-25% of hallucinated findings — the ones that pass the syntactic quote check but fail the semantic reachability check.

A verifier is not free. It costs one LLM call per finding, which on the strong tier is the dominant per-review cost after the primary agents. LGTM uses the same-tier model as the primary agents (not a cheaper one) because a mini-tier verifier tends to defer too easily and lets weak findings through.

A verifier can also introduce false negatives — genuine findings the verifier successfully refutes. We measure this via the golden-set eval harness: fixture PRs with known-good findings, run through the full pipeline, and any finding the verifier drops that our human ground-truth kept counts against verifier precision.

The pragmatic answer is: verifier-off is worse than verifier-on for typical PRs, and the false-negative floor is small enough that Actionable-tier findings survive at ≥95%.