Why Hunt added zero verified classes over Scan on Juice Shop

Summary

Before building more of Hunt, the maintainers measured what it added over the deterministic Scan on Juice Shop. The recorded answer: nothing. Scan alone reached 4 of 9 answer-key classes; Scan plus Hunt reached the same 4. The plan calls the reasons "specific and code-confirmed, not a tuning gap": the credential primitive Hunt's cross-principal proof needed could not authenticate a JSON-plus-JWT login, and Hunt's endpoint knowledge was a raw list of about 3,000 URLs, roughly two-thirds of them content-discovery phantoms. One of the two has since been fixed at the code level. No verified Hunt-only finding is recorded at v2.3.1, and the repository says so.

Question

Does letting a coding agent investigate adaptively find anything the deterministic pipeline does not, on a target where the answer key is known?

Environment

• Source: tag v2.3.1; documents docs/release-2.3.0-plan.md (A0 result), docs/hunt-architecture.md (current state and phase mapping), docs/hunt-investigation-evaluation.md (evaluation protocol), docs/hunt-review-integrity.md, AGENTS.md.
• The measurement is the maintainers' A0 workstream, "measured 2026-09-06, Juice Shop, current build", using "the existing keyless Hunt flow" with "no new build".

Methodology

A0 was defined as: "Measure Scan alone vs Scan + Hunt on the benchmark target with the existing keyless Hunt flow. No new build." The gate was "a recorded table: classes found by Scan, by Hunt, by both; zero false verified findings."

Two rules shape what can count:

• Invariant 7 in AGENTS.md: "AI may create notes, observations, and evidence-backed candidates; only deterministic proof contracts may mark findings verified." docs/hunt-architecture.md restates it as "AI reasoning is not proof."
• The answer key is the same nine-class Juice Shop fixture used for Scan (see the benchmark article).

Results

The plan's conclusion: "Scan alone reaches 4/9. Hunt adds zero verified classes over Scan on this target today."

Reason 1: the auth/session primitives did not converge. Hunt's only deterministic cross-principal proof, authz.verify, required two interactive sessions from auth.session.establish, which supported exactly form_login, oauth_client_credentials, and oauth_password. Juice Shop authenticates with a JSON login that returns a JWT in the body. A bearer-token profile was "rejected outright", and form login found no HTML form. So Hunt could not establish the sessions its BOLA/BFLA proof needed "on the exact class of target (JSON + JWT) that is the common case", while Scan authenticated the same target fine.

Reason 2: the endpoint knowledge base was unstructured and noisy. Hunt's prior-knowledge inventory for the target was "~3,000 endpoints, of which ~2,000 are content-discovery phantoms ... all with an identical generic id,limit,offset,page,token param shape. A reasoning loop handed this raw cannot tell a real route from wordlist noise."

Evidence

• docs/release-2.3.0-plan.md, section "A0 result (measured 2026-09-06, Juice Shop, current build)".
• docs/hunt-architecture.md, section "Current state (measured 2026-09-06/07)", which records the same table and marks what has changed since.

What worked

• The proof boundary held: Hunt "creates only unverified candidates; verification runs through the deterministic proof moat", and the A0 gate's "zero false verified findings" condition was met.
• Reason 1 has a recorded fix. docs/hunt-architecture.md states that a json_login session auth kind (commit a0e0511a) "converges the credential primitive so a JSON+JWT target drives authz.verify through the same registry entry Scan uses. Live-verified: both principals establish sessions and the proof runs with them recognized as distinct." This is the first converged primitive of the shared capability layer.
• An opt-in grouped endpoint frontier (kind="endpoint_groups" on /hunts/{id}/query) was implemented and verified against the running API, so drill-down loses no lead.

What did not work

• No verified Hunt-only finding exists at this tag. The document is explicit: "First verified Hunt finding: not yet." The needed targeted-id BOLA proof "is designed and fail-closed" but scoped for "a focused build with positive-and-negative validation, not shipped opportunistically", because "a new proof path is the most false-positive-sensitive change in the system."
• Endpoint grouping was made deliberately conservative after status-only inference was removed as unsupported, and "no measured live de-noising or recall improvement is claimed."
• The workstream-to-phase table lists the adaptive reasoning loop as "blocked on the targeted-id proof + Phase 2", and human-plus-AI benchmarks as "not started".

Limitations

• One target, one build, one measurement date. A0 is a baseline table, not a study.
• The independent evaluation protocol in docs/hunt-investigation-evaluation.md has not produced a measured autonomous-planner result; the document states that its fixtures and scoring tests "validate the evaluator itself. They do not constitute a measured autonomous-planner benchmark result."
• docs/hunt-review-integrity.md records "two positive leads and the no-lead control" as acceptance evidence, "not a human-productivity study or verified recall gain."
• Hunt depends on the operator's Codex, Claude Code, or OpenCode session; model choice and version are variables the A0 note does not fix.

Reproduce

The evaluation protocol the maintainers require before any efficacy claim, from docs/hunt-investigation-evaluation.md:

1. Use authorized disposable targets with baseline and patched versions, pinned images, and state reset between runs.
2. Give the planner only its objective, exact target, opaque principal references, policy, budgets, and the canonical Hunt API. Do not supply expected weaknesses.
3. Keep an operator-only oracle outside the planner's workspace. Run at least three paired runs per revision, recording the Git revision, worker fingerprint, model version, and fixture digest.
4. Export each terminal Hunt with GET /hunts/{id}/record and fetch GET /findings/{id} for every finding the action trace references.
5. Score offline:

python scripts/score_hunt_investigation.py --record record.json --findings findings.json --oracle oracle.json

The scorer's docstring: it "consumes unmodified API exports, not planner claims. Never invokes a capability, starts a Hunt, or sends the hidden oracle to a model." Thresholds must be set before the runs, and every run, including failures, must be reported.

Product implications

• The Hunt product page carries a maturity note for this reason: Hunt ships, its efficacy is release-gated, and AI leads are hypotheses until deterministic proof confirms them.
• The recorded ordering of work is discovery and shared primitives first, then the reasoning loop, then deep workflow reasoning. The success axis is planned to move from recall-only to human-plus-AI benchmarks.

References

• Release 2.3.0 plan, A0 result
• Hunt architecture
• Hunt investigation evaluation protocol
• Hunt evidence review and continuity
• Offline scorer
• Hunt product page