How ShakerScan measures DAST recall on Juice Shop and crAPI, and what it misses

Summary

ShakerScan's deterministic Scan workflow is measured against two deliberately vulnerable lab applications, OWASP Juice Shop and crAPI. The release answer key names vulnerability classes, each anchored to a route; a separate miss-analysis key names only families and auth states. At v2.3.1 the enforced Juice Shop level is 3 of 9 expected classes (0.33), down from the 4 of 9 (0.44) that shipped from 2.0.0 through 2.2.0, because a proof-policy correction stopped promoting a reachable file to a verified confidential finding. On crAPI there is no valid passing artifact: the one run that briefly passed was invalidated by the project's own integrity ledger. This article explains the method, quotes the recorded numbers, and lists what did not work.

Question

What does ShakerScan actually find on a known-vulnerable target, how is that measured so it cannot be gamed, and what does it miss?

Environment

• Source: github.com/andriyze/shakerscan at tag v2.3.1 (annotated tag dated 2026-09-11).
• Documents quoted: benchmarks/README.md, benchmarks/juice_shop/expected.json, tests/fixtures/benchmarks/juice_shop.yaml, tests/benchmark/analyze_dast_benchmark.py, scanner/scanner_tools/benchmark_summary.py, docs/release-2.3.0-plan.md, docs/ledgers/benchmark-integrity-ledger.md, docs/hunt-architecture.md.
• The Juice Shop measurement cited below was recorded by the maintainers on 2026-09-06 on the then-current build. The crAPI entries were recorded on 2026-07-11. Nothing here was re-run for this article.
• Benchmark scorecards (results/benchmark-runs/*.json) are not in git. The repository's .gitignore states that release benchmark cards are uploaded as certification artifacts and their digests sealed into the certified receipt attached to each GitHub Release.

Methodology

Two answer keys. The release key, tests/fixtures/benchmarks/juice_shop.yaml, lists nine classes, each with a route anchor, minimum severity, and required proof; matching is "by class + route token, not app secrets". The miss-analysis key is simpler. benchmarks/README.md describes the fixtures as "expected vulnerability families and auth states per local lab target" that "encode only what classes of bug a strong scan should confirm and which principals should be exercised — never challenge names, routes, payloads, or scoreboard solutions." The public Juice Shop fixture expects sqli (critical), xss (high), and bola (high), each with at least one confirmed finding, across the anonymous and user1 auth states.

A miss analysis, not a score. tests/benchmark/analyze_dast_benchmark.py compares a scan report with the expected families and produces a miss analysis. The summary builder in scanner/scanner_tools/benchmark_summary.py reports discovery, attempts, auth_workflow, parameters, findings, misses, and proof_or_severity_gaps. A finding counts as confirmed only when it is verified or carries a high or proven confidence tier. Each miss carries a likely_root_cause: missing_required_auth_context, family_not_attempted, no_confirmed_finding_or_proof_gap, or expected_auth_state_untested. The module docstring says it "does not contain lab exploit solutions".

Detector inputs are gated. tests/test_detector_integrity_gate.py parses scanner.py and every module under scanner/scanner_tools and fails the release if product names, benchmark hostnames, or crAPI route prefixes (/community/api, /identity/api, /workshop/api) appear as string literals in detector code. It does not list Juice Shop routes. The ledger records why this gate exists (see below).

Honesty fields are hardcoded. benchmark_summary.py sets principal_identities_validated to False and authenticated_responses_accepted to None as constants. A scorecard cannot claim server-observed identity proof the harness does not have.

Corrections are appended, not erased. docs/ledgers/benchmark-integrity-ledger.md "records benchmark contamination, stale-fleet runs, methodology corrections, target-specific hardcoding discoveries, and score reinterpretations" and requires that entries "preserve the original claim instead of deleting it."

Results

Juice Shop

The release-level answer key has nine expected classes: sqli-search, sqli-login, xss-dom-search, xss-reflected, bfla-users, exposed-metrics, exposed-ftp-listing, exposed-confidential, and nosqli-reviews.

docs/release-2.3.0-plan.md states: "Juice Shop recall has sat at 0.44 (4 of 9 expected classes) against a 0.67 bar (6 of 9) across 2.0.0, 2.0.1, 2.1.0, and 2.2.0, each shipped with the DAST quality bar waived as declared debt." The measurement of 2026-09-06 records the per-class outcome:

The 2.3.0 objective was to raise recall from 4/9 to 6/9. The plan records that "two sessions of measured work ... fixed a real crash and restored the 0.44 baseline, and moved recall by exactly zero above it," after which the operator stopped that effort and set the quality bar to the shipped level.

At v2.3.1 the fixture was reconciled downward. tests/fixtures/benchmarks/juice_shop.yaml sets min_expected_recall: 0.33 with the comment "3 of 9: sqli-search + the two deterministic exposures (metrics, ftp listing)" and explains: "the exposure engine no longer promotes a file's mere reachability to a verified confidential finding — reachability is not confidentiality." exposed-confidential moved to a declared gap, and the declared gap count rose to six. The fixture states that this is "a characterized change to what the benchmark measures, not a blanket threshold cut," and that "a NEW miss, a recall drop below the reconciled level, or a seventh gap still fails." That change is the subject of a separate article.

crAPI

The crAPI fixture expects four findings: bola-vehicle-location, bola-mechanic-report, bola-orders, and sqli-coupon. The ledger records, all dated 2026-07-11:

1. First current-fleet authenticated run: honest FAIL. Two distinct JWT identities were minted and both auth lanes ran, but expected_recall: 0.0. None of the four vulnerable routes appeared anywhere in the report, because discovery never enumerated crAPI's authenticated API surface. The ledger calls this "an authenticated API discovery gap."
2. Seeded detector-isolation run. With the four routes supplied through the custom_endpoints scan option ("not a detector route list"), the BOLA differential fired on /workshop/api/shop/orders/{id} with cross-principal evidence, but was graded medium, verified: False, so the scorecard stayed 0/4.
3. Proof-tier fix, briefly 1/4. A promotion rule turned that differential into a high, verified finding and the gate passed.
4. Correction: the pass is invalid. The same day the ledger records that "the generic equivalent-response lane treated any non-requester identity in a body as proof of unauthorized ownership. That does not establish authorization failure and can promote public profiles or support/contact data." The rescored result is 1/4, overall FAIL, and "this correction reduces the current accepted crAPI recall claim back to no valid passing artifact."

Answer-key correction (2026-09-29, after v2.3.1). The ledger records that sqli-coupon pointed to /community/api/v2/coupon, which the pinned crAPI implements on MongoDB. The key now requires SQL injection at POST /workshop/api/shop/apply_coupon. Later authenticated scans still recorded 0/4 (same ledger entry; docs/releases/2.5.6.md).

Evidence

• Per-class table and reasons: docs/release-2.3.0-plan.md, section "A0 result (measured 2026-09-06, Juice Shop, current build)".
• Reconciled fixture: tests/fixtures/benchmarks/juice_shop.yaml at v2.3.1.
• crAPI sequence: docs/ledgers/benchmark-integrity-ledger.md, the four entries dated 2026-07-11.
• Earlier corrections in the same ledger: 2026-07-08 (retest loop did not improve recall), 2026-07-09 (product-specific route expansion removed; detector integrity gate widened), 2026-07-10 (browser proof was discarded by post-retest scoring, then persisted; corrected scorecard 4/9 to 5/9).
• Highest recorded Juice Shop score: 6/9 (0.67), passing, ledger entry 2026-07-11. It was anonymous, ran on a superseded fleet, was never re-run on the current build, and still missed the three auth-gated classes.
• crAPI answer-key correction: the ledger entry dated 2026-09-29.

What worked and what did not

• sqli-search and the /metrics and /ftp exposures are the stable finds. In the maintainers' 15 local Juice Shop scorecards (2026-09-04 to 2026-10-04), sqli-search was verified in 11.
• With crAPI routes supplied outside detector inputs, the BOLA differential fires. Authenticated API discovery, not proof, is the dominant crAPI gap.
• The hypothesis, refuter, and retest loop moved Juice Shop recall by zero (2026-07-08), and identity-only BOLA promotion was invalidated.

Limitations

• These are the maintainers' recorded measurements, quoted; this article did not re-run any scan.
• docs/release-2.3.0-plan.md still states 0.44 and five gaps. The fixture at the same tag states 0.33 and six gaps. The fixture is the enforced contract; the plan document was not updated.
• The expectation matcher had a recorded evidence-label defect (route_tokens() drops routes shorter than four characters, such as /ftp).
• Scorecard artifacts are release attachments, not repository files, so a reader cannot diff them from the source tree.

Reproduce

The 9-class recall comes from scripts/benchmark_targets.py, which submits one Scan with the fixture's policy and budget, mints two Juice Shop users, and writes a scorecard under results/benchmark-runs/. With Juice Shop on port 3001 and a ShakerScan API on port 8080:

git clone --branch v2.3.1 https://github.com/andriyze/shakerscan.git
cd shakerscan
python3 scripts/benchmark_targets.py juice_shop --auth --enforce-quality

Use --target-url juice_shop=<url> if the scanner reaches the target at an address other than http://host.docker.internal:3001. Only test systems you own or are authorized to assess.

Product implications

The plan adopted a governing rule from an external audit: "No change merges into detection that does not move recall on the funded thorough authenticated benchmark, and no new subsystem ships until finding capability demonstrably improves."

References

• Benchmark fixtures README
• Juice Shop release fixture
• Benchmark runner, miss-analysis script and summary builder
• Detector integrity gate
• Release 2.3.0 plan
• Benchmark integrity ledger
• Hunt architecture, current state
• Scan product page