How ShakerScan measures DAST recall on Juice Shop and crAPI, and what it misses

Summary

ShakerScan's deterministic Scan workflow is measured against two deliberately vulnerable lab applications, OWASP Juice Shop and crAPI. The answer keys encode vulnerability families and required auth states, never routes or payloads, and a miss analysis explains each gap. At v2.3.1 the enforced Juice Shop level is 3 of 9 expected classes (0.33), down from the 4 of 9 (0.44) that shipped from 2.0.0 through 2.2.0, because a proof-policy correction stopped promoting a reachable file to a verified confidential finding. On crAPI there is no valid passing artifact: the one run that briefly passed was invalidated by the project's own integrity ledger. This article explains the method, quotes the recorded numbers, and lists what did not work.

Question

What does ShakerScan actually find on a known-vulnerable target, how is that measured so it cannot be gamed, and what does it miss?

Environment

• Source: github.com/andriyze/shakerscan at tag v2.3.1 (annotated tag dated 2026-09-11).
• Documents quoted: benchmarks/README.md, benchmarks/juice_shop/expected.json, tests/fixtures/benchmarks/juice_shop.yaml, tests/benchmark/analyze_dast_benchmark.py, scanner/scanner_tools/benchmark_summary.py, docs/release-2.3.0-plan.md, docs/ledgers/benchmark-integrity-ledger.md, docs/hunt-architecture.md.
• The Juice Shop measurement cited below was recorded by the maintainers on 2026-09-06 on the then-current build. The crAPI entries were recorded on 2026-07-11. Nothing here was re-run for this article.
• Benchmark scorecards (results/benchmark-runs/*.json) are not in git. The repository's .gitignore states that release benchmark cards are uploaded as certification artifacts and their digests sealed into the certified receipt attached to each GitHub Release.

Methodology

Answer keys encode families, not solutions. benchmarks/README.md describes the fixtures as "expected vulnerability families and auth states per local lab target" that "encode only what classes of bug a strong scan should confirm and which principals should be exercised — never challenge names, routes, payloads, or scoreboard solutions." The public Juice Shop fixture expects sqli (critical), xss (high), and bola (high), each with at least one confirmed finding, across the anonymous and user1 auth states.

A miss analysis, not a score. tests/benchmark/analyze_dast_benchmark.py compares a scan report with the expected families and produces a miss analysis. The summary builder in scanner/scanner_tools/benchmark_summary.py reports discovery, attempts, auth_workflow, parameters, findings, misses, and proof_or_severity_gaps. A finding counts as confirmed only when it is verified or carries a high or proven confidence tier. Each miss carries a likely_root_cause: missing_required_auth_context, family_not_attempted, or no_confirmed_finding_or_proof_gap. The module docstring says it "does not contain lab exploit solutions".

Detector inputs are gated. tests/test_detector_integrity_gate.py parses scanner.py and every module under scanner/scanner_tools and fails the release if benchmark hostnames, product nouns, or answer-key routes appear in detector code. The ledger records why this gate exists (see below).

Honesty fields are hardcoded. benchmark_summary.py sets principal_identities_validated to False and authenticated_responses_accepted to None as constants. A scorecard cannot claim server-observed identity proof the harness does not have.

Corrections are appended, not erased. docs/ledgers/benchmark-integrity-ledger.md "records benchmark contamination, stale-fleet runs, methodology corrections, target-specific hardcoding discoveries, and score reinterpretations" and requires that entries "preserve the original claim instead of deleting it."

Results

Juice Shop

The release-level answer key has nine expected classes: sqli-search, sqli-login, xss-dom-search, xss-reflected, bfla-users, exposed-metrics, exposed-ftp-listing, exposed-confidential, and nosqli-reviews.

docs/release-2.3.0-plan.md states: "Juice Shop recall has sat at 0.44 (4 of 9 expected classes) against a 0.67 bar (6 of 9) across 2.0.0, 2.0.1, 2.1.0, and 2.2.0, each shipped with the DAST quality bar waived as declared debt." The measurement of 2026-09-06 records the per-class outcome:

The 2.3.0 objective was to raise recall from 4/9 to 6/9. The plan records that "two sessions of measured work ... fixed a real crash and restored the 0.44 baseline, and moved recall by exactly zero above it," after which the operator stopped that effort and set the quality bar to the shipped level.

At v2.3.1 the fixture was reconciled downward. tests/fixtures/benchmarks/juice_shop.yaml sets min_expected_recall: 0.33 with the comment "3 of 9: sqli-search + the two deterministic exposures (metrics, ftp listing)" and explains: "the exposure engine no longer promotes a file's mere reachability to a verified confidential finding — reachability is not confidentiality." exposed-confidential moved to a declared gap, and the declared gap count rose to six. The fixture states that this is "a characterized change to what the benchmark measures, not a blanket threshold cut," and that "a NEW miss, a recall drop below the reconciled level, or a seventh gap still fails." That change is the subject of a separate article.

crAPI

The crAPI fixture expects four findings: bola-vehicle-location, bola-mechanic-report, bola-orders, and sqli-coupon. The ledger records, all dated 2026-07-11:

1. First current-fleet authenticated run: honest FAIL. Two distinct JWT identities were minted and both auth lanes ran, but expected_recall: 0.0. None of the four vulnerable routes appeared anywhere in the report, because discovery never enumerated crAPI's authenticated API surface. The ledger calls this "an authenticated API discovery gap."
2. Seeded detector-isolation run. With the four routes supplied through the custom_endpoints scan option ("not a detector route list"), the BOLA differential fired on /workshop/api/shop/orders/{id} with cross-principal evidence, but was graded medium, verified: False, so the scorecard stayed 0/4.
3. Proof-tier fix, briefly 1/4. A promotion rule turned that differential into a high, verified finding and the gate passed.
4. Correction: the pass is invalid. The same day the ledger records that "the generic equivalent-response lane treated any non-requester identity in a body as proof of unauthorized ownership. That does not establish authorization failure and can promote public profiles or support/contact data." The rescored result is 1/4, overall FAIL, and "this correction reduces the current accepted crAPI recall claim back to no valid passing artifact."

Evidence

• Per-class table and reasons: docs/release-2.3.0-plan.md, section "A0 result (measured 2026-09-06, Juice Shop, current build)".
• Reconciled fixture: tests/fixtures/benchmarks/juice_shop.yaml at v2.3.1.
• crAPI sequence: docs/ledgers/benchmark-integrity-ledger.md, the four entries dated 2026-07-11.
• Earlier corrections in the same ledger: 2026-07-08 (retest loop did not improve recall), 2026-07-09 (product-specific route expansion removed; detector integrity gate widened), 2026-07-10 (browser proof was discarded by post-retest scoring, then persisted; corrected scorecard 4/9 to 5/9).

What worked

• Deterministic SQL injection proof on the search endpoint and the deterministic server-state exposures (a browsable /ftp index and a served /metrics endpoint) are found repeatably.
• Two-principal authentication bootstrap on crAPI works end to end on the current fleet.
• When routes are supplied outside detector inputs, the BOLA differential fires. The engine's proof step is not the primary blocker; discovery is.
• The retest scoring bug that discarded a recorded browser proof was fixed so post-processing stays additive to original proof artifacts.

What did not work

• The hypothesis, refuter, and retest loop moved Juice Shop recall by zero on the 2026-07-08 artifact.
• Route expansion that mapped generic nouns onto crAPI service mounts was removed as target-specific fitting, and the detector integrity gate was widened after it was found to inspect only two modules.
• Identity-only BOLA promotion was invalidated; the corrected rule requires an independent control such as an owner object absent from the attacker's authorized listing.
• Authenticated API discovery on crAPI remains the dominant gap. The ledger's proposed fixes are universal techniques: OpenAPI ingestion, OPTIONS and JSON-link discovery, and JS/browser endpoint extraction.

Limitations

• These are the maintainers' recorded measurements, quoted; this article did not re-run any scan.
• docs/release-2.3.0-plan.md still states 0.44 and five gaps. The fixture at the same tag states 0.33 and six gaps. The fixture is the enforced contract; the plan document was not updated.
• The passing Juice Shop scorecards in the ledger were anonymous runs; the auth-gated classes (xss-reflected, bfla-users, nosqli-reviews) remained missed.
• The expectation matcher had a recorded evidence-label defect (route_tokens() drops routes shorter than four characters, such as /ftp).
• Scorecard artifacts are release attachments, not repository files, so a reader cannot diff them from the source tree.

Reproduce

Run Juice Shop locally (for example the official Docker image on port 3001), install ShakerScan, register the target, and run an authorized thorough scan:

curl -fsSL https://install.shakerscan.com | sh
shakerscan scan http://host.docker.internal:3001 --budget-profile thorough --active-testing --confirm-active

Export the completed scan report as JSON from the scan detail page or the API, then analyze misses from a source checkout at the same tag:

git clone --branch v2.3.1 https://github.com/andriyze/shakerscan.git
cd shakerscan
python3 tests/benchmark/analyze_dast_benchmark.py --profile juice_shop \
  --result <scan_report.json> --mode candidate \
  --expected-json benchmarks/juice_shop/expected.json

The release-level answer key with per-class expectations is tests/fixtures/benchmarks/juice_shop.yaml. Only test systems you own or are authorized to assess.

Product implications

• The plan adopted a governing rule from an external audit: "No change merges into detection that does not move recall on the funded thorough authenticated benchmark, and no new subsystem ships until finding capability demonstrably improves."
• Scan is positioned as a discovery and baseline engine; deep, open-ended exploitation is Hunt's job. The companion article covers what Hunt added on the same target.
• A ShakerScan report separates observed risk from examination strength for the same reason this benchmark separates found from attempted: a clean-looking run is not evidence of safety unless the planned work actually ran.

References

• Benchmark fixtures README
• Juice Shop release fixture
• Miss-analysis script and summary builder
• Detector integrity gate
• Release 2.3.0 plan
• Benchmark integrity ledger
• Hunt architecture, current state
• Scan product page