Comparison
Promptfoo is strong for evals. ShakerScan is built for deploy gates.
Promptfoo helps teams test and evaluate LLM behavior. ShakerScan focuses on signed allow, block, and needs_approval decisions that CI/CD can verify.
Sample gate decision
Deploy gate needs human approval
The release has AI risk evidence that does not hard-block. CI waits for an eligible scoped approval-token workflow before deployment.
Use Promptfoo when your main need is local or CI eval authoring and red-team test iteration.
Use ShakerScan when engineering needs signed evidence, release policy, approval tokens, and deploy verification.
Use both when prompt evals feed a broader release-control workflow.
Why ShakerScan
The output is a release control, not just a report.
ShakerScan is built around release evidence: a tested target, a policy result, a verifier command, and an approval path when risk needs human review.
Signed evidence
Evidence hashes and AI Gate attestations bind the decision to the target, environment, policy, probe pack, and release scope when signing is configured.
CI-verifiable decision
GitHub Actions or the shakerscan CLI can verify that the decision matches the expected repo, commit, branch, environment, target, policy, and evidence hash.
Approval workflow
When an eligible workflow is approved, scoped approval tokens record the reason, audience, expiry, and decision path instead of bypassing the gate silently.
Checklist
Choose by workflow
Choose Promptfoo for eval authoring, prompt tests, and red-team iteration.
Choose ShakerScan for signed deploy decisions and CI-verifiable release evidence.
Avoid migrating until you know whether the buyer wants testing depth, release control, or both.
Limitations
What this page does not claim
This comparison is based on public product positioning and should be refreshed because competitor products change.
ShakerScan does not replace human security review, threat modeling, or a scoped penetration test.
AI Gate decisions depend on the configured target, probe pack, policy, scan profile, and available evidence.
Production targets require authorization, safe scope, rate limits, and operational approval.
FAQ
Is ShakerScan an AI pentesting replacement?
No. ShakerScan is a verifiable security gate for release workflows. It complements deeper manual testing by producing repeatable runtime evidence and CI-verifiable allow, block, or needs_approval decisions.
Can ShakerScan scan any target?
No. Targets must be owned by the customer or explicitly authorized. Production scans should use safe profiles, rate limits, and defined scope.