Use Case
Test chatbot behavior before it reaches customers.
Customer-facing chatbots need runtime testing because the risk is in how they respond with real prompts, tools, context, and session state.
Sample gate decision
Chatbot followed untrusted instructions
A prompt-injection probe caused the chatbot to ignore its system policy. The finding is tied to transcript evidence and blocks the release.
Register a chat API or widget target.
Run smoke probes first, then expand to prompt injection and data exfiltration coverage.
Review transcripts and normalized findings.
Require a verified allow decision before release.
Why ShakerScan
The output is a release control, not just a report.
ShakerScan is built around release evidence: a tested target, a policy result, a verifier command, and an approval path when risk needs human review.
Signed evidence
Evidence hashes and AI Gate attestations bind the decision to the target, environment, policy, probe pack, and release scope when signing is configured.
CI-verifiable decision
GitHub Actions or the shakerscan CLI can verify that the decision matches the expected repo, commit, branch, environment, target, policy, and evidence hash.
Approval workflow
When an eligible workflow is approved, scoped approval tokens record the reason, audience, expiry, and decision path instead of bypassing the gate silently.
Checklist
Chatbot gate checklist
Test both API and widget surfaces when both exist.
Check prompt leakage, data exfiltration, output handling, and session behavior.
Avoid using production secrets in test credentials.
Retest after prompt, tool, model, or retrieval changes.
Limitations
What this page does not claim
ShakerScan does not replace human security review, threat modeling, or a scoped penetration test.
AI Gate decisions depend on the configured target, probe pack, policy, scan profile, and available evidence.
Production targets require authorization, safe scope, rate limits, and operational approval.
FAQ
Is ShakerScan an AI pentesting replacement?
No. ShakerScan is a verifiable security gate for release workflows. It complements deeper manual testing by producing repeatable runtime evidence and CI-verifiable allow, block, or needs_approval decisions.
Can ShakerScan scan any target?
No. Targets must be owned by the customer or explicitly authorized. Production scans should use safe profiles, rate limits, and defined scope.