An AI vendor security checklist verifies where data travels, how it may be reused, what actions the system can take, and who can stop it. Buyers should demand contract terms, access logs and repeatable failure tests for their specific use case before approving production access.
The core problem
A convincing AI demo can hide broad access, unclear data reuse and an inability to explain a wrong action. Buyers approve the answer they see while the deployed system gains access to email, files or business applications they have not tested.
The B3 read
Treat the model, retrieval store, agent tools, identity system and subcontractors as one purchase. A safe-sounding model policy cannot compensate for a connector that accepts excessive privileges or a server that fails to verify authorization.
Minimum defensible standard
Approve a named use case with bounded data and actions, a buyer-owned acceptance test, useful logs, a tested stop procedure and a named human accountable for failure. Keep consequential write actions disabled until those conditions are met.
02 / THE EVIDENCE
The Core Decision Framework
Data boundaries and training limits
What the vendor may claim
Your data stays private and is never used for training.
Evidence to demand
Map prompts, uploaded files, retrieved passages, outputs, embeddings, logs, support access and backup copies. Require contract terms for each purpose, region, recipient, retention period and deletion exception, including subprocessors.
Red flag
A no-training promise covers prompts but excludes telemetry, feedback, support uploads or a downstream model provider.
Agent permissions, tokens and stop controls
What the vendor may claim
The agent only does what the user asks.
Evidence to demand
List allowed tools and OAuth scopes. Use separate service identities, short-lived credentials where supported and approval before consequential actions. Test revocation against queued work and active sessions, with authorization enforced by the destination system.
Red flag
The approval dialog appears after execution, a shared token has broad access, or stopping the interface leaves background tasks running.
Wrong answers and operational failure
What the vendor may claim
Accuracy is excellent in our benchmark.
Evidence to demand
Use buyer-selected, representative tasks with known answers and unacceptable failure cases. Measure unsupported assertions, unauthorized actions and safe escalation separately. Require a human fallback and a process to find and correct affected outputs.
Red flag
One aggregate accuracy score hides serious errors, or the vendor cannot identify which model version produced an answer.
Governance, change and exit
What the vendor may claim
Our platform is compliant and enterprise-ready.
Evidence to demand
Record intended use, internal owner, assessment scope, model and connector versions, change notices and export formats. Review who can enable new integrations and what triggers reapproval.
Red flag
The seller substitutes a certification logo for your use-case assessment, or changes a model or data recipient without a review path.
NIST AI RMF and its Generative AI Profile are voluntary risk-management references; they are not product certifications. NIST says AI RMF 1.0 is being revised. OMB M-25-22 addresses covered federal AI acquisitions and emphasizes competition, performance and portability. Federal buyers must also review subsequent OMB guidance, agency rules and the actual solicitation; this checklist is not a complete federal acquisition specification.
Buyer questions answered
Does no training mean the vendor stores no data?
No. Training use and storage are different questions. A provider may still retain logs, abuse-monitoring records, support material or backups. Require separate terms for use, retention, access, deletion and each downstream processor.
What should an AI agent stop button do?
It should trigger a documented containment procedure for new jobs, queued work, sessions and connected-system permissions. Test the entire procedure. Some in-flight actions may already be committed, so record those limits and require preventive approval for irreversible actions.
Does OMB M-25-22 apply to every business?
No. It addresses covered U.S. federal agencies and covered AI acquisitions, with exclusions and scope conditions. Other buyers can borrow its questions about competition, performance and portability without presenting it as a universal legal requirement.
03 / THE WORKING CHECKLIST
Practical Evaluation Checklist
Use the questions in a review meeting. Check a box after the requirement has been reviewed, then capture the actual evidence and decision in the downloadable worksheet. Page checkmarks are temporary and are not submitted.
The B3 evidence scale
Record each material requirement as 0: claim only, 1: dated evidence reviewed, or 2: buyer-witnessed test passed. Record the owner, evidence location, test date and unresolved condition alongside the score. This is B3’s meeting tool, not an industry rating or certification.
A total must never hide a failed critical control. Choose approval blockers before testing; keep a blocked control open until it passes or an authorized buyer explicitly accepts the exception.
Phase 1: Pre-commitment / discovery
Phase 2: Technical & operational proof
Phase 3: Incident, failure & exit scenarios
04 / FROM REPORTING TO REQUIREMENTS
Daily Brief Intelligence Bridge
These links preserve the original edition and its reporting context. The control recommendations below are B3 analysis; the incidents do not establish that any particular vendor passes or fails this checklist.
The buyer-side read: The OpenAI report describes unauthorized access to hospital-bed statistics, not patient medical records. Use it to test server-side token validation and action logging; a model’s stated purpose is not an access-control boundary.
The buyer-side read: The email-to-agent story makes approval timing testable. Require proof that a forbidden action is blocked before it executes, including when the instruction arrives through retrieved content.
The buyer-side read: Anthropic’s self-replication work used a simplified research environment. Use it as a reason to test network and tool boundaries, not as evidence that an uncontrolled real-world outbreak occurred.
Inventory the AI vendors and connectors with access to sensitive information. Pick one proposed or active use case and document its allowed data, actions, owner and stop path.
Next 30 days
Run the buyer-selected evaluation and stop-control exercise in a bounded test environment. Review data terms and access scope with security, procurement and the business owner together.
Next 90 days
Make acceptance evidence, incident duties, change notice and exit rights part of the purchase or renewal. Schedule retesting when the model, connector, data class or business use changes.
Primary sources and scope
References checked October 17, 2026. The tests and decision rules in this guide are B3 recommendations unless a source is explicitly identified.
