MSP due diligence
Questions to Ask an MSP Before You Sign or Renew
A service-level promise is useful only when someone can show who acts, how quickly, and with which authority.
Open the checklistPillar · Vendor Evaluation
Vendor Evaluation
Read from the buyer’s chair, vendor evaluation is how you tell a real solution from a decorated demo and hold a vendor to what it promised. Now that every vendor claims an AI story, the questionnaire has to reach the model, the agent and the supply chain behind the product.
Reviewed October 5, 2026 · Standards in force
The buyer's problem
A traditional questionnaire asks about encryption, access control and SOC 2. It does not ask which model the product runs, where that model came from, whether an agent can call tools it was never approved to use, or whether anyone has tried to break it with adversarial prompts. Those are now the questions that predict an incident. This week’s attacks also showed that known, unpatched flaws stay in play for months, so a vendor’s patch speed and advisory honesty belong in the score.
How to evaluate a vendor honestly
- Where did the model come from? Ask for model provenance: the provider, version, training-data policy, fine-tuning sources and an AI bill of materials.
- Are secrets kept out of code and prompts? Ask for evidence of secret scanning in repositories, pipelines, logs and prompt history, and for proof that rotation works.
- Can the agent call a tool we did not approve? Require allowlisted tools, blocking and alerting on unauthorized tool calls, and logs you can export.
- Who has tried to break it? Ask for recent adversarial red-team results covering prompt injection, data exfiltration and jailbreaks, and how findings were fixed.
- How fast will they tell us, and fix it? Put breach notification, advisory publication and fixed-build timelines in the contract.
- Can they prove where the build came from? Ask for signed releases and build provenance, such as SLSA attestations verified with Sigstore, for the product and its AI components.
- What do they do with our conversations? Score retention, human review, training use, encryption and deletion of LLM conversation data, including logs held by sub-processors.
- How broad are their API permissions? Score every OAuth scope and API key the product requests against what the feature needs, and reject standing admin grants.
- Will they tell us what their AI did? Score whether the vendor can produce per-customer logs of its AI agents' actions, how fast, and whether it commits to notify you of AI incidents.
- How fast did they patch the last exploited flaw? Ask edge and gateway vendors for the dates of their last three exploited-vulnerability fixes and whether a first patch had to be replaced, as NetScaler SAML customers must do now.
Reference frameworks for a 2026 vendor review
- OWASP Top 10 for Agentic Applications 2026 Use it to scope red-team testing and tool-misuse controls, starting with ASI01 Agent Goal Hijack.
OWASP GenAI Security Project ↗ - Careful Adoption of Agentic AI Services Five Eyes guidance from May 1, 2026 to limit agent autonomy and access and keep oversight and monitoring. It is a ready-made checklist for vendor claims.
Crowell & Moring ↗ - Executive Order 14409 Signed June 2, 2026. Directs federal AI cybersecurity work, including a clearinghouse to find and fix software vulnerabilities at scale and a classified benchmark for frontier-model cyber capability.
The White House ↗ - ISO/IEC 42001 and SOC 2 Ask for both: ISO/IEC 42001 for AI management, SOC 2 Type II for operating controls. Neither replaces testing the product itself.
Aprio ↗ - CISA BOD 26-04 A useful benchmark for vendor patch commitments: three days for actively exploited, automatable flaws that give control of internet-facing systems.
Cybersecurity Dive ↗ - SLSA v1.0 (Supply-chain Levels for Software Artifacts) A framework for build provenance. Ask vendors which SLSA build level their releases meet and for the attestations.
OpenSSF SLSA ↗ - Sigstore Open tooling for signing and verifying software artifacts and their provenance, so you can check a release came from the vendor's build system.
Sigstore ↗
Third-party evaluation matrix: IT controls and AI supply chain
B3 scoring guide. Weight the AI rows higher when the product holds credentials or acts on your systems.
| Domain | What to ask | Evidence to request | Red flag |
|---|---|---|---|
| Model provenance | Which models, versions and providers power the product? | AI bill of materials, model cards, change log | “Proprietary” with no named base model or version history |
| Secret scanning | How are keys and tokens kept out of code, logs and prompts? | Scanner output, rotation runbook, last rotation date | Long-lived shared API keys |
| Tool-call control | Can agents call tools outside an approved list? | Allowlist configuration, blocked-call logs, alerting demo | Agents inherit the full permissions of the user |
| Adversarial red-teaming | Who tested the product, how recently, against what? | Red-team summary mapped to OWASP agentic risks, fix dates | Only the model provider’s generic safety testing |
| Traditional controls | Do SOC 2, ISO 27001, MFA, encryption and logging cover AI features? | SOC 2 Type II scope, ISO certificates, pen-test letter | AI features excluded from the audit scope |
| Patch and disclosure | How fast do advisories, workarounds and fixes ship? | Last three advisories with timestamps | Exploited flaws disclosed only after customers find them |
| Third-party AI agents | What can the vendor's agent reach in our environment, and who controls its control plane? | Agent permission map, control-plane version, identity-provider configuration | Agents share one service account or run without an identity provider |
| Supply-chain dependency scanning | How are open-source and model dependencies scanned before release? | SBOM and AI-BOM per release, dependency-scan results, pinned versions | No SBOM, or dependencies pulled at runtime |
| Cryptographic origin verification | Are releases and model artifacts signed, with verifiable provenance? | Sigstore signatures, SLSA provenance attestations, key-management policy | Unsigned updates or checksums published on the same server |
| API permission scoring | Which OAuth scopes and API keys does the product request, and why? | Scope list mapped to features, automated over-permission report | Standing tenant-wide admin consent |
| LLM conversation data handling | How are prompts, outputs and chat logs retained, reviewed and deleted? | Retention schedule, human-review policy, deletion certificate | Indefinite retention or reviewer access by default |
| Sandbox isolation | How is AI-generated or untrusted code isolated, and how is the hypervisor patched? | Isolation design, hypervisor patch SLA, egress and credential limits per sandbox | Shared containers for untrusted code or no stated patch window |
| AI incident disclosure | Can you produce logs of your agents' actions on our data, and when will you notify us? | Sample action log, notification clause, retention period | No per-customer agent logs or no AI incident clause |
| Edge patch history | When did you fix your last three exploited flaws, and did any fix need a second upgrade? | Advisory dates, time to fixed build, re-patch history | Repeated re-patches or no published advisories |
| Agent prompt-injection resistance | Can content from email or documents make your agent run code or send data? | Test results, pre-execution blocking design, connected-account scoping | Warnings only after execution or no testing |
Vendor security questionnaire: IT controls and AI supply chain
- Identity and access. SSO and MFA for all users and administrators, scoped service and agent identities, and quarterly access reviews.
- Data protection. Encryption in transit and at rest, customer-managed keys, data residency, retention limits on prompts and outputs.
- Vulnerability management. Patch timelines by severity and exploitation status, an SBOM for each release, a disclosure policy and a published advisory history.
- AI supply chain. Model provenance, third-party model and plugin inventory, dataset sources, and how model updates are validated before release.
- Agent controls. Tool allowlists, human approval for high-impact actions, an emergency stop, and exportable action logs.
- Testing. Annual penetration tests plus adversarial AI red-teaming, with findings and fix dates shared under NDA.
- Incident duties. Notification within a contracted number of hours, cooperation with your investigation, and log retention long enough to support it.
- LLM conversation data (scored 0–3). 0: retained indefinitely or used for training. 1: retained with opt-out. 2: limited retention, no training, no human review by default. 3: zero retention for regulated data, verified by audit.
- API access governance (scored 0–3). 0: standing admin grants. 1: broad scopes with documentation. 2: least-privilege scopes mapped to features. 3: least-privilege, time-bound tokens with automated over-permission reporting.
- Origin verification. Signed releases and model artifacts, SLSA provenance attestations and a published verification procedure.
- AI incident evidence (scored 0–3). 0: no agent action logs. 1: logs on request with no SLA. 2: per-customer logs within 72 hours. 3: customer-accessible action logs and a contractual AI incident notification clause.
- Prompt-injection resistance (scored 0–3). 0: no testing. 1: post-execution warnings only. 2: pre-execution blocking for known patterns. 3: independent testing with obfuscated payloads, pre-execution policy enforcement and scoped connected accounts.
This is one of the six domains in the readiness assessment. See where your organization actually stands.
Take the readiness assessmentScore vendors on their last exploited flaw, not their brochure.
Citrix customers who patched NetScaler for September's zero-days must upgrade again for CVE-2026-88779 if they use SAML. Salt Labs showed one email could hijack the Manus agent before its warning appeared. Both are evaluation questions you can ask now.
Test now: Add two lines to every edge and AI vendor review: your last three exploited-flaw fix dates, and your prompt-injection test results.
Read the October 5 brief →
Watch
Public talks, with a buyer-side read.
Curated public-sector and regulated-technology speeches, each with buyer-side commentary.
Loading…