How to evaluate an AI vendor for a regulated environment.
Most AI evaluations test the demo. Very few test the things that decide whether a system can actually be authorized, operated, and eventually replaced. This guide is the list of questions we would want asked of us.
Written for program offices, Information System Security Officers (ISSOs), contracting officers, and AI governance leads. Vendor-neutral. Use it on us too. To save this guide as a PDF, use your browser print command (Ctrl+P on Windows, Cmd+P on macOS).
“Air-gapped” is a claim, not a feature.
Many vendors describe a system as deployable on-premises when what they mean is that a thin client runs locally while inference, telemetry, or licensing still calls out to a commercial cloud. Ask questions that make the distinction unavoidable.
Ask: what breaks with no route to the internet?
Do not stop at “can it run offline.” Ask what specific functionality degrades. Licensing checks, model updates, telemetry, error reporting, and authentication are the usual dependencies. A vendor who has genuinely done this will answer immediately and specifically.
Ask: how does an update reach a disconnected enclave?
There should be a described, repeatable transfer process with signed bundles and integrity verification on the receiving side, not “we would work that out during implementation.”
Ask: what is the deployment artifact?
A container image with a Helm chart or equivalent is a real answer. “We will provision it for you” usually means a managed service that cannot cross into your enclave.
Ask: has this been run at your classification level?
If the answer is no, that is not automatically disqualifying, but it should change your pilot scope and your schedule assumptions. Be suspicious of a yes with no detail attached.
Ask who does the paperwork, and when it gets done.
The gap between “our product is secure” and “here is the evidence package your Authorizing Official (AO) needs” is usually measured in months of your staff's time. Find out early whose months those are.
- What artifacts does the product generate on its own, without our staff assembling them?
- Do you produce an SBOM? In which format, and is it generated per build or per release?
- Can you show a control mapping to our baseline, or only to a generic one?
- Are findings delivered in a format our POA&M tracker can import?
- Is the evidence cryptographically bound to the artifact we actually deploy?
- Can evidence be re-verified independently, by us, without your involvement?
The vendor hands you a sample evidence package before you ask twice. It contains a manifest, content hashes, an SBOM in a standard format, normalized findings with ownership, and a mapping to named controls. You can verify the hashes yourself.
The vendor is also clear about what the evidence does not do: notably, that continuous evidence generation supports ongoing authorization but does not replace an initial ATO. A vendor claiming their tool “gets you an ATO” is overselling.
Where your data goes is a contract question, not a technical one.
Get “we do not train on your data” in writing
Marketing pages say it; contracts often do not. Ask for the specific clause. Ask separately about telemetry, logs, prompts, and error reports. These are frequently carved out of a no-training commitment.
Ask for an AI Bill of Materials
You should be able to see what models are in the system, where they came from, what they were trained on at a provenance level, and what changes between versions. If the vendor cannot inventory their own AI components, they cannot help you govern yours.
Ask for model cards and data cards
These should describe intended use, known limitations, evaluation results, and failure modes. A model card that lists no limitations is a marketing document.
Ask what adversarial testing has been done
Prompt injection, data poisoning, and model extraction are the baseline. Ask for the assessment report, the test cases, and, most tellingly, what the testing found and what was done about it.
Confirm the governance framework being claimed is current
Federal AI policy has moved repeatedly. A vendor citing a superseded memo may simply have stale collateral, but check that their compliance work tracks the policy actually in force for your system type.
Plan the exit during the evaluation, not during the dispute.
Ask for the exit path in writing
A written description of how you extract your data, configurations, and generated artifacts, in open formats, without vendor assistance.
Test the export during the pilot
Export everything once while you still have leverage. Confirm you can read it with tools you already own.
Check what the artifacts depend on
Evidence packages and policy configurations only the vendor's product can read are lock-in, whatever the contract calls them.
Signals worth slowing down for.
- Accuracy or time-savings percentages with no methodology behind them
- A compliance certification implied rather than stated, or listed without scope
- “FedRAMP” used loosely; ask for the impact level and the authorization status
- A claim that the tool produces or guarantees an ATO
- Customer logos the vendor will not let you contact
- Blanket “patent pending” across an entire product line
- Demo environments the vendor controls and will not let you drive
- Reluctance to support government-led independent testing
- No written limitations section anywhere in the documentation
- Security documentation that arrives only after contract award
- An architecture diagram that stops at the trust boundary
- Pricing that changes materially once air-gapped deployment is mentioned
A simple evaluation matrix.
Score each vendor 0 to 3 per criterion: 0 no answer, 1 claim only, 2 documented, 3 demonstrated in your environment. Anything scoring 0 or 1 on the first four lines is not ready for a regulated deployment regardless of how good the demo was.
Disconnected operation
Runs with no route to the internet; update path is defined, signed, and verifiable on receipt.
Weight: criticalEvidence generation
Produces SBOM, findings, control mappings, and a verifiable manifest with no work by your staff.
Weight: criticalData rights
Contractual no-retention and no-training terms covering prompts, logs, and telemetry.
Weight: criticalDocumented exit path
Open-format export of data, configuration, and artifacts, testable during the pilot.
Weight: criticalAI transparency artifacts
AI-BOM, model cards, data cards, and adversarial testing results with findings disclosed.
Weight: highInteroperability
Integrates with existing SIEM, SOAR, ITSM, and DevSecOps tooling rather than replacing it.
Weight: highIndependent testability
Supports government-led evaluation, red teaming, and source review under agreement.
Weight: highStated limitations
Documentation says plainly what the product does not do and where human judgment is required.
Weight: moderateRun this list against us.
Send us the criteria that matter most for your program and we will answer them in writing, including the ones where the honest answer is “not yet.”