The inventory nobody has
Before any control can apply, the organization needs an enumeration of deployed models, their versions, the data they were trained or fine-tuned on, the prompts and system instructions in production, the retrieval corpora they read, the tools and APIs they may call, and the identity each agent uses.
That last item is the one most frequently missing and most frequently exploited. An agent with a broad service credential is an unaudited privileged user with a natural-language interface.
- Model registry with version, owner, purpose and approval state
- Prompt and system-instruction versioning under change control
- Retrieval corpus classification, including data residency
- Tool permission scope per agent, with the identity it executes as
Evaluations are the control, not the demo
An evaluation harness runs a versioned suite against every model change and reports pass rates by category: prompt injection resistance, data leakage, unsafe content, tool misuse, jailbreak robustness, and task-level accuracy on the business function the model actually performs.
Two engineering requirements make evaluations trustworthy. The suite must include adversarial cases derived from your own incidents and red-team findings, not only public benchmarks. And results must gate deployment, with thresholds agreed by the business owner in advance.
- Versioned suites with per-category thresholds that gate release
- Adversarial cases sourced from internal red-team and incident history
- Regression tracking across model, prompt and retrieval changes
- Human review sampling on production traffic, with documented rubric
Runtime detection for AI-specific abuse
Static evaluation cannot cover live behaviour. Runtime detection must observe prompt and response pairs for injection patterns, sensitive data egress, anomalous tool invocation sequences, and volume or entropy shifts that indicate systematic extraction of the model or its corpus.
These detections belong in the same case management as endpoint and cloud incidents. An agent exfiltrating retrieval content is a data incident, and it needs the same evidence handling and containment discipline.
Mapping to frameworks that auditors accept
The control set maps to the NIST AI Risk Management Framework functions and to ISO/IEC 42001 clauses, with EU AI Act obligations attached where the use case is in scope. One control model, multiple mappings, one evidence store — the same principle applied to classic compliance.
- NIST AI RMF: govern, map, measure and manage evidenced per system
- ISO/IEC 42001: management system artifacts generated from operational data
- EU AI Act: risk classification recorded per use case with review dates
Acceptance criteria for an AI Sec program
Accept the program on inventory completeness, evaluation coverage and detection efficacy, measured on your own systems.
- 100% of production models and agents in inventory with a named owner
- Evaluation suite executed on every model, prompt or corpus change, with recorded gate decision
- Runtime detection coverage for injection, exfiltration and tool misuse classes
- Shadow AI discovery reconciled monthly against sanctioned inventory