Governing the AI-era attack surface

Executive summary

AI risk is frequently reduced to model behavior. That is too narrow for deployed systems. A production AI capability is an assembly of user identities, service accounts, prompts, retrieval data, model endpoints, software libraries, plugins, tools, memory, logs, approval logic, and external destinations. The attack surface exists in the relationships between those parts.

NIST AI 600-1 provides a cross-sector profile for generative-AI risk. NIST's Cyber AI Profile is currently preliminary and is organized around securing AI components, using AI for defense, and defending against AI-enabled attacks. This paper translates those directions into a system architecture centered on six controls:

1. Complete AI use-case and dependency inventory.
2. Separate identities for reading, proposing, and acting.
3. Untrusted-content treatment for prompts and retrieval.
4. Schema and destination boundaries around every tool.
5. Evidence and approval outside the model.
6. Revocation, reconstruction, and containment designed in advance.

1. Define the system, not only the model

An AI system record should include:

• Business use case, owner, users, affected decisions, and failure consequence.
• Model provider, model and version policy, hosting mode, region, contractual terms, retention, and training-use settings.
• Prompt templates, system instructions, retrieval sources, vector stores, memory, fine-tuning data, evaluations, and output destinations.
• User identities, application identities, API keys, roles, delegated access, tenant boundaries, and emergency access.
• Orchestration frameworks, packages, containers, plugins, tool servers, browser surfaces, and build pipeline.
• Every tool's permitted operations, destination allowlist, transaction limit, approval condition, and audit output.
• Monitoring, incident contacts, stop mechanism, credential-revocation path, and evidence-retention rule.

Inventory quality should be measured. Unknown owner, unknown provider retention, unknown tool permission, or unreviewed data source is a control gap, not a neutral field.

2. Threat model

2.1 Instruction injection

Untrusted text can attempt to override system intent, extract context, redirect tools, or suppress safeguards. The instruction may arrive through a user prompt, web page, document, source repository, ticket, email, image-derived text, API output, or persistent memory.

2.2 Excessive agency

A broadly privileged tool turns an incorrect or manipulated model output into a side effect. High-risk effects include sending externally, changing access, deleting state, publishing, purchasing, moving funds, executing code, or touching customer systems.

2.3 Data disclosure and cross-boundary mixing

Prompts, retrieval, logs, evaluation traces, caches, and vendor support channels can move data outside its intended tenant or purpose. Output can also reveal system prompts, secrets, personal data, or proprietary context.

2.4 Identity confusion

An agent can act under the wrong customer, subscription, operator, or service identity. Shared sessions and ambient credentials make it difficult to attribute or revoke the action.

2.5 Software and model supply chain

Libraries, containers, model files, plugins, tools, and update channels can be malicious, compromised, stale, or incompatible. A model name without a version policy reduces reproducibility.

2.6 Unreliable inference

Confident output can be unsupported, incomplete, stale, or sensitive to small input changes. For security work, an invented asset, vulnerability, scope rule, or remediation state can be operationally dangerous.

3. Reference control architecture

3.1 Provider-neutral capability boundary

The agent should call a stable capability interface rather than receive a shell and a repository path. Each capability declares input schema, output schema, maturity, side effects, provenance, and execution boundary. Provider updates can change the conversational shell without bypassing the control plane.

3.2 Identity-bound session

Every run binds operator, provider account, customer or project, task, and authorization context. Credentials are selected from that identity rather than inherited from an ambient environment. Mismatch fails before a request leaves the host.

3.3 Read, propose, act separation

Read-only discovery produces evidence. A proposal produces a reviewable plan or draft. State-changing action requires a separate capability and gate. The most privileged step should not share credentials or automatic retry behavior with the least privileged step.

3.4 Untrusted content boundary

Retrieved content cannot grant itself authority. The orchestrator strips or isolates instructions from data, labels source and trust, constrains context, and validates tool arguments against external policy. The model never decides whether the content is allowed to expand scope.

3.5 Network and destination control

Network tools enforce exact-origin or destination allowlists, resolve hostnames under policy, bind resolved addresses to pacing, revalidate redirects, limit response bodies, reject credentials in URLs, and audit direct egress paths.

3.6 Evidence store

Results enter a normalized store with asset identity, source, time, model or tool version, evidence reference, and review status. Model summaries can help navigate evidence but cannot replace it.

4. Write gates for tool-enabled agents

A write gate should verify facts outside the model:

1. The caller identity and active project match.
2. The requested target is in the current written scope.
3. The capability is allowed for the role and current task.
4. Required deduplication or conflict checks passed.
5. The payload matches an exact schema and size ceiling.
6. Secrets are absent from fields that will be logged or returned.
7. The destination, recipient, or resource is fixed or allowlisted.
8. The operator confirmation covers this action, not a broad future class.
9. The result can be reconciled through a receipt or state readback.

Retry logic must understand side effects. A timeout after submission is ambiguous; repeating the action can create duplicate messages, tickets, payments, changes, or reports. Idempotency keys, receipts, and read-after-write checks are control requirements.

5. Provenance and evidence

A useful AI-assisted security record identifies:

• Provider, model, and relevant version or alias at execution time.
• Capability code identity and manifest.
• Prompt or instruction template version.
• Source records and retrieval references used.
• Tool arguments after validation and redaction.
• Authorization, approval, and policy decision.
• Tool result, receipt, and resulting state where applicable.
• Analyst review and corrections.

Logs should be useful without becoming a secret archive. Store digests or references for sensitive evidence, redact tokens and credentials before serialization, and apply bounded retention by data class.

6. Evaluation program

Evaluation should test the assembled system, not only the base model:

• Direct and indirect prompt injection from every retrieval source.
• Cross-tenant and wrong-project identity attempts.
• Tool-argument smuggling, duplicate keys, oversized values, Unicode separators, and schema confusion.
• Redirect, DNS rebinding, private-address, and response-size behavior for network tools.
• Ambiguous write results, retries, stale receipts, and duplicate action prevention.
• Secret leakage through output, error, trace, summary, and report fields.
• Revocation during a running job and cleanup of descendants.
• Unsupported claims, missing sources, stale context, and disagreement between models or tools.

Each control needs a positive test and a one-factor negative or mutant that proves the test fails when the control is removed. Passing a happy path is not evidence that the boundary is load-bearing.

7. Incident response for AI systems

1. Stop: disable the agent route, revoke tool credentials, stop queued work, and isolate affected connectors.
2. Preserve: retain bounded prompts, sources, tool calls, approvals, receipts, and state changes without spreading secrets.
3. Scope: identify users, projects, models, tools, destinations, and data classes involved.
4. Reconcile: determine which proposed actions became real side effects and which remained drafts.
5. Correct: fix policy, identity, parser, prompt, tool, or workflow control rather than only changing wording.
6. Verify: rerun the failing control and mutants, review adjacent paths, rotate affected secrets, and monitor recurrence.

8. Governance metrics

• AI use cases with named owner, approved provider, data classification, and dependency inventory.
• Tool-enabled workflows with separate read and write identities.
• High-impact actions protected by deterministic gates and readback.
• Retrieval sources covered by injection tests.
• Capabilities with explicit input, output, side effect, provenance, and maturity contracts.
• Percentage of AI-supported security claims linked to independent evidence.
• Time to revoke all credentials for one agent or project.
• Time to reconstruct state-changing actions after an incident.
• Evaluation coverage by threat class and last successful negative control.

9. Argus implementation direction

Argus uses a provider-neutral CLI and MCP layer, capability manifests, project isolation, request safety, private draft defaults, outbound write gates, source-bound evidence, deterministic release checks, and agent skills that teach discovery before execution. These are foundations, not a claim that every AI risk is solved.

The next maturity step is to keep provider wrappers thin while concentrating policy, identity, side effects, receipts, and evidence in the shared Argus control plane. A provider update should improve the interface without changing who can do what to which system.

Conclusion

AI security becomes manageable when the model is placed inside an ordinary security architecture: inventory, identity, least privilege, untrusted input handling, bounded network access, gated writes, evidence, testing, and response. The novel interface does not repeal those principles. It makes their enforcement more urgent.