Before releasing Applied AI into production, link scope, data, authorization, the audit trail, and recovery to owners and evidence. Check where RBAC and ABAC limit actions and how to reconstruct the request, authorization, attempt, and confirmation. The decision needs to distinguish declared controls from tested controls, and stopping actions from recovering from effects that have already occurred.
What should the unit of assessment be?
Choose a business operation and identify the agents, vendors, models, tools, channels, and teams involved. Record its purpose, the data used, and the possible actions. An initial inventory may be partial; keep the parts that are still unknown and who will investigate them visible, without presenting the list as a complete discovery of the company.
Assess the consequence of the action in context. Preparing a draft, looking up a document, and changing a record call for different questions. Define who can accept the result and who can authorize changes in scope. Business, security, and technology need to share this description to prevent each area from assessing a different version of the same operation.
How do you turn policy into a concrete check?
Write the rule in terms the team can test: which identity can execute which operation, on which resources, and under what conditions? Then identify where the check takes place. A rule may depend on authorization in the tool, parameter validation, human review, or a restriction in the target system. Note the paths that do not pass through this point.
RBAC links roles to allowed operations. ABAC considers attributes of the resource, action, and context, such as purpose or an approved limit. Enforce authorization at the point that carries out the action: tool, integration, or target system. A model refusal does not block a credential that allows direct access. Also check administrative routines, alternative calls, and reprocessing that may bypass this point.
Link approval to the parameters that will be executed and check its validity at the time of the action. A change in recipient, scope, or amount requires new authorization. Test an allowed action, a role without permission, a condition outside the limit, expired approval, and an attempt through the alternative path. Record the effect observed in the target system.
Distinguish declared, configured, and verified controls. Documentation describes intent; a configuration shows part of the implementation; a representative test demonstrates behavior under specific conditions. None of these records alone proves universal protection. Evidence needs to indicate scope, version, and verification conditions without exposing personal data, secrets, or unnecessary content.
How do you assess data, tools, and external content?
Trace data flows through storage, search, models, tools, and support. The fact that a document remains in the company’s environment does not indicate which excerpts will be sent to another service. Record the destination, purpose, access, and retention conditions to be confirmed. The infrastructure option should be assessed together with these paths.
Consider that a message, a document, or a tool response may contain instructions inappropriate to the operation’s objective. The integration needs to keep authority in the rules and permissions defined by the application, instead of granting it to the received content. Include attempts to induce actions outside the scope in testing and observe the effects on the system, not just the agent’s text response.
What does the audit trail need to reconstruct?
Link four events: the request received, the authorization granted or denied, the attempt submitted, and the confirmation from the target system. Record the time, decision, policy version, and result with the minimum information necessary. A model response or a submitted call does not prove completion. An absent response should remain an unknown result until the target system is queried and reconciliation takes place.
Define retention, access, protection against changes, and an investigation procedure. Keep personal data, secrets, prompts, tokens, and sensitive values out of logs. Use protected references to correlate authorized evidence and access recovery material in separate storage, with its own permissions and retention; do not copy the previous content of the record into the trail. The reference also needs protection and must not expose sensitive identifiers.
Test whether an authorized person can reconstruct a blocked attempt and a confirmed execution, and whether unauthorized access to the evidence is denied. Record volume does not demonstrate coverage: the team needs to locate missing links and keep explicit what it still cannot verify.
How do you stop, reverse, or compensate for an action?
Agree on who receives an incident, who can suspend an integration, and how work continues with the team. Stopping blocks new actions; it does not undo a change that has already been confirmed. Before retrying or recovering, check the target system and the actions in progress to avoid duplicating effects.
Reversal restores a previous state when the system allows it; compensation addresses the consequence through another action, such as an authorized refund. An external transmission may not be recoverable. For each effect, define an owner, condition, limit, and evidence of completion. Material needed for recovery should be accessed through a protected reference, not by copying sensitive values into logs.
In the case of missing confirmation or partial reversal, keep the item pending and route it to the responsible team. If there is no viable recovery, reduce autonomy and require human approval before the action with greater impact. Practice the stop and recovery procedure before expanding the scope.
Which evidence allows the operation to be released or resumed?
Review the matrix with business, security, and technology. Each control needs an enforcement point, evidence to check, an owner, and an explicit outstanding item. Record the decision on the scope, the accepted limitations, and who authorizes release. Do not treat a completed row as an already proven control.
Review the design when the model, source, tool, permission, vendor, or purpose changes. The review should be proportional to the impact of the change and record what was reassessed. The NIST AI RMF offers a voluntary reference for organizing this work; following a roadmap is not equivalent to obtaining certification and does not prove a deployment’s compliance.
Illustrative example · no customer data
A reimbursement that requires a person’s decision
Fictional example: the agent organizes supporting documents and identifies a request outside the expected condition. The process allows preparation of the case file but reserves approval for a responsible person. The control should operate in the integration that would carry out the payment, as well as guide the agent.
Verification needs to show that the attempt without approval was blocked, that the team received the outstanding item, and that a later decision corresponds to the same parameters. Complete documentation remains a different state from an authorized or completed payment.
If the payment response does not arrive, the team queries the target system before trying again. A stop prevents new payments; any refund requires another procedure and authorization. Supporting documents and recovery material stay in a protected repository, referenced by authorized evidence, without copying sensitive values into logs. The matrix shows what to check in this scenario; it does not report tests performed at a customer.
| Control | Enforcement point | Evidence to check | Owner | Outstanding item before release |
|---|---|---|---|---|
| Restrict scope and data | Authorized sources and outbound flows to tools/models | Destination map and a test of a blocked outbound flow outside the scope. | Business and data owner | Confirm sources, destinations, and retention conditions. |
| Enforce RBAC/ABAC and approval | Payment integration and target system | Attempt without an allowed role or attribute blocked; approval linked to the executed parameters. | Security and integration owner | Test direct credentials, expired approval, and a change after approval. |
| Reconstruct the request, authorization, attempt, and confirmation | Protected integration trail | Correlated events, policy decision, and confirmation or an unknown state; no sensitive content. | Operations and security | Validate access, retention, and reconstruction of a blocked attempt and a confirmed execution. |
| Stop new actions | Integration and payment queue | Test of suspension and a query of actions that were already in progress. | On-call operations owner | Define stop authority and continuity by the team. |
| Reverse or compensate for effects | Financial system and protected recovery repository | Query before retrying; protected reference and confirmation of an authorized refund, when possible. | Finance and system owner | Practice handling an absent response and partial compensation; record effects with no reversal. |
| Authorize release and resumption | Version and scope review | Recorded decision with test results, limits, and outstanding items. | Business, security, and technology | Resolve blockers and define changes that require a new assessment. |
Use one row per control in your operation. Indicate where it is enforced, which evidence will be examined, who is responsible, and what is missing. In this example, all evidence consists of proposed checks; release depends on the results and the decision of those responsible.
When the approach needs to change
If there is insufficient evidence, record the control as unverified and define the next check. When a credential cannot be limited to the required scope, reduce the available actions or keep the work in a preparation stage with review. Restrictions and decisions need to be confirmed in the specific deployment.
A point to watch: Considering a written policy or a correct response to be sufficient evidence of protection.
The worksheet is ready to move forward when…
- Scope, data, and responsibilities are identified.
- Each control has an enforcement point, evidence, an owner, and a recorded outstanding item.
- The request, authorization, attempt, and confirmation can be reconstructed with restricted access and minimized content.
- Stopping, reversal, or compensation have explicit procedures and limits.
- Release and resumption depend on evidence and defined owners.
Bring the matrix to the team and select an action for verification. The Starya assessment helps organize the questions; it does not inspect systems or issue certification.
Your worksheet
Complete it with your team.
Record what you know and what still needs confirmation. Answers stay in memory and are not submitted.
References for further reading
- OWASP — Agentic AI: Threats and Mitigations ↗
A reference for modeling threats to agent systems and discussing mitigations. The roadmap and scenarios below are Starya working proposals, not a full reproduction of the document.
- NIST AI RMF Playbook ↗
A voluntary reference for organizing governance, understanding context, and measuring and managing AI risks. It is not a Starya certification and does not replace assessment of the specific case.