Improving an operation requires choosing a problem, understanding its causes, and monitoring a change. A short, consistent routine helps the team learn from use without turning every occurrence into a complete redesign.
Define the scope before comparing
Choose a journey and an observable period. Record who fell within that scope, which channels were active, and whether there were significant changes in audience, volume, or rules. Compare equivalent situations: a campaign involving new people and recurring service may have different results without the agent having deteriorated.
Choose a problem the team recognizes in concrete instances. “We need to improve quality” is broad; “interactions reach the team without the reason for contact” allows investigation. Separate the number of occurrences from their proportion within the scope. Without knowing the total volume or the conditions during the period, it is difficult to interpret whether the problem became more frequent.
Investigate before changing behavior
Gather enough examples to recognize the pattern, retaining only the information needed for analysis. Identify the request, response, tool invoked, system state, and handoff. Classify what happened: inappropriate interpretation, outdated information, integration error, ambiguous rule, or incomplete handoff to a person.
Do not force a single cause when the examples point to different problems. Write down what has been observed and what is still a hypothesis. If the agent asked for information already provided, for example, there may be a failure in memory, the summary, or the way the data arrives. Changing the prompt without locating that difference may hide the problem for a short time.
Prepare a change that can be evaluated
Write the hypothesis as “by changing this point, we expect to observe this effect.” Define who will make the change, who will check it, and which situations need to keep working. An improvement may involve the agent's instruction, the integration, the source, or the human routine. The responsible person should match where the cause was found.
Record the version and date of the change so you can distinguish the periods. Avoid changing several independent points at once if you need to determine the effect of each one. Prepare verification examples, including situations in which the flow should hand off or stop. Agree on how to return to the previous behavior if an unintended effect appears.
Check the effect and retain what you learned
After the change, observe the same problem in the same type of situation. Also ask the team whether the deliverable has become usable. Fewer handoffs may seem positive and still hide interactions that should reach a person. For this reason, monitor the desired outcome and a signal that quality or continuity has not deteriorated.
End the round with a recorded decision: keep, adjust, undo, or observe for longer. A lack of sufficient data is also a useful result when accompanied by the next observation scope. Document the cause found and the change made so another responsible person can understand the work without reconstructing the entire conversation.
Illustrative example · no customer data
The team has to ask everything again
Illustrative example: sales interactions are handed off without the terms the person wanted to negotiate. Reading the instances shows that the information was in the conversation but not in the summary delivered to the salesperson. The hypothesis is to include the request, requested terms, and pending issue in the summary.
Before releasing the change, the team needs to check whether the salesperson receives the necessary context and whether the summary accurately preserves the request. Complete but incorrect text is still an inadequate handoff. The table records a fictional initial observation; the new version has not yet been evaluated.
| Field | Completed fictional record |
|---|---|
| Unit of analysis | One unique handoff to the sales queue. Attempts with the same identifier count once. |
| Window and scope | Observation week 1: five fictional business days. All 12 handoffs to this queue during the period were reviewed; other queues were excluded. |
| Version observed | Summary v1. The proposed v2 is still being prepared and has no usage results. |
| Classification criterion | The request, requested terms, and pending issue are present and faithful to the original conversation. A missing field or contradictory information makes the context unusable. |
| Numerator | 8 handoffs with usable context, according to the agreed criterion. |
| Denominator | 12 eligible and reviewed handoffs: 8 usable, 3 with missing terms, and 1 with incorrect terms. |
| Observed measure | 8 of 12 usable handoffs in version v1. No measure of version v2 was collected. |
| Responsible people | The sales queue lead checks the classification. Integration engineering prepares the summary adjustment. |
| Criterion for the next test | Check the fields against the source, including missing information, contradictions, and duplicates. Correct any discrepancies before releasing the monitored scope. |
| Recorded decision | Keep the current scope and test the adjustment. Do not expand or declare improvement based on this initial observation; after the test, define a comparable window. |
The counts describe only this fictional scope. There is no before/after comparison, savings estimate, or evidence of gains. If records are missing to classify an occurrence, record the number and the gap instead of silently removing it from the count.
When the approach needs to change
If the integration is failing, prioritize the reliability of the path before redesigning the conversation. If volume is low, review individual instances and record that limitation. If the business process has changed, create a new comparison baseline. Do not automatically attribute every variation to the agent.
A point to watch: Changing the prompt before checking whether the cause lies in the process, the source, or the integration.
The worksheet is ready to move forward when…
- The problem, scope, and hypothesis are separated from general impressions.
- The change, responsible person, and method for reversing it have been agreed.
- The review produced a decision and the next point to observe.
Bring an instance, the hypothesis, and the chosen indicator to the improvement discussion. This context helps distinguish engineering work, operational adjustments, and the need for new data.
Your worksheet
Complete it with your team.
Record what you know and what still needs confirmation. Answers stay in memory and are not submitted.
References for further reading
- NIST AI RMF Playbook · Measure 1.1 and 1.2 ↗
Provides guidance on choosing measures appropriate to the context, documenting what was not measured and reviewing the assessment when conditions change. It is an evaluation reference; the examples and criteria in this material are proposed ways of working.