General lesson
Agent teams often treat recovery as an error-handling detail: catch the exception, retry the tool, and ask a human if retries fail. That model assumes the error accurately describes the external result. In real workflows, a timeout can arrive after a payment, message, deployment, or database mutation already succeeded. Retrying may create a second side effect while doing nothing to repair the first.
The sharper model is a recovery contract attached to the action before execution. It states what the action can change, how success will be observed, which duplicate guard applies, what can undo or offset the effect, who owns unresolved outcomes, and when automation must stop. Reliability is not the absence of tool errors. It is the ability to reconcile intended and observed outcomes without improvising authority after the incident begins.
One Retry Policy Cannot Fit Every Side Effect
A draft update, a customer email, a payment, and a production deletion do not share the same recovery physics. A versioned draft may be reversible by restoring the prior value. An email cannot be unsent, but its consequences may be compensatable with a correction. A runaway campaign may be containable by pausing future sends even though delivered messages remain delivered. A disclosed secret or destructive physical action may be effectively irreversible.
Treating these actions alike hides the decision that matters most: what remedy is still available after partial success? Classify each mutating tool as reversible, compensatable, containable, or irreversible. The class determines whether the agent may auto-execute, whether it needs preview or human approval, which evidence it must persist, and whether a retry is safe, conditional, or forbidden.
Project example
Authority Router provides a public portfolio example of the pre-execution boundary: a workflow should resolve who or what may perform an action, under which scope, before a tool mutates an external system. The portfolio Content & Growth Engine and Prospr show the adjacent operational boundary: an approved article must not silently become public, and a reviewed application artifact must not silently become a sent application. Public project context: portfolio projects.
The transferable lesson is that valid authorization and safe recovery answer different questions. Authorization asks whether this actor may attempt this action. Recovery asks how the product will prove what happened, prevent duplicate effects, and restore, compensate, contain, or escalate the result. A trustworthy workflow needs both contracts, linked by the same planned action identifier.
Implementation pattern
Define an action contract such as {action_id, tool, target, authority_plan_ref, risk_class, expected_preconditions, idempotency_key, intended_effects, forbidden_effects, verification_probe, retry_policy, recovery_operation, recovery_owner, escalation_deadline}. Persist it before execution. After the tool returns—or times out—write an append-only receipt containing the request fingerprint, provider or system reference, timestamps, reported result, observed postconditions, cost, and recovery state.
Drive the state machine from evidence rather than exceptions: planned → executing → effect_unconfirmed → verified, with branches to recovering, contained, compensated, or needs_operator. An unknown result is not the same as failure. Query the external system using the idempotency key or provider reference before retrying. If verification remains ambiguous past the deadline, stop autonomous action and hand the receipt, evidence gap, and allowed remedies to the named owner.
Failure modes and trade-offs
The most dangerous failure is blind retry after an ambiguous timeout. Others include writing the receipt only after success, storing an idempotency key that the downstream provider does not honor, declaring compensation without testing it, and allowing the same agent that caused a high-risk effect to expand its own recovery authority. Recovery can also fail through time: a refund window closes, a deployment rollback becomes unsafe after a schema change, or a queued correction reaches the wrong audience because the target changed.
Recovery machinery has a cost. Not every low-impact edit needs a saga framework, and compensation is not equivalent to erasure: a corrected message still consumed attention, a refund still creates customer effort, and restored data may not retract downstream copies. Start with actions that cross organizational boundaries, create cost, affect public truth, change permissions, or expose personal data. Keep the mechanism proportional, but never describe a compensating action as if the original effect did not happen.
Concrete diagnostic
Choose one agent tool that mutates an external system and inject six failures: timeout before the request leaves, timeout after acceptance, duplicated callback, partial multi-step success, verification endpoint outage, and failed compensation. For each case, ask whether the system can identify one planned action, discover the actual external effect, block unsafe retries, select an allowed remedy, and give an operator enough evidence to decide without reading raw chat history.
The contract fails if two side effects can share the same action id, if an unknown result is automatically labelled failed, if compensation has no owner or deadline, or if the workflow can resume autonomous execution while the previous effect remains unresolved. Useful measures include ambiguous-effect rate, duplicate-effect blocks, mean time to verification, compensation success, containment time, unresolved receipts past deadline, and incidents where the executed remedy exceeded the original authority scope.
What changes in practice
Once recovery is risk-tiered, product teams can make autonomy proportional to consequence. Low-risk reversible edits can execute quickly. Compensatable actions require stronger receipts and tested remedies. Containable actions need kill switches and bounded blast radius. Irreversible actions move toward explicit human control, narrow scopes, and stronger preconditions because the recovery contract honestly says that no technical undo exists.
Apply this tomorrow by listing the five mutating tools in one agent workflow and assigning each a recovery class. Pick the highest-risk action, add one stable action id, one postcondition probe, one retry prohibition for ambiguous outcomes, and one named recovery owner. Then simulate a timeout after the external effect succeeds. If the system cannot determine what happened without repeating the action, the workflow is not ready for more autonomy.