A long running agent stops being a simple model integration once it can work for hours, call external systems, wait for approval, survive a deployment, and continue on another worker. The model may know which tool to call next. The real issue starts after a crash, when an action may have completed but the result was never recorded.

Consider a support agent that approves a refund. It records the decision, calls the payment provider, and waits. The provider commits the refund, but the response is lost during a network timeout. Before the agent records success, its worker crashes. A replacement loads the last checkpoint, sees an unfinished refund step, and calls the provider again.

The agent recovered exactly as designed. The customer received two refunds.

The same failure appears when an agent creates a ticket, sends a notification, changes a cloud resource, or updates a customer record. Repeating the last tool call will eventually repeat an action that already succeeded.

A transcript is not execution state

Many agent implementations persist the conversation and call that checkpointing. Stored messages rebuild model context, but they do not recover execution safely.

A production run needs an explicit execution model:

agent_run
step
step_attempt
model_decision
tool_operation
tool_observation
checkpoint

A step represents logical progress. An attempt represents one execution of that step. A tool operation represents an intended external effect. An observation records what the agent learned, which is different from whether that effect committed.

Diagram showing an agent run divided into durable workflow state and model context. Workflow state contains steps, approvals, and facts. Each step records attempts, model decisions, and tool operations with prepared, unknown, committed, or rejected states. Unknown operations go through reconciliation before the workflow continues.

Model context helps the agent continue reasoning. Durable workflow state records what has already happened and determines what the system can safely do next.

Persist state around model calls, tool calls, approval waits, timers, handoffs, and context resets. Do not persist every token. Persist decisions, effects, ownership, and recovery evidence.

Define the state machine before writing the loop

A common agent loop looks simple:

while goal_not_complete:
    response = call_model(context)
    result = execute_tool(response.tool_call)
    context.add(result)

This hides every important recovery decision. A durable implementation should move through named states:

READY
DECIDING
DECISION_RECORDED
TOOL_PREPARED
TOOL_RUNNING
TOOL_STATUS_UNKNOWN
TOOL_COMMITTED
WAITING_FOR_APPROVAL
COMPLETED
FAILED

Every transition must be persisted with enough information to continue. The worker executes. The database owns the run state. A new worker should read the durable state, current owner, and next legal transition instead of inferring progress from the last message.

Tool execution needs a commit protocol

Before calling a tool that can change external state, create a tool operation record. Give it a stable identifier derived from the run and logical step, not from the current attempt.

operation_id = hash(run_id, step_id, logical_action)

Store the validated arguments, policy decision, approval reference, provider account, and operation status. Commit this record before the external call, then send the same identifier as an idempotency key when supported.

record operation intent
commit local transaction
invoke external tool
record returned result
mark operation committed
advance workflow state

There is still a gap between the external commit and the local commit. No database transaction can atomically cover your database and an arbitrary external API. That gap cannot be removed by retries. It must be handled through idempotency or reconciliation.

A stable key works only when the receiver enforces it. The adapter must know the provider scope and retention period because keys may expire, be scoped to one account or endpoint, or reject changed arguments.

Follow one failure all the way through

10:31:12
step 18 enters TOOL_PREPARED
operation_id = refund_run_841_step_18

10:31:13
provider accepts the request
provider_ref = rf_98321

10:31:43
client times out
local status = TOOL_STATUS_UNKNOWN

10:31:45
worker lease expires

10:32:02
replacement worker receives fencing token 72

10:32:03
reconciliation queries the provider using operation_id

10:32:04
provider returns refund rf_98321

10:32:05
operation marked TOOL_COMMITTED
workflow advances without another refund
Sequence diagram showing an agent worker creating a refund operation, the payment provider committing the refund, and the response being lost before the local database records the result. After the worker crashes, a replacement worker acquires a fencing token, reconciles using the original operation ID, finds the existing refund, and continues without issuing it again.

The payment provider commits the refund, but the result is not recorded locally. The replacement worker reconciles the unknown operation and finds the existing refund instead of creating a second one.

The timeout proves only that the caller stopped waiting. The replacement worker reconciles before retrying. Treating transport failure as business failure creates the duplicate action.

Classify tools by recovery behavior

Do not apply one retry policy to every tool. Register recovery semantics with each tool definition.

A pure read can usually be repeated, although it may return newer data. An idempotent write can be repeated with the original operation identifier. A non idempotent write must be reconciled first. An irreversible action may require manual resolution when status is unknown.

The tool registry should include:

effect_type
idempotency_support
idempotency_scope
idempotency_retention
reconcile_handler
timeout_policy
retry_policy
approval_policy
compensation_handler

This turns tool safety into executable policy. The model can propose an action, but it should not decide whether an unknown write is safe to repeat. Compensation is also a new operation with its own authorization and failure modes, not a rollback.

Recovery starts with reconciliation

After a crash, an operation can be prepared, committed, failed before commit, or unknown. Unknown is the dangerous state because the external system may have committed even though the caller saw a timeout.

if operation.status == TOOL_COMMITTED:
    continue_workflow()

elif operation.status == TOOL_PREPARED:
    execute_with_original_id()

elif operation.status == TOOL_STATUS_UNKNOWN:
    external_state = reconcile(operation)

    if external_state == COMMITTED:
        record_external_result()
        continue_workflow()

    elif external_state == NOT_FOUND:
        execute_with_original_id()

    else:
        pause_for_manual_resolution()

Unknown is a real state. Do not silently convert it to failed because it is inconvenient to represent.

Exactly once is not the guarantee

A workflow runtime should not claim exactly once tool execution across its database and arbitrary external systems. A worker can crash after the external system commits and before the local transaction records the result. The workflow engine cannot remove that ambiguity.

Workers and events usually have at least once behavior. Correctness comes from stable operation identity, idempotency, deduplication, fencing, reconciliation, and manual resolution. Every repeated attempt carries the same logical identity, and an uncertain result is reconciled before another effect is created.

Prevent two workers from owning the same run

A paused worker can wake after its lease expires while a replacement is continuing the run. Both may execute the next tool.

Use a lease with a monotonically increasing fencing token. Every time ownership changes, increment the token. Include it in state updates and, where possible, downstream operations.

A stale worker holding token 41 must be rejected after token 42 has been issued. A timestamp based lease without fencing is not enough because the old worker may continue after a long pause or network partition.

update agent_run
set state = next_state
where run_id = ?
and version = expected_version
and fence_token = current_token

If zero rows are updated, the worker lost ownership and must stop.

Context recovery is a separate problem

A workflow can resume correctly and still give the model bad context. Long runs accumulate stale observations and abandoned branches. Replaying the complete transcript eventually becomes expensive and confusing.

Store canonical workflow state separately from model context. It should contain the objective, completed steps, verified facts, open operations, approvals, constraints, artifacts, and next allowed actions. Build context from this state plus selected recent events.

When context is compacted or reset, create a validated, versioned handoff artifact. Free form summaries often omit an unresolved operation or turn an assumption into a fact. Workflow completion must come from durable state.

Approval must bind to an exact action

Human approval is not a boolean attached to a run. It must bind to the exact tool, arguments, policy version, and resource version that the reviewer saw.

approval_hash = hash(
    tool_name,
    normalized_arguments,
    policy_version,
    resource_version
)

Calculate the fingerprint again before execution. A mismatch returns the workflow to approval instead of allowing the model to reinterpret the earlier decision.

While waiting, persist the condition, release the worker, and resume from a deduplicated approval event.

Define the execution invariants

A durable agent runtime should maintain these invariants:

  1. Every logical external action has one stable operation identifier across retries and worker replacements.
  2. Only the worker holding the current fencing token may advance the run.
  3. Every side effect is recorded as prepared, committed, rejected, or unknown.
  4. An unknown operation is reconciled before it is repeated.
  5. Approval applies only to the exact action and arguments reviewed.
  6. Model decisions may be regenerated. External effects are not regenerated without checking prior execution.
  7. Workflow completion comes from durable execution state, not from the transcript.

These invariants remain useful when the runtime, model, or tool stack changes.

Version everything that affects replay

A run may survive longer than a deployment. During that time, prompts, tools, schemas, policies, and models can change.

Record the versions used for every decision. A tool schema change can reinterpret stored arguments. A policy change can invalidate approval. A prompt change can produce a different decision from the same durable state.

Define migration rules for active runs. Some changes apply only to new steps. Others require rebuilding context, repeating approval, or finishing on the old worker version. Deployment timing should not make this choice.

Observe decisions, effects, and recovery

A trace of model calls is not enough. The incident timeline must connect the model decision to the external effect and the recovery decision.

Record the run identifier, step identifier, attempt number, operation identifier, fence token, prompt version, model version, tool version, approval identifier, external reference, timeout stage, reconciliation result, and state transition.

Alert on correctness risk, not only latency. Useful signals include unknown operations older than their reconciliation objective, repeated lease loss, approval fingerprint mismatch, compensation frequency, runs with no legal transition, and operations requiring manual resolution.

The most useful production metric is often not agent success rate. It is the number of runs whose external effect is uncertain.

Long running agents do not become reliable because the model can plan for longer. They become reliable when every important decision and external effect has a durable identity, a legal state transition, and a defined recovery path. The model can remain nondeterministic. The execution system cannot remain ambiguous.