Start
Lilium
Journal
Engineering

From production trace to safe replay

A practical workflow for turning one confusing agent run into a repeatable experiment without losing production context.

July 16, 20261 min read

Large agent traces often contain healthy work around one bad decision. The job is to isolate that decision without discarding the context that produced it.

#Begin with the exact failing branch

Start by selecting the smallest branch that still contains the failure: the parent decision, the affected tool or model span, and the resulting output.

This keeps replay focused. It also makes comparison clearer, because unrelated spans do not overwhelm the signal you are trying to change.

#Decide what stays live

Not every dependency should be called again. External writes, billing operations, and nondeterministic search APIs usually need mocks or captured responses. Read-only tools may be safe to run live when fresh data is part of the test.

A replay plan should record those decisions. Future reviewers need to know whether an apparent improvement came from a prompt change or from a different dependency response.

#Promote the result into an experiment

One successful replay is evidence, not a rollout strategy. Save the candidate configuration, attach the source trace, then evaluate it against a dataset of representative cases.

Production lineage should remain attached as the candidate moves through evaluation and canary traffic. That is how a debugging artifact becomes a trustworthy operational change.

The replay reference covers the mechanics, and why observability needs lineage explains the data model this workflow depends on.