The agent that improves your agents
Lilium proactively finds what’s holding your agents back, tests improvements against real production scenarios, and brings proven recommendations to Slack. Better outcomes. A smaller inference bill.
At home in your agent stack.
One continuous improvement loop.
Observe
Lilium watches your agents in production, including every model call and tool response.
- Traces
- Errors
- Cost
Understand
Find friction and wasted inference. Turn real production scenarios into evals automatically.
- Signals
- Patterns
Recursively improve
Test model and prompt changes. Get the improvements worth applying in Slack, then repeat.
- Experiments
- Evals
A better version, back in production.
Find what’s holding your agents back.
Lilium watches production traces for failures, user frustration, and expensive behavior. It connects each finding to the runs behind it, so you know what to improve first.
Observe
Follow every model call, tool, and decision in context. See the inputs and outputs behind each step, from the first request to the final response.
Explore production tracesChoose the next action
The replacement policy calls for item-level confirmation before a replacement is issued.
Order context + replacement policy + customer request
Ask which item was damaged. Confirm the item before calling create_replacement.
Signals
Spot pulses of user frustration, repeated requests, and human handoffs. Go from a behavior pattern to the exact conversation behind it.
Explore the observability docsHourly signals, linked to the conversations behind them.
User frustration
The customer repeats the issue after the agent asks for information they already provided.
“I’ve already explained this twice. Can you please just help me?”
The agent asked for the order number again, even though it was already present in the conversation.
Patterns
See related conversations take shape. Discover what your users ask for most, and explore the traces inside each cluster.
Larger circles group more traces. Select a pattern to explore.
Order changes
Customers updating an address, changing an item, or checking whether an order can still be edited.
of 114 in this example
Turn production traces
into tests.
Your agents already encounter the scenarios that matter. Lilium turns them into evals automatically.
Each eval carries the scenario, context, and expected behavior from a real production run. Every proposed improvement is tested against the situations your agents actually face.
The scenario, context, and expected behavior come from the production trace.
Check policy before a refund
The customer asks for a refund outside the return window.
Retrieve the refund policy and escalate an exception. Do not issue a refund without approval.
Each case checks whether the agent meets the expected behavior.
refund_0327Keep the scenarios. Change the agent.
Find better results at a lower cost.
Lilium tries different models and prompt changes, compares them against your evals, and recommends the improvements worth applying. See the quality, inference cost, and latency tradeoffs before you make a change.
Model choices and prompt changes, tested against the same production-derived evals.
Candidate vs. baseline
Example comparisonEvals passed
Higher is better90 / 96 passed
94 / 96 passed
Cost per run
Lower is better$0.0042
$0.0022
The combined candidate improves the eval pass rate and reduces cost in this example.
- Eval pass rate
- 97.9%vs 93.8%
- Cost per run
- $0.0022vs $0.0042
- Latency
- 1.9sExample avg.
Illustrative models, costs, and results. This demo runs locally and does not call a model.
Proactive improvements, delivered in Slack.
Get a message when Lilium finds an issue, finishes an experiment, or identifies a better-performing option. Review the results and decide what to apply.
Lilium does the testing. You make the call.
For agents doing real work.
Example use cases. The same improvement loop, grounded in the work your agents do.
Support agents
Improve resolution quality, catch frustration, and reduce unnecessary escalations.
Sales agents
Improve lead qualification and follow-up while keeping inference costs under control.
Operations agents
Make tool-driven workflows more reliable and cut wasted retries.
npm install @lilium/tracingimport { startSession } from '@lilium/tracing'import { runAgent } from './agent'const session = startSession({ agentId: 'support.answerer', environment: 'production',})const output = await session.run(async () => { return await runAgent()})await session.end({ output })Your stack.
An agent working to improve it.
Keep the tools you already use. Connect with the native TypeScript SDK or send the OpenTelemetry spans you already collect. Lilium takes it from there.
- Native SDKs and standard OTLP ingest
- Model, tool, and sub-agent context
- No-op safe when configuration is missing
A little more clarity.
A few things you might be wondering.
Lilium proactively finds opportunities in production traces, creates evals from real scenarios, and runs model and prompt experiments automatically. It brings findings, experiment results, and recommended improvements to Slack. Your team decides which recommendations to apply; Lilium does not automatically deploy changes to your production agents.
Lilium turns your production datasets into eval cases automatically. Each case carries the scenario, relevant context, and expected behavior from the original run. Every new agent version can then be tested against those cases, without rebuilding the suite by hand.
Lilium tries different models and changes to your prompts, then tests each candidate against your production-derived evals. It compares quality, inference cost, and latency so it can recommend improvements with the evidence attached.
Lilium looks for model and prompt changes that use inference more efficiently, then tests them against your evals. You can see which candidates lower estimated cost while maintaining or improving results, and inspect any tradeoffs before applying a recommendation. Savings depend on your agent and workload.
Connect through the native TypeScript SDK, a provider integration, or OpenTelemetry. Lilium supports OpenAI, Anthropic, and Vercel AI SDK telemetry. You can also send existing OTLP spans without replacing your collector.
Find your integration guideYour agents can
keep getting better.
Let Lilium find the next improvement.