The agent that improves your agents

Lilium proactively finds what’s holding your agents back, tests improvements against real production scenarios, and brings proven recommendations to Slack. Better outcomes. A smaller inference bill.

Explore the platform

At home in your agent stack.

One continuous improvement loop.

  1. Observe

    Lilium watches your agents in production, including every model call and tool response.

    • Traces
    • Errors
    • Cost
  2. Understand

    Find friction and wasted inference. Turn real production scenarios into evals automatically.

    • Signals
    • Patterns
  3. Recursively improve

    Test model and prompt changes. Get the improvements worth applying in Slack, then repeat.

    • Experiments
    • Evals

A better version, back in production.

Find what’s holding your agents back.

Lilium watches production traces for failures, user frustration, and expensive behavior. It connects each finding to the runs behind it, so you know what to improve first.

Observe

Follow every model call, tool, and decision in context. See the inputs and outputs behind each step, from the first request to the final response.

Explore production traces
Production trace
Illustrative run
support-agent / run_10424 steps
Every step stays connected to its context.
Trace context

Choose the next action

The replacement policy calls for item-level confirmation before a replacement is issued.

Input

Order context + replacement policy + customer request

Output

Ask which item was damaged. Confirm the item before calling create_replacement.

Every step stays connected to its context.Example data

Signals

Spot pulses of user frustration, repeated requests, and human handoffs. Go from a behavior pattern to the exact conversation behind it.

Explore the observability docs
Behavior signals
Illustrative workspace
Signals over timeMoments worth a closer look
Last 24 hours

Hourly signals, linked to the conversations behind them.

Behavior signal14:32

User frustration

The customer repeats the issue after the agent asks for information they already provided.

“I’ve already explained this twice. Can you please just help me?”
What happened

The agent asked for the order number again, even though it was already present in the conversation.

support_0781
Signals stay connected to their source traces.Example data

Patterns

See related conversations take shape. Discover what your users ask for most, and explore the traces inside each cluster.

Conversation patterns
Illustrative workspace
Patterns across conversationsRelated traces, grouped automatically
114 traces

Larger circles group more traces. Select a pattern to explore.

Conversation pattern

Order changes

Customers updating an address, changing an item, or checking whether an order can still be edited.

42related traces
of 114 in this example
Conversations in this cluster
Change a delivery address
Update an unshipped order
Remove an item before dispatch
Related conversations, grouped by pattern.Example data

Turn production traces
into tests.

Your agents already encounter the scenarios that matter. Lilium turns them into evals automatically.

Each eval carries the scenario, context, and expected behavior from a real production run. Every proposed improvement is tested against the situations your agents actually face.

Production traces Evals generated automatically Every new agent version
Generated eval suite3 cases

The scenario, context, and expected behavior come from the production trace.

Policy compliance

Check policy before a refund

Production scenario

The customer asks for a refund outside the return window.

Expected behavior

Retrieve the refund policy and escalate an exception. Do not issue a refund without approval.

Evaluation resultPassFail

Each case checks whether the agent meets the expected behavior.

Source tracerefund_0327
Every new version gets tested.

Keep the scenarios. Change the agent.

Illustrative generated evals and simulated test results.No production calls

Find better results at a lower cost.

Lilium tries different models and prompt changes, compares them against your evals, and recommends the improvements worth applying. See the quality, inference cost, and latency tradeoffs before you make a change.

Automatic experiments
Same 96 evals. Every candidate.

Model choices and prompt changes, tested against the same production-derived evals.

Candidate vs. baseline

Example comparison

Evals passed

Higher is better

90 / 96 passed

BaselineModel A / Prompt v4

94 / 96 passed

CandidateModel B / Prompt v5
4 more evals passed

Cost per run

Lower is better

$0.0042

BaselineUSD / run

$0.0022

CandidateUSD / run
47.6% lower inference cost

The combined candidate improves the eval pass rate and reduces cost in this example.

Eval pass rate
97.9%vs 93.8%
Cost per run
$0.0022vs $0.0042
Latency
1.9sExample avg.
Production context stays attached to every experiment.

Illustrative models, costs, and results. This demo runs locally and does not call a model.

Try different modelsExperiment with promptsCompare against your evalsKeep improving with production

Proactive improvements, delivered in Slack.

Get a message when Lilium finds an issue, finishes an experiment, or identifies a better-performing option. Review the results and decide what to apply.

Lilium does the testing. You make the call.

Illustrative Slack updateExample message
agent-improvements
LiliumApp

An improvement worth reviewing

Your support agent passed more production-derived evals with a lower estimated inference cost per run.

Quality
More evals passed
Inference cost
Lower estimated cost per run

Model and prompt changes tested. Ready for your team to review.

View experiment

For agents doing real work.

Example use cases. The same improvement loop, grounded in the work your agents do.

  • Support agents

    Improve resolution quality, catch frustration, and reduce unnecessary escalations.

  • Sales agents

    Improve lead qualification and follow-up while keeping inference costs under control.

  • Operations agents

    Make tool-driven workflows more reliable and cut wasted retries.

npm install @lilium/tracing
agent.tsTypeScript
import { startSession } from '@lilium/tracing'import { runAgent } from './agent'const session = startSession({  agentId: 'support.answerer',  environment: 'production',})const output = await session.run(async () => {  return await runAgent()})await session.end({ output })
A few lines. The whole story.

Your stack.
An agent working to improve it.

Keep the tools you already use. Connect with the native TypeScript SDK or send the OpenTelemetry spans you already collect. Lilium takes it from there.

  • Native SDKs and standard OTLP ingest
  • Model, tool, and sub-agent context
  • No-op safe when configuration is missing
Make yourself at home in the docs

A little more clarity.

A few things you might be wondering.

Your agents can
keep getting better.

Let Lilium find the next improvement.

Get started with Lilium.

Tell us about your team and where you are with LLMs.

What best describes your role?
How are you working with LLMs?

We’ll use these details to follow up about Lilium.