Most apps now ship some kind of in-app AI agent — a support assistant, a copilot that drafts content, a feature that plans a task on the user’s behalf. Standard event tracking wasn’t built for this. A traditional feature is deterministic: the same tap produces the same screen every time, so a funnel of “tapped X, saw Y, completed Z” tells you almost everything. An agent is probabilistic — the same prompt can produce a different response, call a different tool, or fail in a different way each time — so a simple “used the feature” event tells you almost nothing about whether it actually worked.
Teams that track in-app agents the way they’d track a button end up with a usage number that goes up while user trust quietly goes down, because nothing in the data captures whether the agent actually resolved what the user asked for.
Data Points to Track
- Containment rate: the share of agent interactions resolved without the user falling back to a human, a search bar, or a manual workflow — the single clearest signal of whether the agent is actually useful
- Tool-call success rate: for agents that call internal functions or APIs on the user’s behalf, the percentage of tool calls that complete without error, separate from whether the final response was well-phrased
- Task completion vs. abandonment: whether a multi-turn agent session ends in the user’s goal being met, or the user leaving mid-conversation
- Regeneration and correction rate: how often a user asks the agent to try again or manually edits its output — a proxy for response quality that doesn’t require human review of every conversation
- Latency per turn: time from user input to agent response, tracked separately from the LLM call itself so you can isolate app-side overhead from model latency
- Cost per resolved task: token spend divided by successfully contained interactions, since raw token cost alone doesn’t tell you if the spend is buying real outcomes
Setup Steps
- Define “resolved” explicitly for each agent surface before instrumenting anything — a support agent resolves by avoiding escalation, a drafting agent resolves by the user accepting output without heavy edits. Without this definition, containment rate is meaningless.
- Log every tool call the agent makes with its arguments, success/failure state, and latency, not just the final rendered response — this is where most production agent failures actually live.
- Capture regenerate, edit, and abandon actions as first-class events on the same session ID as the original agent turn, so quality signals can be joined back to the interaction that caused them.
- Track cost alongside outcome from day one — token usage without a resolution metric next to it makes it impossible to tell if spend is increasing because usage is growing or because the agent is failing and retrying.
- Sample raw conversation transcripts for manual review on a rolling basis, since automated metrics catch failure rate but miss failure mode — you still need humans reading actual transcripts to know why containment dropped.
Actionable Insights
A falling containment rate with a rising regeneration rate almost always means the agent’s underlying prompt, tool set, or context window needs attention, not that users have stopped wanting the feature. Conversely, high containment with low usage volume suggests a discoverability problem — the agent works when people find it, but the entry point is buried.
Because agent behaviour isn’t deterministic, resist the urge to treat any single session as representative. Look at trends across days and cohorts rather than reacting to one bad transcript, but treat a sustained multi-day dip in containment rate with the same urgency as a crash-rate spike — it’s the equivalent signal for a feature that fails silently instead of loudly.
Related Resources
Need help tracking this in your app?
Our team sets up analytics pipelines for mobile and web teams every day. Talk to us and get your first events flowing in under an hour.
Talk to an expert