Shipping an AI feature used to mean watching engagement metrics. Now it also means watching a bill that scales with usage in a way most product teams have never had to model before. A support-chat feature or an in-app copilot doesn’t cost a fixed infrastructure fee — every message, every regenerated answer, every long conversation directly consumes tokens that convert into a dollar figure at the end of the month. Without deliberate tracking, that spend arrives as a single opaque line on an invoice, with no visibility into which feature, which user segment, or which single power user is driving it.
That’s a dangerous gap for a product team to have. A feature can look like a clear engagement win — high usage, strong retention lift, good qualitative feedback — while quietly running at a cost per user that makes it unprofitable at scale. By the time that shows up in a margin review, the feature is already deeply embedded in the product and painful to throttle or redesign. Teams that instrument cost attribution from day one can catch that mismatch while it’s still cheap to fix — a prompt that’s needlessly verbose, a retry loop that triples token usage on failure, a small segment of users generating a disproportionate share of spend.
Data Points to Track
- Tokens per session/request: input and output token counts logged per LLM call, tagged with the feature and user that triggered it
- Cost per user and per feature: token counts converted to a dollar figure using current model pricing, rolled up by user, cohort, and feature — not just totalled across the whole app
- Cost per unit of business output: dollars spent per completed task, per resolved support ticket, or per converted user — the benchmark that actually tells you if the spend is worth it, rather than cost per token alone
- Retry and failure-driven token waste: tokens consumed by calls that errored, timed out, or were retried, since this spend produces no user value at all
- Heavy-user concentration: the share of total AI spend coming from your top 1-5% of users by usage, which tells you whether pricing or usage limits need to account for outliers
Setup Steps
- Instrument every LLM call at the request boundary, attaching user ID, session ID, and feature name as metadata so token usage can be attributed without threading identifiers through every function call.
- Log input and output tokens separately, since output tokens are typically priced higher and a feature that generates long responses costs disproportionately more than one that consumes long prompts.
- Convert raw token counts to cost using current model pricing in a scheduled job, rather than only at invoice time, so cost data is available in the same dashboards as your usage and engagement metrics.
- Track retries and failures as their own event, so wasted spend from timeouts or error-handling loops is visible separately from productive usage.
- Set a per-user or per-cohort cost ceiling alert, so an unusually heavy user or a runaway agent loop surfaces as an anomaly before it shows up as a surprise on the monthly bill.
Actionable Insights
Cost per unit of business output — dollars per resolved ticket, per completed task, per converted trial — is the number to report upward, not cost per token or even cost per session, because it’s the only version of this metric that tells you whether the spend is earning its keep. A feature with rising token cost but flat or falling cost-per-outcome is getting more efficient even as the bill grows, which is a good story; the reverse is a warning sign regardless of how good engagement numbers look elsewhere.
Heavy-user concentration data feeds pricing and packaging decisions directly: if a small fraction of users are generating a large share of AI spend, that’s a case for usage-based pricing tiers or soft limits on that feature, rather than absorbing the cost across your whole base and quietly compressing margin on every other customer.
Related Resources
Need help tracking this in your app?
Our team sets up analytics pipelines for mobile and web teams every day. Talk to us and get your first events flowing in under an hour.
Talk to an expert