Shipping velocity has changed faster than review capacity has. With a large share of new code now AI-assisted, features that used to take weeks land in days, but the number of engineers actually reading every line hasn’t grown to match. Independent testing has found a meaningful share of AI-generated code samples carry a known security vulnerability, and that rate has stayed roughly flat even as the models producing the code have improved — the risk hasn’t gone away, it’s just gotten faster to ship. The specific danger for product teams isn’t that AI-generated code is uniformly worse; it’s that it’s inconsistent in a way traditional code isn’t. The same prompt can produce a working fix today and a subtly different, adjacent-breaking one tomorrow, and nobody re-checks the parts that weren’t the target of the prompt.
That inconsistency is exactly why regression tracking needs a source dimension it didn’t need before. A crash spike or error-rate increase is actionable on its own, but a crash spike that correlates with a specific class of commits — AI-authored, minimally reviewed, touching shared code paths — tells you where to tighten process, not just where to hotfix. Without that link, teams end up treating every regression as equally random, when in practice the risk is concentrated in a specific, identifiable slice of how code enters the codebase.
Data Points to Track
- Crash and error rate segmented by commit authorship source — AI-generated, AI-assisted-and-reviewed, or fully human-authored, tagged at merge time so the split survives into production telemetry
- Time-to-regression from merge — how long after an AI-assisted commit ships before an associated crash or error signal appears, since silent regressions often surface days later via an unrelated user path
- Adjacent-functionality breakage rate — errors occurring in code paths near, but not inside, the code an AI-assisted commit directly touched, the classic “fixed the bug, broke the neighbour” pattern
- Review depth per commit — lines changed versus lines actually reviewed by a human, as a leading indicator of which commits are under-scrutinised regardless of who or what wrote them
- Security-relevant pattern flags — known-vulnerability-class matches (injection, auth bypass, unsafe deserialisation) surfaced automatically on AI-assisted commits before merge, not discovered post-release
Setup Steps
- Tag every commit with its authorship source at merge time — AI-generated, AI-assisted, or human-authored — using whatever your AI coding tool exposes, so the tag survives into your crash and error reporting pipeline.
- Propagate the commit tag through to release builds so a crash reported in production can be traced back to the commit class that introduced it, not just the file or line.
- Add an automated security-pattern scan gate specifically for AI-assisted commits, since the vulnerability rate in that class doesn’t drop just because the code looks syntactically clean.
- Track review depth as its own metric — lines changed versus lines a human actually commented on or modified during review — separate from whether review happened at all.
- Build a dashboard that plots regression rate against release velocity by commit source, so a spike in shipping speed and a spike in regressions can be seen as the same event rather than two unrelated ones.
Actionable Insights
The segmented crash rate is the number that tells you whether your review process has actually kept pace with your shipping speed, or just looks like it has because aggregate crash rate hasn’t moved yet. A rising adjacent-functionality breakage rate is a specific, fixable signal — it means AI-assisted changes need a wider blast-radius check before merge, not just a check on the lines that changed. And low review depth on AI-assisted commits is worth treating as a leading indicator, not a footnote — it’s the metric most likely to predict next month’s regressions before they happen, which is the entire point of tracking it separately in the first place.
Related Resources
Need help tracking this in your app?
Our team sets up analytics pipelines for mobile and web teams every day. Talk to us and get your first events flowing in under an hour.
Talk to an expert