Culprit
The flag that broke checkout, and the tool we built so nobody has to hunt for it again
Why mobile teams need to know which feature flag caused a crash spike, what Culprit does about it, where it honestly falls short, and where we're taking it next.
A night every mobile team knows
This isn't one specific incident. It's a composite, and if you've worked on a mobile app with feature flags, you've probably lived some version of it.
It's past midnight. The crash-free rate alert fires. The on-call engineer opens the crash dashboard: crashes are clearly up, but the stack traces point at a dozen different places. They open the flag tool. Eleven flags and three experiments are running. Two rollouts were bumped this afternoon. They scroll Slack to see who changed what. They filter crashes by app version, then by OS, then try to line the timing up with a rollout.
Forty-five minutes later they have a guess: probably the new checkout flow, probably on the latest version. They aren't sure enough to switch it off, so they message the person who owns it, who is asleep.
Notice what went wrong. The tools worked fine. The crash tool caught the crashes and the flag tool held the flags. But the one question that mattered, which change caused this, sat in the gap between them, and a tired person had to bridge it by hand.
On the web, the safety net is a rollback. On mobile there isn't one: once a build is in the store, it lives on people's phones for months. A feature flag is the fastest lever a mobile team has. So the gap between "crashes went up" and "this flag did it" isn't a nice-to-have. It's the difference between a five-minute fix and a lost night.
Where Culprit came from
Culprit didn't start as a crash tool. It started with a broader question: what slows a growing mobile team down? The usual suspects came up straight away: experimentation and feature flags, pull request review, testability, reliability, and monitoring.
Our first instinct was to build one platform that covered all five. Then we looked honestly at what already exists. Feature flags are a solved problem (Firebase Remote Config, Statsig, LaunchDarkly). Crash reporting is a solved problem (Crashlytics, Sentry). PR linting, test frameworks, staged rollouts: all solved, and mostly well. Building a sixth platform that did all of it slightly worse would have helped no one.
So we asked a sharper question instead: where are mobile tools still behind? Almost every answer came back to one fact: mobile can't roll back. Most tools were designed for the web, where that never happens. Among the gaps we listed, one stood out as the most painful and the most fixable:
Crash tools tell you crashes went up. Flag tools tell you what's switched on. Nothing tells you which one caused the other, and nothing switches it off.
That gap became Culprit. The name says the job: find the culprit.
We set ourselves three rules from day one. Don't replace anything a team already uses; read from it. Keep it small: a short wrapper in the app and one scheduled job, with no new servers to run. And never let automation make things worse: anything that can switch off a feature has to be cautious, auditable and easy to undo.
What it does, in plain words
Culprit has three parts.
1. The app tags every crash with the flags the user had
Your app reads watched flags through a tiny Flags wrapper (about 30 lines, no vendor imports). It attaches the variant the user actually got, like ff_new_checkout=on, to every crash and session in your crash tool. It also checks a killswitch_new_checkout switch first, so the app can be told to fall back to the safe version.
2. A job compares every variant with its control
Every 15 minutes, a small job on Google Cloud counts, for the last 7 days, how many users each variant had and how many of them hit a problem. It does this separately for each app version and each kind of problem: crashes, ANRs (the app freezing) and non-fatal errors. Then it asks a simple question: is this variant clearly worse than the people who didn't get it?
"Clearly" is deliberately strict. There must be enough users, enough affected users, a rate at least twice control's, and a statistical test that holds up even though dozens of flags are checked every 15 minutes. Because the test stays valid however often it looks, evidence keeps building over the week, so even a small rollout gets caught once enough users have seen it. Most noise never makes it through.
3. It tells your team, and can step in
When something is clearly worse, your team gets one Slack message that answers the questions the on-call engineer spent 45 minutes on:
killswitch_new_checkout off in Statsig.
Concentrated in: Android 14 (62% vs 20% control) · Samsung SM-S918B (41% vs 9% control).Which flag, which variant, which version, how much worse, how sure, where it's concentrated, and what was done about it. If you've opted that flag in, Culprit has already switched it to the safe path. On their next refresh, users get the control experience and stop crashing.
It works with Crashlytics or Sentry for crash data and Firebase Remote Config or Statsig for flags, in any combination. Every finding is logged to BigQuery, and you can see the history in Grafana or Looker Studio.
Why it helps more than another dashboard
It's tempting to think this is a dashboard problem: put crashes and flags on one screen and let people spot the pattern. We don't think it is, for a few human reasons.
People are bad at spotting causes at 3am, and nobody should have to. Dashboards still ask a person to notice the pattern, form a hypothesis and check it. Culprit does the checking first and hands over a conclusion with the evidence attached. The engineer's job changes from detective to reviewer.
The comparison is fair by design. Comparing users who got a variant with users who didn't, on the same app version and over the same days, filters out most of what fools people: a bad day for one OS, a spike in traffic, an unrelated bug in that release. If the variant users crash and the control users on the same version don't, the variant is the likely culprit.
It catches what crash-free rates hide. A variant rolled out to 5% of users can double their crash rate while barely moving the overall number. Per-version, per-signal comparison finds it anyway, including freezes (ANRs) that never show up as crashes.
It shortens the path from "we know" to "it's fixed". Knowing the cause is half the battle. The other half is finding someone who can flip the flag. For flags you've opted in, Culprit does that within minutes, and leaves a clear note on how to undo it.
It builds trust instead of asking for it. Culprit starts in alert-only mode. You run it against the last 30 days of your own data first and check its findings against incidents you remember. You only let it act once it has earned that. We think that's the right relationship between a team and its automation.
The honest ledger: pros and cons
No tool is free. Here's what Culprit gives you, and what it asks of you.
What you get
- A named cause, not a symptomFlag, variant, app version and signal, with the evidence attached.
- Few false alarmsMinimum samples, a 2× ratio, and multiple-comparison correction.
- Crashes, ANRs and errorsEach checked on its own, so freezes aren't hidden by flat crash rates.
- Where it happensAn OS and device breakdown in every alert.
- Cautious automationOff by default, opt-in per flag, only ever toward the safe path, at most one kill a day.
- Fits your stackCrashlytics or Sentry, Remote Config or Statsig. Nothing is replaced.
- Small footprintA wrapper in the app and one scheduled job. Open source, easy to read.
- A record of what happenedEvery finding and kill is logged, with dashboards ready to use.
What it costs, and its limits
- It's earlyThe logic is well tested offline, but it hasn't been battle-tested in production yet. Start in staging.
- It needs disciplineWatched flags must be read through the wrapper, and the flag list lives in two places that must match.
- It assumes fair assignmentIf a variant is targeted at specific users (say, only new OS versions), the comparison can be biased. Random rollouts work best.
- Minutes, not secondsIt runs every 15 minutes over a 24-hour window. It's a fast safety net, not an instant one.
- Low-traffic flags are slow to judgeFixed thresholds need enough users before they can say anything.
- Some setup costsCrashlytics allows about 10 watched flags. Sentry needs app-start traces at 100%, which uses quota. The job runs on Google Cloud even if your tools don't.
- Breakdowns show shares, not rates"62% of variant crashes are on Android 14" says where to look, not each device's crash rate.
- Abuse is reduced, not impossibleFake crash reports could, in theory, push a flag to control. Caps and limits make this hard and harmless, but not impossible.
Our take: if your team ships behind feature flags and has ever lost an evening to "which change did this?", the costs are small next to the hours it saves. If you rarely use flags, or your rollouts are heavily targeted rather than random, Culprit will have less to offer you.
How to start using it
You can be in alert-only mode in an afternoon. The interactive setup guide shows the exact code and commands for your tools and platform. Here's the shape of it:
- Connect your crash data. With Crashlytics, turn on BigQuery export. With Sentry, sample app-start traces at 100% and create a read-only token.
- Add the
Flagswrapper to your iOS or Android app and connect it to your flag and crash tools. That's a few lines at startup. - Create a kill switch per watched flag: a Remote Config boolean or a disabled Statsig gate.
- List your watched flags in
flags.json, with each flag's safe value. - Backtest. Run
python main.py --backtest 30and check that what it would have flagged matches incidents you remember. - Deploy with one script, in alert-only mode. Watch the alerts for a couple of weeks.
- Turn on auto-kill for one flag once you trust it, and widen from there.
Want to see how strict it is before installing anything? The detector simulator on our homepage runs the same rules in your browser. Try a small, noisy sample and watch it stay quiet.
Where we're going
Culprit is the first step of a bigger idea: tools built for the fact that mobile can't roll back. Here's what's next, roughly in order. These are directions, not dates. Real-world feedback will reorder them.
Make Culprit sharper
- Remember undos and false alarms, so the dashboard shows each kill switch's current state and how often alerts were right.
- Detection per OS version, for regressions too small to move the overall rate.
Meet more teams where they are
- More flag tools such as LaunchDarkly and Unleash, and more crash tools such as Bugsnag and Embrace. Each is one small adapter.
- Field reports from staging and production, so the thresholds and filters are tuned on real data, not just tests.
Siblings for the other gaps we found
- Safe flag cleanup: knowing when a flag can be deleted because almost no one is still on an app version that reads it.
- Backend changes vs old app versions: catching API changes that would break versions still on people's phones.
- Fairer experiment results: correcting A/B results for the uneven way users update their apps.
- Mobile-aware PR review: spotting main-thread work, new permissions and app-size jumps before they ship.
- Store-rejection checks: flagging likely App Store and Play rejections before you submit.
Every one of these comes from the same belief that started Culprit: the hard part of mobile reliability isn't collecting data. It's turning the data you already have into the right decision, quickly, while people sleep.