A fraud model can hold its AUC all quarter and still lose you millions. I know because I’ve watched it happen, from the inside.
Before Tezo, I spent years in specialized analytics at a global US bank, building and living with exactly these models. Every peak season taught me the same lesson from a new angle: the dashboard wasn’t wrong, exactly. It was answering a question nobody was asking anymore.
Here are the five metrics I learned to distrust between October and January, and the cut I’d demand instead of each.
The most reassuring number on the dashboard, and the slowest to notice trouble.
AUC tells you the model still rank-orders across the whole population. But December’s losses don’t happen “across the whole population.” They happen in pockets: one attack pattern, one product, one customer segment. A model can rank-order beautifully on average while a fraud ring walks through a single corridor it has stopped seeing.
What I’d ask for instead: discriminatory power cut by segment and by fraud typology, weekly. The aggregate is where the pockets go to hide.
“The model is well calibrated” is a sentence that needs a follow-up question: for whom?
Peak season flips the population toward first-time and thin-file customers, precisely the cohort where probability estimates are shakiest. Global calibration stays green because the established-customer majority behaves. Meanwhile the model is confidently wrong about the exact people your acquisition budget just paid to bring in.
The cut I’d demand: calibration by cohort, with new customers as a first-class segment. Not a footnote in the annual validation. A weekly view.
My least favorite metric in December, because it’s technically true and practically misleading.
Hold the rate steady and multiply volume by peak: the count of good customers declined explodes, concentrated in the highest-revenue weeks of the year. The model team reports a stable FP rate and gets a nod. Finance ends the quarter asking why December conversion dipped. Nobody connects the two, because they live in different decks.
What belongs on the dashboard: FP counts and projected revenue-at-risk at peak volume, reviewed by fraud and revenue owners in the same meeting. Rates are for statisticians. Counts are for the business.
Stability metrics on a monthly review cycle are drift detectors with the batteries removed.
A promotional weekend can shift input distributions in three days. Averaged into a monthly PSI, that spike becomes a rounding error, and the review meeting happens two weeks after the drift already repriced your scores. The math is fine. The cadence is the lie.
The fix costs almost nothing: weekly drift monitoring, with baselines built from last year’s peak weeks rather than the calendar-year average. December should be compared to December.
KS tells you where the model separates good from bad best. It says nothing about whether that point is still where you’re operating.
A global KS in the 40s reads as a healthy model, full stop. But peak season doesn’t just move volume, it moves the score distribution itself: first-time buyers and gift-card-funded accounts pile up in a different part of the score range than the rest of the year. The maximum gap between good and bad can hold steady in aggregate while it quietly collapses at the exact score band your threshold sits on, because that’s no longer where the population’s best separation lives.
What I’d ask for instead: KS by segment, tracked alongside the score band where it peaks, weekly. A KS of 40 means nothing if it peaks two hundred points away from your cutoff.
None of these is a broken metric. Each is a correct answer to a question that stopped mattering when the season changed: averages, when the losses live in pockets; rates, when the damage lives in counts; monthly views, when the drift lives in weeks; a single separation point, when the operating threshold has moved.
The models I built weren’t fooled by December. My dashboards were.
Here’s where this stopped being theoretical for me. Take a real-time transaction-fraud model at a global US bank, scoring every card swipe, digital payment, and account login across millions of customers from October through January (for a co-branded card): a gradient-boosted classifier, roughly 40 features spanning device and network signals, velocity checks, historical behavior, and transaction attributes, retrained quarterly on a trailing 18 months of labelled outcomes, with a single probability threshold deciding approve, decline, or step up to additional authentication.
Here’s what the calendar-year, monthly dashboard reported in the first week of December, next to what the same model looked like cut by segment and by week:
| Metric | Standard view (monthly, whole population) | Segmented, weekly view | What it hid |
|---|---|---|---|
| Discriminatory power | Global AUC: 0.93 | New-customer segment: 0.81. Account-takeover corridor: 0.74 | Two of the highest-loss corridors were quietly losing rank-order while the aggregate stayed green |
| Calibration | Brier score: 0.041 (“well calibrated”) | New-customer cohort Brier score: 0.089, systematically overconfident | The model was most wrong about the exact customers acquisition had just paid to bring in |
| False positives | FP rate: 2.1%, steady since August | FP count: ~8,400/week in October vs. ~21,000/week at Black Friday peak (~$3.1M in blocked legitimate transaction volume that week, plus a spike in call-center and branch complaints) | A stable rate masked a 2.5x jump in good customers turned away, in the busiest payment week of the year |
| Stability | Monthly PSI: 0.08, no material drift | Weekly PSI, Black Friday week: 0.31, major drift | The monthly average buried a three-day distribution shift; the review meeting caught it twelve days after it had already repriced scores |
| Separation (KS) | Global KS: 42, calendar-year average | Black Friday week KS: 39 overall, but peak separation shifted to a score band 210 points above the live cutoff | The headline number barely moved, but the model’s best separation was no longer happening anywhere near where the threshold actually cut |
None of those five monthly numbers were false. Each one was true and green. The model’s actual performance (the one customer and the P&L experienced that week) only showed up once the same numbers were cut by segment and by week.
If your fraud metrics are green right now, ask a simple question: green compared to what?
If the baseline is a calendar-year average and the cadence is monthly, you’re not monitoring the model. You’re monitoring last quarter’s memory of it.
Don’t miss out on the latest tips, tools, and tactics at the forefront of HR and Employee
We love what we do so much and we're always looking for the next big challenge, the next problem to be solved, the next idea that simply needs the breath of life to become a reality. What's your challenge?
