Failure 16: Silent degradation

Fraud got through for four straight weeks. The dashboard said 98% accuracy the whole time.

The scenario

A fraud filter flags transactions over $900. For three weeks it catches every fraud case. Then the fraudsters notice the threshold and start staying under it, five more cases each week. By week 7 the filter catches none of them.

The team has a monitor, and it works exactly as built. It watches overall accuracy and alerts below 95%. With 980 legitimate transactions and 20 fraudulent ones each week, a filter that misses every fraud case is still right 98% of the time. The alert never fires.

This isn’t a crash and it isn’t a wrong answer you can see in one output. Each week’s result looks fine on its own. The only sign is a slow trend in a number nobody watches. The code is in 16-silent-degradation. The transactions are generated by a deterministic function, so the numbers below come from running it.

The bug

From broken.py:

ACCURACY_ALERT_BELOW = 0.95

def monitor(stats):
    # THE BUG: the only metric is overall accuracy. With 2% fraud, a model
    # that misses every fraud case still scores 98%.
    return stats["accuracy"] < ACCURACY_ALERT_BELOW

The run:

week 1: accuracy 100.0%, missed fraud 0/20, alert=False
week 4: accuracy 99.5%, missed fraud 5/20, alert=False
week 5: accuracy 99.0%, missed fraud 10/20, alert=False
week 6: accuracy 98.5%, missed fraud 15/20, alert=False
week 7: accuracy 98.0%, missed fraud 20/20, alert=False

(Weeks 2 and 3 match week 1, and week 8 matches week 7.) Accuracy never drops below 98%. The filter went from catching all the fraud to catching none, and the dashboard stayed green.

Why it happens

Accuracy counts every correct answer equally. When one class is rare, the common class dominates the score. The metric answers “how often is the model right?” when the question that matters is “how much of the thing I fear does it catch?”

The second cause is organizational. A monitor with a threshold but no owner and no first action produces a notification and no response. Someone has to be named, and something has to happen by default.

The third cause is that the world moves. The model was fine at launch. Nothing in the code changed. Inputs drifted, and the model had no way to say so.

The fix

fixed.py changes the metric and the contract around it.

Measure recall.

"recall": caught / fraud_total if fraud_total else 1.0,

Recall is the share of real fraud the filter caught. It falls the moment fraud slips through, however rare fraud is.

Make every monitor carry an owner and an action.

@dataclass(frozen=True)
class MonitorConfig:
    metric: str
    min_value: float
    owner: str
    action: str

    def __post_init__(self):
        if not self.owner.strip() or not self.action.strip():
            raise ValueError("a monitor needs a named owner and a first action")

You can’t configure a monitor without saying who responds and what they do first. The code refuses to start.

Alert with an instruction.

if value < config.min_value:
    return {"alert": True, "owner": config.owner, "action": config.action,
            "detail": f"{config.metric} {value:.0%} is below {config.min_value:.0%}"}

Run it:

week 3: accuracy 100.0%, recall 100%
week 4: accuracy 99.5%, recall 75%  ALERT -> fraud-ops: route all transactions to manual review (recall 75% is below 90%)
week 5: accuracy 99.0%, recall 50%  ALERT -> fraud-ops: route all transactions to manual review (recall 50% is below 90%)

The first alert fires in week 4, when the first five cases slip. Accuracy at that point is 99.5%, so the old monitor had nothing to say.

The proof

The tests in tests/test_fixed.py:

• test_fixed_stays_quiet_while_the_model_works checks weeks 1 to 3 raise no alert.
• test_fixed_alerts_when_recall_drops_even_though_accuracy_is_high checks week 5 alerts at 50% recall with accuracy at 99%.
• test_fixed_alert_names_an_owner_and_a_first_action checks the alert says who and what.
• test_fixed_refuses_a_monitor_with_no_owner checks the config rejects a blank owner.
• test_fixed_first_alert_is_week_four pins the detection point.
• test_broken_never_alerts_while_every_fraud_case_is_missed pins the original failure.

Where the fix stops

• Recall alone can be gamed by over-flagging. Flag everything and recall is 100% while the review queue drowns. Track precision or the review load next to recall, and set a threshold for each.
• You need labels, and they arrive late. The demo knows which transactions were fraud. In production, confirmed fraud shows up days or weeks after the fact. Watch input drift, such as the amount distribution, as an earlier and weaker signal.
• The threshold is a judgment. 90% recall is a number picked for the demo. Set yours from the cost of a miss against the cost of a review.
• An alert needs a drilled response. “Route everything to manual review” only helps if that path exists and someone has tested it. Practice the fallback before you need it.
• The adversary reads your thresholds too. A fixed cutoff teaches attackers where to stay. Vary what you can, and review what slips under the line.

The takeaway

Monitor the metric that matches the harm, give it a threshold, and name who acts when it breaches.

← Previous: Failure 15 · Next: Failure 17 → · All failures