Fraud Detection in Mobile Money: Mastering Extreme Class Imbalance

In digital payments and Mobile Money ecosystems, fraudulent transactions represent a minute fraction of overall activity: typically less than 0.05% to 0.1% of total volume. Yet, the financial and regulatory damage caused by an undetected incident is disproportionately high.

When training supervised learning models on such datasets, data scientists face extreme class imbalance. Here is a practical retrospective on common pitfalls and field-tested strategies to build a reliable scoring engine.

1. The Accuracy Paradox

The most common mistake is evaluating a model based solely on raw accuracy. If 99.9% of your transactions are legitimate, a dummy baseline predicting 'legitimate' every single time reaches a 99.9% accuracy score.

On paper, performance looks stellar; in production, you fail to intercept 100% of fraudsters.

In fraud detection, overall accuracy is practically meaningless. The vital metrics are:

• Recall: What fraction of actual fraudulent events did we successfully flag?
• Precision: Out of all generated alerts, how many are true fraud?
• Area Under the Precision-Recall Curve (PR-AUC): Far more reliable and informative than ROC-AUC when positive instances are scarce.

2. Resampling vs. Cost-sensitive Learning

Two main schools of thought address imbalance:

A. Class Weighting (Cost-sensitive Learning)

This is the most computationally efficient technique. Gradient boosting algorithms such as XGBoost and LightGBM provide parameters like scale_pos_weight, which heavily penalize errors on positive (fraudulent) examples during gradient updates.

B. Synthetic Over-sampling (SMOTE)

SMOTE synthesizes minority class instances along lines connecting k-nearest neighbors. Crucial rule: always execute resampling exclusively within the training fold to avoid severe data leakage.

3. Hands-on Implementation with XGBoost

Here is a battle-tested pipeline configured to resist extreme skew:

import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.metrics import classification_report, average_precision_score
from xgboost import XGBClassifier

# Stratified train/test split
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, stratify=y, random_state=42
)

# Compute negative-to-positive ratio
neg_count = np.sum(y_train == 0)
pos_count = np.sum(y_train == 1)
scale_pos_weight = neg_count / max(pos_count, 1)

# Train model with weighted loss
clf = XGBClassifier(
    n_estimators=400,
    learning_rate=0.03,
    max_depth=6,
    subsample=0.8,
    colsample_bytree=0.8,
    scale_pos_weight=scale_pos_weight,
    eval_metric="aucpr",
    random_state=42
)

clf.fit(X_train, y_train)

# Evaluate using PR-AUC
y_probs = clf.predict_proba(X_test)[:, 1]
pr_auc = average_precision_score(y_test, y_probs)
print(f"Test PR-AUC score: {pr_auc:.4f}")

4. Decision Threshold Tuning

By default, predict() uses a fixed 0.5 threshold. In real-world fraud engines, this cutoff should be calibrated to business costs:

• Lowering the cutoff (e.g. to 0.25) increases recall—intercepting more threats at the expense of extra manual reviews.
• Raising the cutoff (e.g. to 0.75) keeps alerts to high-confidence cases, preserving seamless user experience for honest customers.

Key Takeaways

An effective fraud engine relies far more on solid behavioral features (velocity of funds, deviation from typical transaction size, recent device switches) and appropriate minority-class metrics than raw model complexity.