alianassmaaa/notification-bad-timing-detector

🔔 Notification Bad-Timing Probability Detector

Overview

Predicts the probability that now is a bad time to send a push notification. Uses 21 contextual signals: user activity patterns, battery status, and notification interaction history.

Performance

Metric	Calibrated Model	5-Model Ensemble
ROC-AUC	0.8338	0.8344
PR-AUC	0.8525	0.8595
Brier Score	0.1657	—
Accuracy	0.7583	—
F1	0.7766	—

Architecture

• Base model: LightGBM gradient boosted trees — SOTA for tabular data per Grinsztajn et al. 2022 and TabArena 2025
• Calibration: Isotonic regression for well-calibrated probability output
• Ensemble: 5 models with different random seeds (averaged for best robustness)
• Hyperparameter tuning: Random search over 40 configurations with early stopping

Features (21 input signals)

Category	Features
Time	hour_of_day, day_of_week, hour_sin, hour_cos, is_weekend, is_night
Battery	battery_level, is_charging, battery_change_rate
Activity	screen_on, screen_on_duration_30min, app_opens_last_hour, session_length_current, time_since_last_interaction
Notifications	notif_shown_last_30min, notif_clicked_last_30min, notif_dismissed_last_30min, notif_ignored_last_30min, notif_shown_last_24h, notif_ctr_last_7d, recent_notification_density

Top Features by Importance

1. notif_ctr_last_7d — 7-day notification click-through rate
2. time_since_last_interaction — seconds since last user action
3. battery_level — current battery percentage
4. notif_shown_last_30min — notification fatigue signal
5. battery_change_rate — battery drain rate

Usage

import pickle, numpy as np

with open("calibrated_model.pkl", "rb") as f:
    model = pickle.load(f)

# Feature order: hour_of_day, day_of_week, hour_sin, hour_cos, is_weekend, is_night,
# battery_level, is_charging, battery_change_rate, screen_on, screen_on_duration_30min,
# app_opens_last_hour, session_length_current, time_since_last_interaction,
# notif_shown_last_30min, notif_clicked_last_30min, notif_dismissed_last_30min,
# notif_ignored_last_30min, notif_shown_last_24h, notif_ctr_last_7d, recent_notification_density

features = np.array([[14, 2, 0.97, -0.22, 0, 0, 85.0, 0, -1.0, 1, 800, 6, 300, 15, 1, 1, 0, 0, 22, 0.4, 1.0]])
bad_timing_prob = model.predict_proba(features)[:, 1][0]
print(f"Bad timing probability: {bad_timing_prob:.3f}")  # ~0.07 = good time!

Decision Thresholds

Probability	Action
P < 0.3	✅ Send notification
0.3 ≤ P ≤ 0.5	⚠️ Consider priority
P > 0.5	🚫 Delay notification
P > 0.8	🔴 Definitely delay

Training Details

• Based on C-3PO (Cheetah Mobile, 600M MAU production system)
• 100K synthetic samples with realistic correlation patterns from mobile behavior research
• 70/15/15 train/val/test split
• Dataset: alianassmaaa/notification-timing-dataset

Files

File	Description
calibrated_model.pkl	Calibrated model (recommended for deployment)
ensemble_models.pkl	5-model ensemble (best accuracy)
model_metadata.json	Features, hyperparameters, metrics
feature_importances.csv	Feature importance rankings
sweep_results.csv	Full hyperparameter sweep results