🎯 F1-Score — When Accuracy lies to your face! 📊💥

---

📖 Definition

F1-Score = the metric that doesn't get fooled by imbalanced classes! Accuracy says "99% correct!" but your model just predicts "not cancer" every time. F1-Score says "LIAR, you detected 0% of cancers!" It's the harmonic mean of Precision and Recall, the metric that actually matters.

Principle:

• Harmonic mean of Precision and Recall
• Ranges 0-1: 0 = trash, 1 = perfect
• Balances false positives and false negatives
• Immune to class imbalance: can't cheat with majority class
• Standard metric for imbalanced classification problems! 🔥

---

⚡ Advantages / Disadvantages / Limitations

✅ Advantages

• Detects fake accuracy: 99% accuracy = meaningless if F1 = 0.05
• Single number: combines Precision and Recall elegantly
• Imbalance resistant: works great for 99:1 class ratio
• Industry standard: medical, fraud detection, spam all use it
• Interpretable: 0.9 F1 = excellent, 0.5 F1 = mediocre

❌ Disadvantages

• Ignores true negatives: doesn't care about correctly rejected cases
• Weights errors equally: false positive = false negative (not always true)
• Hard to interpret alone: need to see Precision AND Recall separately
• Not differentiable: can't use directly as loss function
• Threshold dependent: different threshold = different F1

⚠️ Limitations

• Binary focus: multi-class needs variants (macro/micro/weighted F1)
• Assumes equal cost: false positive ≠ false negative in reality
• Harmonic mean harsh: one bad metric (P or R) tanks F1
• No probabilistic info: just 0/1 predictions, loses confidence
• Replaced sometimes: AUC-ROC for ranking, MCC for very imbalanced

---

🛠️ Practical Tutorial: My Real Case

📊 Setup

• Task: Cancer detection from medical images (highly imbalanced!)
• Dataset: 10,000 images (9,900 healthy, 100 cancer = 1%)
• Model: ResNet-18 fine-tuned
• Hardware: GTX 1080 Ti 11GB (batch size 64)
• Config: epochs=50, lr=0.001, optimizer=Adam

📈 Results Obtained

Naive Model (predict "no cancer" every time):
- Accuracy: 99.0% (looks amazing!)
- Precision: 0.0 (undefined, never predicted positive)
- Recall: 0.0 (detected 0 out of 100 cancers)
- F1-Score: 0.0 ❌ (EXPOSED THE LIE!)

Baseline ResNet-18 (no balancing):
- Accuracy: 98.5%
- Precision: 0.20 (20% of positive predictions correct)
- Recall: 0.15 (detected 15 out of 100 cancers)
- F1-Score: 0.17 ❌ (terrible!)

With Class Weighting (1:99 ratio):
- Accuracy: 95.2% (lower but who cares)
- Precision: 0.65 (65% of positive predictions correct)
- Recall: 0.78 (detected 78 out of 100 cancers)
- F1-Score: 0.71 ✅ (much better!)

With Focal Loss + Oversampling:
- Accuracy: 94.8%
- Precision: 0.82 (82% of positive predictions correct)
- Recall: 0.85 (detected 85 out of 100 cancers)
- F1-Score: 0.83 ✅🏆 (excellent!)

GTX 1080 Ti Performance:
- Batch size 64: 9.8GB VRAM used
- Training time: 180 it/s
- Total training: 45 minutes
- F1 tracking: sklearn real-time computation

🧪 Real-world Testing

Spam Detection (90% non-spam, 10% spam):

Accuracy-optimized model:
- Accuracy: 92%
- Precision: 0.45
- Recall: 0.38
- F1-Score: 0.41 ❌

F1-optimized model:
- Accuracy: 88%
- Precision: 0.78
- Recall: 0.82
- F1-Score: 0.80 ✅ (catches 82% of spam!)

Fraud Detection (99.5% legitimate, 0.5% fraud):

Naive model:
- Accuracy: 99.5% (predicts "legitimate" always)
- F1-Score: 0.0 ❌ (catches 0 fraud)

Optimized model:
- Accuracy: 96.8%
- Precision: 0.75 (75% of fraud alerts are real)
- Recall: 0.88 (catches 88% of fraud)
- F1-Score: 0.81 ✅

Multi-class (10 classes, CIFAR-10):
- Macro-F1: 0.89 (average F1 across classes)
- Micro-F1: 0.91 (weighted by class frequency)
- Weighted-F1: 0.90 (weighted by support)

Verdict: 🎯 F1-SCORE = ESSENTIAL FOR IMBALANCED CLASSES

---

💡 Concrete Examples

Understanding Precision, Recall, F1

Medical diagnosis example (100 patients, 10 have cancer)

Model predictions:
- Predicted cancer: 12 patients
- Actually has cancer (of those 12): 8 patients
- Missed cancers: 2 patients

Confusion Matrix:
                 Predicted No    Predicted Yes
Actually No          88              2         (90 healthy)
Actually Yes          2              8         (10 cancer)

True Positive (TP): 8 (correctly detected cancer)
False Positive (FP): 2 (false alarm, healthy flagged as cancer)
False Negative (FN): 2 (missed cancer, cancer flagged as healthy)
True Negative (TN): 88 (correctly identified healthy)

Accuracy = (TP + TN) / Total = (8 + 88) / 100 = 96% ✅
(looks good but misleading!)

Precision = TP / (TP + FP) = 8 / (8 + 2) = 8/10 = 0.80
(80% of positive predictions are correct)

Recall = TP / (TP + FN) = 8 / (8 + 2) = 8/10 = 0.80
(detected 80% of actual cancers)

F1-Score = 2 × (Precision × Recall) / (Precision + Recall)
         = 2 × (0.80 × 0.80) / (0.80 + 0.80)
         = 2 × 0.64 / 1.60
         = 0.80 ✅

F1 = harmonic mean, punishes imbalance between P and R

Why Harmonic Mean matters

Case 1: Balanced Precision and Recall

Precision = 0.80, Recall = 0.80
F1 = 2 × (0.80 × 0.80) / (0.80 + 0.80) = 0.80 ✅

Arithmetic mean would give: (0.80 + 0.80) / 2 = 0.80
Same result when balanced!

Case 2: Imbalanced (high Precision, low Recall)

Precision = 0.95, Recall = 0.30
(very cautious model, rarely predicts positive)

F1 = 2 × (0.95 × 0.30) / (0.95 + 0.30) = 0.46 ❌
(heavily penalized!)

Arithmetic mean: (0.95 + 0.30) / 2 = 0.625
(would hide the problem!)

Harmonic mean punishes imbalance harshly!

Case 3: Imbalanced (low Precision, high Recall)

Precision = 0.30, Recall = 0.95
(aggressive model, predicts positive often)

F1 = 2 × (0.30 × 0.95) / (0.30 + 0.95) = 0.46 ❌
(same penalty as case 2!)

Need BOTH metrics high to get good F1!

Multi-class F1 Variants

3-class example (Animal classification)

Class "Cat": Precision=0.90, Recall=0.85, F1=0.87
Class "Dog": Precision=0.80, Recall=0.90, F1=0.85
Class "Bird": Precision=0.70, Recall=0.60, F1=0.65

Macro-F1 (average F1 per class):
= (0.87 + 0.85 + 0.65) / 3 = 0.79
(treats all classes equally)

Support: Cat=100, Dog=150, Bird=50

Weighted-F1 (weighted by class frequency):
= (0.87×100 + 0.85×150 + 0.65×50) / 300
= (87 + 127.5 + 32.5) / 300
= 0.82
(common classes matter more)

Micro-F1 (global TP/FP/FN):
Total TP = 85 + 135 + 30 = 250
Total FP = 9 + 30 + 13 = 52
Total FN = 15 + 15 + 20 = 50

Precision = 250 / (250 + 52) = 0.828
Recall = 250 / (250 + 50) = 0.833
Micro-F1 = 2 × (0.828 × 0.833) / (0.828 + 0.833) = 0.83

Real applications

Medical Diagnosis 🏥

• Cancer detection, disease screening
• F1 critical: missing cancer = fatal
• High Recall priority (catch all cases)
• Example: F1 = 0.85 minimum required

Fraud Detection 💳

• Credit card fraud, insurance claims
• Balance: catch fraud vs false alarms
• Cost of false positive < cost of false negative
• Target: F1 = 0.80+

Spam Filtering 📧

• Email spam detection
• High Precision priority (don't block real email!)
• But also high Recall (catch most spam)
• Target: F1 = 0.90+

Information Retrieval 🔍

• Search engines, recommendation systems
• Precision = relevant results / total returned
• Recall = relevant results / total relevant
• Target: F1 = 0.75+

---

📋 Cheat Sheet: F1-Score

🔍 Core Formulas

Confusion Matrix:
                Predicted Negative    Predicted Positive
Actually Negative      TN                   FP
Actually Positive      FN                   TP

Accuracy = (TP + TN) / (TP + TN + FP + FN)
(misleading for imbalanced classes!)

Precision = TP / (TP + FP)
"Of positive predictions, how many are correct?"

Recall = TP / (TP + FN)
"Of actual positives, how many did we detect?"

F1-Score = 2 × (Precision × Recall) / (Precision + Recall)
OR
F1 = 2×TP / (2×TP + FP + FN)
(harmonic mean of Precision and Recall)

F-Beta Score (generalizes F1):
Fβ = (1 + β²) × (Precision × Recall) / (β² × Precision + Recall)

β=1: F1 (equal weight)
β=2: F2 (favor Recall 2x)
β=0.5: F0.5 (favor Precision 2x)

⚙️ Multi-class F1 Variants

Macro-F1:
- Calculate F1 per class
- Average them (equal weight)
- Use: when all classes equally important

Micro-F1:
- Sum all TP, FP, FN globally
- Calculate single Precision/Recall/F1
- Use: when accuracy-like metric needed

Weighted-F1:
- Calculate F1 per class
- Weight by class frequency (support)
- Use: when class imbalance exists

🛠️ Interpretation Guide

F1-Score Interpretation:
0.90 - 1.00: Excellent 🏆
0.80 - 0.90: Very Good ✅
0.70 - 0.80: Good 👍
0.60 - 0.70: Acceptable 😐
0.50 - 0.60: Poor ❌
0.00 - 0.50: Very Poor ❌❌

Trade-offs:
High Precision, Low Recall:
- Conservative model (rarely predicts positive)
- Few false alarms, but misses many cases
- Example: F1=0.50 (P=0.95, R=0.35)

Low Precision, High Recall:
- Aggressive model (often predicts positive)
- Catches most cases, but many false alarms
- Example: F1=0.50 (P=0.35, R=0.95)

Balanced:
- Both Precision and Recall high
- Best overall performance
- Example: F1=0.85 (P=0.85, R=0.85)

🛠️ When to use what

Use Accuracy when:
✅ Classes are balanced (50-50, 30-30-40, etc.)
✅ All errors equally costly
✅ Simple binary task with similar class sizes

Use F1-Score when:
✅ Classes are imbalanced (1:99, 10:90, etc.)
✅ Both false positives AND false negatives matter
✅ Medical, fraud, spam detection
✅ Need single metric for model comparison

Use Precision when:
✅ False positives are VERY costly
✅ Example: spam filter (don't block real email!)
✅ Example: recommender system (don't recommend bad items)

Use Recall when:
✅ False negatives are VERY costly
✅ Example: cancer detection (don't miss cancer!)
✅ Example: fraud detection (catch all fraud)

Use AUC-ROC when:
✅ Need threshold-independent metric
✅ Ranking quality matters (not just classification)
✅ Probabilistic outputs important

Use MCC (Matthews Correlation) when:
✅ Extreme imbalance (0.1% positive class)
✅ Need most robust metric
✅ All four confusion matrix values matter

---

💻 Simplified Concept (minimal code)

import numpy as np
from sklearn.metrics import precision_score, recall_score, f1_score
from sklearn.metrics import classification_report, confusion_matrix

class F1ScoreDemo:
    def calculate_metrics(self, y_true, y_pred):
        """Calculate all metrics from predictions"""
        
        precision = precision_score(y_true, y_pred)
        recall = recall_score(y_true, y_pred)
        f1 = f1_score(y_true, y_pred)
        
        print(f"Precision: {precision:.3f}")
        print(f"Recall: {recall:.3f}")
        print(f"F1-Score: {f1:.3f}")
        
        return precision, recall, f1
    
    def manual_f1_calculation(self, y_true, y_pred):
        """Calculate F1 manually from confusion matrix"""
        
        tn, fp, fn, tp = confusion_matrix(y_true, y_pred).ravel()
        
        print(f"\nConfusion Matrix:")
        print(f"TP={tp}, FP={fp}, FN={fn}, TN={tn}")
        
        precision = tp / (tp + fp)
        recall = tp / (tp + fn)
        f1 = 2 * (precision * recall) / (precision + recall)
        
        print(f"\nManual calculation:")
        print(f"Precision: {precision:.3f}")
        print(f"Recall: {recall:.3f}")
        print(f"F1-Score: {f1:.3f}")
        
        return f1
    
    def imbalanced_example(self):
        """Show F1 on imbalanced dataset"""
        
        y_true = np.array([0]*95 + [1]*5)
        
        naive_pred = np.array([0]*100)
        print("Naive model (always predict 0):")
        print(f"Accuracy: {(naive_pred == y_true).mean():.3f}")
        
        balanced_pred = np.array([0]*92 + [1]*4 + [0]*1 + [1]*3)
        print("\nBalanced model:")
        accuracy = (balanced_pred == y_true).mean()
        print(f"Accuracy: {accuracy:.3f}")
        self.calculate_metrics(y_true, balanced_pred)
    
    def multiclass_f1(self, y_true, y_pred):
        """Calculate F1 for multi-class"""
        
        print("Multi-class F1 variants:")
        
        macro_f1 = f1_score(y_true, y_pred, average='macro')
        print(f"Macro-F1: {macro_f1:.3f}")
        
        micro_f1 = f1_score(y_true, y_pred, average='micro')
        print(f"Micro-F1: {micro_f1:.3f}")
        
        weighted_f1 = f1_score(y_true, y_pred, average='weighted')
        print(f"Weighted-F1: {weighted_f1:.3f}")
        
        print("\nPer-class report:")
        print(classification_report(y_true, y_pred))

demo = F1ScoreDemo()

y_true = np.array([0, 0, 1, 1, 0, 1, 0, 1, 1, 0])
y_pred = np.array([0, 0, 1, 0, 0, 1, 1, 1, 1, 0])

demo.calculate_metrics(y_true, y_pred)
demo.manual_f1_calculation(y_true, y_pred)
demo.imbalanced_example()

The key concept: F1-Score is the harmonic mean of Precision and Recall. It punishes imbalance between the two. You can't cheat by optimizing just one! Need both high to get good F1. Perfect for imbalanced classes where accuracy lies! 🎯

---

📝 Summary

F1-Score = harmonic mean of Precision and Recall! The metric that doesn't get fooled by imbalanced classes. Ranges 0-1, punishes models that optimize only Precision OR Recall. Essential for medical, fraud, spam where class imbalance is extreme (1:99 ratio). Variants exist: Macro-F1 (average per class), Micro-F1 (global), Weighted-F1 (by frequency). On GTX 1080 Ti, real-time F1 tracking with sklearn during training. Don't trust accuracy alone! 🔥

---

🎯 Conclusion

F1-Score is essential for real-world classification where classes are imbalanced. Accuracy can show 99% but be useless if the model just predicts the majority class. F1-Score exposes this lie by forcing both Precision and Recall to be high. The harmonic mean is intentionally harsh: one bad metric tanks F1. Modern applications (medical diagnosis, fraud detection, spam filtering) all rely on F1 as the primary metric. Multi-class variants (macro/micro/weighted) extend F1 to complex scenarios. Always report F1 alongside accuracy for honest evaluation. On GTX 1080 Ti, sklearn computes F1 in milliseconds! 🏆📊

---

❓ Questions & Answers

Q: My model has 95% accuracy but F1-Score is only 0.30, what's wrong? A: Your dataset is highly imbalanced and your model is cheating by predicting the majority class! Example: 95% "no cancer", 5% "cancer". Model predicts "no cancer" always = 95% accuracy but F1=0 because it catches 0 cancers. Solutions: (1) Class weighting in loss function, (2) Oversampling minority class (SMOTE), (3) Focal Loss to focus on hard examples, (4) Adjust decision threshold (instead of 0.5, use 0.3), (5) Use F1 as optimization target during training!

Q: Should I optimize for Precision or Recall? A: Depends on cost of errors! (1) High Precision when false positives are costly (spam filter: don't block real email, legal system: don't convict innocent), (2) High Recall when false negatives are costly (cancer detection: don't miss cancer, fraud detection: catch all fraud), (3) Balanced (F1) when both matter equally. Use F-beta score: F2 favors Recall 2x, F0.5 favors Precision 2x. Adjust decision threshold to trade Precision for Recall!

Q: Macro-F1 vs Micro-F1 vs Weighted-F1, which one? A: Macro-F1: average F1 per class (equal weight). Use when all classes equally important. Micro-F1: global TP/FP/FN, single F1. Use when accuracy-like metric needed, dominated by frequent classes. Weighted-F1: F1 per class weighted by frequency. Use when class imbalance exists but frequent classes matter more. For medical diagnosis (rare disease), use Macro-F1 to give equal importance to rare class. For general classification, use Weighted-F1!

---

🤓 Did You Know?

F1-Score was developed in the 1960s in the field of information retrieval for evaluating search engines! Back then, they needed to measure both Precision ("are the results relevant?") and Recall ("did we find all relevant documents?"). The "F" stands for "F-measure" or "F-score", named after F-beta the generalized version. Fun fact: the harmonic mean was chosen instead of arithmetic mean because it heavily penalizes imbalance—if either Precision or Recall is low, F1 is low! This was intentional: you can't game the metric by sacrificing one for the other. In the 1990s-2000s, F1-Score exploded in machine learning as datasets became more imbalanced (spam detection, fraud detection). Today, Kaggle competitions often use F1 as the primary metric for imbalanced datasets. The 2019 cancer detection challenge required F1 > 0.85, which is considered clinical-grade! Even more interesting: GPT models are internally evaluated using token-level F1-Score for tasks like named entity recognition and question answering. The GLUE benchmark (language understanding) reports F1 for several tasks. Modern object detection (YOLO, Faster R-CNN) uses mAP (mean Average Precision) which is closely related to F1! 🎯📊🧠

---

Théo CHARLET

IT Systems & Networks Student - AI/ML Specialization

Creator of AG-BPE (Attention-Guided Byte-Pair Encoding)

🔗 LinkedIn: https://www.linkedin.com/in/théo-charlet

🚀 Seeking internship opportunities

🔗 Website : https://rdtvlokip.fr