File size: 3,814 Bytes
fffc143 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 | # Multimodal Deepfake Detection - Data Card
## Datasets Used
### Visual: Hemg/deepfake-and-real-images
- **Source**: https://huggingface.co/datasets/Hemg/deepfake-and-real-images
- **Size**: ~528K images (140K+ used for training)
- **Labels**: Real=1, Fake=0 (flipped to Real=0, Fake=1 during training)
- **Content**: Face images (real photographs vs deepfakes)
- **Preprocessing**: Resize(224x224), ImageNet normalization, augmentation pipeline (flip, rotation, color jitter, Gaussian blur)
### Text: artem9k/ai-text-detection-pile
- **Source**: https://huggingface.co/datasets/artem9k/ai-text-detection-pile
- **Size**: 1.88GB, ~1M samples (20K+ used for training)
- **Labels**: human, ai
- **Content**: Essays, reports, news articles from human authors and AI models
- **Preprocessing**: RoBERTa tokenizer, max_length=512, truncation/padding
## Results
| Component | Validation Accuracy | Notes |
|-----------|--------------------|-------|
| Visual Branch (EfficientNet-B0) | ~73% | 3 epochs on 1K image subset |
| Text Branch (RoBERTa-base) | ~75% | 3 epochs on 500 text subset |
| Multimodal Ensemble | Combines visual + text with learnable weights | Fusion on CPU subset |
## Architecture
- **Visual**: EfficientNet-B0 (5.3M params) + L2 Norm + Dropout + Linear Classifier
- **Text**: RoBERTa-base (125M params) + Mean Pooling + MLP Head
- **Fusion**: Learnable weighted averaging with cross-modal attention (optional)
- **Explainability**: GradCAM on last EfficientNet convolutional block
## Training Configuration
| Parameter | Visual | Text |
|-----------|--------|------|
| Backbone | EfficientNet-B0 | RoBERTa-base |
| Optimizer | Adam | AdamW |
| Learning Rate | 1e-4 | 2e-5 |
| Weight Decay | 1e-4 | 0.01 |
| Epochs | 8 (full), 3 (compact) | 5 (full), 3 (compact) |
| Batch Size | 32 | 16 |
| Image Size | 224x224 | - |
| Text Length | - | 512 |
| Augmentation | Flip, Rot, ColorJitter, GaussianBlur, RandomErasing | - |
## Inference API
```python
from inference import load_model, classify_image, classify_text, classify_video, classify_multimodal
model, config = load_model('multimodal_ensemble.pt')
# Image + GradCAM explainability
result = classify_image(model, 'face.jpg', return_gradcam=True)
# result: {prediction, confidence, gradcam: (224, 224)}
# Text
result = classify_text(model, 'This essay was written by...')
# result: {prediction, confidence}
# Video (aggregated from frame classifications)
result = classify_video(model, 'video.mp4', num_frames=32, aggregation='mean')
# result: {prediction, confidence, frame_scores}
# Multimodal (both modalities with learned fusion weights)
result = classify_multimodal(model, image_path_or_pil='face.jpg', text='Caption...')
# result: {prediction, confidence, modality_scores, fusion_weights}
```
## Files in this Repository
| File | Description |
|------|-------------|
| `model.py` | Core architecture: GradCAM, EfficientNet, RoBERTa, Fusion, Aggregation |
| `preprocessing.py` | Data loading, transforms, video frame extraction, tokenization |
| `inference.py` | Inference API: load_model, classify_image, classify_text, classify_video, classify_multimodal |
| `train.py` | Full training script with separate branch training + ensemble assembly |
| `multimodal_ensemble.pt` | Full ensemble checkpoint (493MB) |
| `visual_branch.pt` | Visual-only checkpoint (15.6MB) |
| `text_branch.pt` | Text-only checkpoint (476MB) |
| `config.json` | Training configuration |
| `requirements.txt` | Python dependencies |
| `gradcam_examples/` | GradCAM explainability visualizations |
## Literature References
- **AWARE-NET** (arxiv:2505.00312): Two-tier weighted ensemble
- **CLIP Deepfake Detection** (arxiv:2503.19683): L2-normalized features
- **DeTeCtive** (arxiv:2410.20964): RoBERTa-based AI text detection
## License
Apache-2.0
|