Llama-2-7b-best_ratio70 (70% Parameters)
This model is a pruned and finetuned version of meta-llama/Llama-2-7b-hf, retaining approximately 70% of parameters while maintaining strong performance through genetic algorithm pruning and RMSNorm fine-tuning.
Model Details
- Base Model: meta-llama/Llama-2-7b-hf
- Parameter Retention: ~70%
- Pruning Method: Genetic Algorithm
- Fine-tuning Method: RMSNorm calibration
Performance
| Metric | Value |
|---|---|
| PPL (Before Fine-tuning) | 11.54 |
| PPL (After Fine-tuning) | 8.25 |
| Improvement | 28.54% |
Performance Comparison
| Model | PPL (After FT) |
|---|---|
| 50% params | 13.84 |
| 70% params | 8.25 |
| 80% params | 6.71 |
| 90% params | 5.62 |
Files Included
- : Full model state dict
- : This documentation
License
Llama 2 Community License (inherited from base model)
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support
Model tree for ra225/Llama-2-7b-best_ratio70_rmsnorm_finetuned
Base model
meta-llama/Llama-2-7b-hf