Add comprehensive README.md with documentation
Browse files
README.md
CHANGED
|
@@ -1,10 +1,94 @@
|
|
| 1 |
---
|
| 2 |
title: Universal Model Trainer
|
| 3 |
-
emoji:
|
| 4 |
-
colorFrom:
|
| 5 |
-
colorTo:
|
| 6 |
sdk: docker
|
| 7 |
-
pinned:
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 8 |
---
|
| 9 |
|
| 10 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
---
|
| 2 |
title: Universal Model Trainer
|
| 3 |
+
emoji: π
|
| 4 |
+
colorFrom: blue
|
| 5 |
+
colorTo: purple
|
| 6 |
sdk: docker
|
| 7 |
+
pinned: true
|
| 8 |
+
license: mit
|
| 9 |
+
short_description: Universal ML training dashboard for HuggingFace
|
| 10 |
+
tag:
|
| 11 |
+
- training
|
| 12 |
+
- machine-learning
|
| 13 |
+
- transformers
|
| 14 |
+
- pytorch
|
| 15 |
+
- docker
|
| 16 |
---
|
| 17 |
|
| 18 |
+
# π Universal Model Trainer
|
| 19 |
+
|
| 20 |
+
A comprehensive, production-ready dashboard for training machine learning models on the HuggingFace ecosystem. Supports multiple task types, datasets, and model architectures with full user control.
|
| 21 |
+
|
| 22 |
+
## β¨ Features
|
| 23 |
+
|
| 24 |
+
### Supported Task Types
|
| 25 |
+
- **Causal Language Modeling** - Text generation (GPT-style)
|
| 26 |
+
- **Masked Language Modeling** - BERT-style pretraining
|
| 27 |
+
- **Sequence-to-Sequence** - Translation, summarization
|
| 28 |
+
- **Token Classification** - NER, POS tagging
|
| 29 |
+
- **Text Classification** - Sentiment, topic classification
|
| 30 |
+
- **Question Answering** - Extractive QA
|
| 31 |
+
- **Image Classification** - Vision tasks
|
| 32 |
+
- **Audio Classification** - Sound classification
|
| 33 |
+
|
| 34 |
+
### Training Features
|
| 35 |
+
- **PEFT/LoRA Support** - Memory-efficient fine-tuning
|
| 36 |
+
- **DeepSpeed Integration** - Distributed training
|
| 37 |
+
- **Quantization Options** - 4-bit, 8-bit training
|
| 38 |
+
- **Custom Hyperparameters** - Full control over training config
|
| 39 |
+
- **Real-time Progress** - Live training metrics
|
| 40 |
+
- **Job Queue System** - Manage multiple training jobs
|
| 41 |
+
- **Model Versioning** - Track experiment history
|
| 42 |
+
|
| 43 |
+
### Dataset Support
|
| 44 |
+
- **HuggingFace Hub** - 100,000+ datasets
|
| 45 |
+
- **Custom Upload** - CSV, JSON, JSONL support
|
| 46 |
+
- **Dataset Preprocessing** - Automatic tokenization
|
| 47 |
+
|
| 48 |
+
## π§ API Endpoints
|
| 49 |
+
|
| 50 |
+
### Training
|
| 51 |
+
```
|
| 52 |
+
POST /api/train/start - Start a new training job
|
| 53 |
+
GET /api/train/status - Get job status
|
| 54 |
+
POST /api/train/stop - Stop running job
|
| 55 |
+
GET /api/train/history - View training history
|
| 56 |
+
```
|
| 57 |
+
|
| 58 |
+
### Models & Datasets
|
| 59 |
+
```
|
| 60 |
+
GET /api/models/search - Search HuggingFace models
|
| 61 |
+
GET /api/datasets/search - Search HuggingFace datasets
|
| 62 |
+
GET /api/models/info - Get model info
|
| 63 |
+
```
|
| 64 |
+
|
| 65 |
+
### System
|
| 66 |
+
```
|
| 67 |
+
GET /api/system/status - System health and resources
|
| 68 |
+
GET /api/system/gpus - Available GPU info
|
| 69 |
+
```
|
| 70 |
+
|
| 71 |
+
## ποΈ Architecture
|
| 72 |
+
|
| 73 |
+
- **FastAPI Backend** - Modern async Python web framework
|
| 74 |
+
- **Redis Queue** - Background job processing
|
| 75 |
+
- **SQLite/PostgreSQL** - Job and experiment persistence
|
| 76 |
+
- **HuggingFace Hub** - Model and dataset hosting
|
| 77 |
+
|
| 78 |
+
## π Configuration
|
| 79 |
+
|
| 80 |
+
Set the following environment variables:
|
| 81 |
+
- `HF_TOKEN` - Your HuggingFace API token (required for pushing models)
|
| 82 |
+
- `WANDB_API_KEY` - Weights & Biases API key (optional, for experiment tracking)
|
| 83 |
+
|
| 84 |
+
## π Documentation
|
| 85 |
+
|
| 86 |
+
Full API documentation available at `/docs` when the Space is running.
|
| 87 |
+
|
| 88 |
+
## π€ Contributing
|
| 89 |
+
|
| 90 |
+
Contributions welcome! See issues for planned features.
|
| 91 |
+
|
| 92 |
+
## π License
|
| 93 |
+
|
| 94 |
+
MIT License - See LICENSE file for details.
|