| --- |
| base_model: |
| - Qwen/Qwen3-8B |
| library_name: peft |
| pipeline_tag: text-generation |
| anguage: |
| - id |
| tags: |
| - base_model:Qwen/Qwen3-8B |
| - lora |
| - sft |
| - transformers |
| - trl |
| - lm-eval |
| - bakat |
| - indonesian |
| license: apache-2.0 |
| datasets: |
| - internal-curated |
| --- |
| |
| # Bakat-8B-Base |
|
|
| ## Model Details |
|
|
| ### Model Description |
|
|
| **Bakat-8B-Base** adalah base model bahasa Indonesia yang dirancang untuk **Continued Pre-Training (CPT)** pada domain kebijakan dan pengawasan ruang digital. Model ini dibangun di atas arsitektur **Qwen3-8B**, dengan pendekatan **LoRA (Low-Rank Adaptation)** dan **4-bit quantization** untuk efisiensi memori dan komputasi. |
|
|
| * **Developed by**: Tim 1 AITF |
| * **Model type**: Causal Language Model (LoRA Adapter) |
| * **Base architecture**: Qwen3-8B |
| * **Primary language**: Indonesian (id) |
| * **License**: Apache-2.0 |
|
|
| --- |
|
|
| ## Training Data Composition |
|
|
| | Kategori | Elemen | Jumlah Token (M) | Persentase | |
| | ---------------- | ----------------------------------------------------------------------------------------------------- | ---------------- | ---------- | |
| | **DTP** | Okupasi PON TIK, Tren Pekerjaan, Kompetensi & SDM, Kebijakan & Regulasi DTP, Teknologi Digital Talent | 94 | 43.9% | |
| | **PRD** | Judi Online, Hoax, Perlindungan Anak, Konten Edukasi, Kebijakan & Regulasi PRD, Kekerasan Masyarakat | 92 | 42.9% | |
| | **Wikipedia ID** | Pengetahuan Umum Berbahasa Indonesia | 28.2 | 13.2% | |
| | **Total** | – | **214.2** | **100%** | |
|
|
| --- |
|
|
| ## Intended Use |
|
|
| ### Direct Use (Recommended) |
|
|
| Model ini **ditujukan untuk Continued Pre-Training**, khususnya untuk: |
|
|
| * Adaptasi domain kebijakan publik dan regulasi digital |
| * Pengayaan pengetahuan spesifik Indonesia |
| * Pre-adaptation sebelum Instruction Tuning atau SFT |
|
|
| ### Out-of-Scope Use |
|
|
| * **Long-context conversations** (belum dioptimalkan) |
| * **High-stakes decision making** (legal, medis, finansial) |
| * **Chat-oriented instruction following** tanpa fine-tuning lanjutan |
|
|
| --- |
|
|
| ## Bias, Risks, and Limitations |
|
|
| * Dataset didominasi oleh domain kebijakan dan pengawasan ruang digital, sehingga bias topikal dapat muncul pada domain non-terkait. |
| * Model belum melalui tahap preference alignment (RLHF/DPO). |
| * Konten Wikipedia digunakan sebagai penyeimbang, namun tidak menjamin netralitas penuh. |
|
|
| Pengguna disarankan melakukan evaluasi tambahan sebelum penggunaan produksi. |
|
|
| --- |
|
|
| ## Recommendations |
|
|
| * Gunakan **Qwen3 chat template** untuk hasil generasi terbaik. |
| * Lakukan **Instruction Fine-Tuning** atau **Preference Tuning** sebelum deployment ke end-user. |
| * Verifikasi keluaran model untuk informasi kritikal. |
|
|
| --- |
|
|
| ## How to Get Started |
|
|
| Load the model using **HuggingFace Transformers**: |
|
|
| ```python |
| import torch |
| from transformers import AutoTokenizer, AutoModelForCausalLM |
| |
| # 1. Configuration |
| model_id = "aitfindonesia/Bakat-8B-Base" # Replace with your actual Hub ID |
| |
| # 2. Load Model |
| # Use bfloat16 for A100/A10G, float16 for T4 |
| tokenizer = AutoTokenizer.from_pretrained(model_id) |
| model = AutoModelForCausalLM.from_pretrained( |
| model_id, |
| torch_dtype=torch.bfloat16, |
| device_map="auto" |
| ) |
| |
| # 3. Inference Example (Completion) |
| input_text = "Strategi utama untuk mengurangi gap talenta digital di Indonesia adalah" |
| inputs = tokenizer(input_text, return_tensors="pt").to("cuda") |
| |
| with torch.no_grad(): |
| outputs = model.generate( |
| **inputs, |
| max_new_tokens=100, |
| do_sample=True, |
| temperature=0.7 |
| ) |
| print(tokenizer.decode(outputs[0], skip_special_tokens=True)) |
| ``` |
|
|
| --- |
|
|
| ## Training Details |
|
|
| ### Training Data |
|
|
| * **Total size**: ~214M tokens |
| * **Domains**: Digital Talent Policy (DTP), Pengawasan Ruang Digital (PRD), Wikipedia Indonesia |
| * **Split**: Train (90%) / Validation (10%) |
|
|
| ### Training Procedure |
|
|
| Model dilatih menggunakan **Continued Pre-Training (CPT)** dengan LoRA pada HuggingFace Transformers. |
|
|
| #### Hyperparameters |
|
|
| * **Precision**: bf16 (mixed precision) |
| * **Quantization**: 4-bit (nf4) |
| * **LoRA Rank (r)**: 8 |
| * **LoRA Alpha**: 16 |
| * **Target modules**: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj |
| * **Batch size**: 4 / device |
| * **Gradient accumulation**: 16 (effective batch size = 32) |
| * **Learning rate**: 2e-4 (linear schedule) |
| * **Warmup ratio**: 0.03 |
| * **Epochs**: 1 |
| * **Optimizer**: adamw_8bit |
|
|
| --- |
|
|
| ## Evaluation |
|
|
| ### Results |
|
|
| * **Final Training Loss**: ~1.2685 |
| * **Final Validation Loss**: ~1.264 |
| * **Training Perplexity**: ~3.56 |
| * **Validation Perplexity**: ~3.55 |
|
|
| ### Benchmark (General) |
|
|
| * **MMLU**: ~74.20 |
| * **IndoMMLU**: ~65.66 |
| * **XCOPA-ID**: ~75.80 |
|
|
| --- |
|
|
| ## Environmental Impact |
|
|
| Estimasi emisi karbon mengikuti metodologi Lacoste et al. (2019). |
|
|
| * **Hardware**: NVIDIA A100 80GB |
| * **Training time**: ~36 jam |
| * **Compute region**: Indonesia |
| * **Infrastructure**: University / Private Server |
|
|
| --- |
|
|
| ## Framework Versions |
|
|
| * Transformers: 4.x |
| * PyTorch: 2.x |
| * Datasets: 2.x |
| * Tokenizers: 0.x |
