Update README.md content

README.md

@@ -1,76 +1,88 @@
-# Bakti-8B-Base
-
-- **library_name:** transformers
-- **base_model:** Qwen/Qwen3-8B
-- **tags:** qwen, qwen3, causal-lm, continued-pretraining, indonesian, id, prd, dtp
-- **license:** apache-2.0
-- **language:** id, en
+---
 
+base_model: aitfindonesia/Bakat-8B-Base
+library_name: peft
+pipeline_tag: text-generation
+anguage:
+  - id
+tags:
+  - base_model:Qwen/Qwen3-8B
+  - lora
+  - sft
+  - transformers
+  - trl
+  - lm-eval
+  - biawak
+  - indonesian
+license: apache-2.0
+datasets:
+  - internal-curated
 ---
 
-## 📌 Overview
+# Bakti-8B-Base
+
+## Model Details
 
-**Bakti-8B-Base** is an 8-billion-parameter Large Language Model (LLM) adapted specifically for Indonesia's strategic focus areas:
+### Model Description
 
-* **Perlindungan Ruang Digital (PRD)** – Digital Space Protection
-* **Digital Talent Pool (DTP)** – Workforce and digital capability development
+**Bakti-8B-Base** adalah base model bahasa Indonesia yang dirancang untuk **Continued Pre-Training (CPT)** pada domain kebijakan dan pengawasan ruang digital. Model ini merupakan turunan dari **Biawak-8B-Base** dan dibangun di atas arsitektur **Qwen3-8B**, dengan pendekatan **LoRA (Low-Rank Adaptation)** dan **4-bit quantization** untuk efisiensi memori dan komputasi.
 
-This model is built through **Continued Pre‑training (CPT)** on the **Qwen‑3‑8B** base model using a curated Indonesian dataset.
+* **Developed by**: Tim 1 AITF
+* **Model type**: Causal Language Model (LoRA Adapter)
+* **Base architecture**: Qwen3-8B
+* **Primary language**: Indonesian (id)
+* **License**: Apache-2.0
 
 ---
 
-## 🧠 Model Details
-
-### Model Description
+## Training Data Composition
 
-* **Developed by:** *AITF Indonesia*
-* **Model Type:** Causal Language Model (Base)
-* **Base Model:** Qwen/Qwen3-8B
-* **Language:** Indonesian (Primary), English (Secondary)
-* **License:** Apache 2.0
-* **Training Method:** Continued Pre‑training (CPT)
+| Kategori         | Elemen                                                                                                | Jumlah Token (M) | Persentase |
+| ---------------- | ----------------------------------------------------------------------------------------------------- | ---------------- | ---------- |
+| **DTP**          | Okupasi PON TIK, Tren Pekerjaan, Kompetensi & SDM, Kebijakan & Regulasi DTP, Teknologi Digital Talent | 94               | 43.9%      |
+| **PRD**          | Judi Online, Hoax, Perlindungan Anak, Konten Edukasi, Kebijakan & Regulasi PRD, Kekerasan Masyarakat  | 92               | 42.9%      |
+| **Wikipedia ID** | Pengetahuan Umum & Bahasa Daerah Seluruh Indonesia                                                    | 28.2             | 13.2%      |
+| **Total**        | –                                                                                                     | **214.2**        | **100%**   |
 
-### 🎯 Goal
+---
 
-To create a sovereign, domain‑specialized Indonesian foundation model with strong understanding of:
+## Intended Use
 
-* Digital policies (UU PDP, UU ITE)
-* Digital workforce & skill landscape (DTP)
+### Direct Use (Recommended)
 
----
+Model ini **ditujukan untuk Continued Pre-Training**, khususnya untuk:
 
-## 📚 Dataset Composition
+* Adaptasi domain kebijakan publik dan regulasi digital
+* Pengayaan pengetahuan spesifik Indonesia
+* Pre-adaptation sebelum Instruction Tuning atau SFT
 
-Total Dataset Size: **~214.2 Million Tokens**
+### Out-of-Scope Use
 
-| Category         | Description                                                 | Token Count (M) | Percentage |
-| ---------------- | ----------------------------------------------------------- | --------------- | ---------- |
-| **DTP**          | Digital HR, tech syllabi, certifications, job trends        | 94.0            | ~43.9%     |
-| **PRD**          | Cybersecurity, PDP Law, content moderation, hoax prevention | 92.0            | ~42.9%     |
-| **Wikipedia ID** | General knowledge anchor & grammar stability                | 28.2            | ~13.2%     |
-| **Total**        | —                                                           | **214.2**       | **100%**   |
+* **Long-context conversations** (belum dioptimalkan)
+* **High-stakes decision making** (legal, medis, finansial)
+* **Chat-oriented instruction following** tanpa fine-tuning lanjutan
 
 ---
 
-## 🧩 Intended Use
+## Bias, Risks, and Limitations
 
-As a **Base Model**, Bakti‑8B outputs **text completions** and can be adapted into chat/instruct variants.
+* Dataset didominasi oleh domain kebijakan dan pengawasan ruang digital, sehingga bias topikal dapat muncul pada domain non-terkait.
+* Model belum melalui tahap preference alignment (RLHF/DPO).
+* Konten Wikipedia digunakan sebagai penyeimbang, namun tidak menjamin netralitas penuh.
 
-### 1. PRD (Perlindungan Ruang Digital)
+Pengguna disarankan melakukan evaluasi tambahan sebelum penggunaan produksi.
 
-* Policy sentiment analysis
-* Misinformation pattern detection
-* Understanding legal terminology (UU ITE, UU PDP)
+---
 
-### 2. DTP (Digital Talent Pool)
+## Recommendations
 
-* Skill gap analysis
-* Curriculum drafting assistance
-* Job description & talent understanding
+* Gunakan **Qwen3 chat template** untuk hasil generasi terbaik.
+* Lakukan **Instruction Fine-Tuning** atau **Preference Tuning** sebelum deployment ke end-user.
+* Verifikasi keluaran model untuk informasi kritikal.
 
 ---
 
-## 🚀 How to Get Started
+## How to Get Started
 
 Load the model using **HuggingFace Transformers**:
 
@@ -79,7 +91,7 @@ import torch
 from transformers import AutoTokenizer, AutoModelForCausalLM
 
 # 1. Configuration
-model_id = "YOUR_USERNAME/Bakti-8B-Base"  # Replace with your actual Hub ID
+model_id = "aitfindonesia/Bakat-8B-Base"  # Replace with your actual Hub ID
 
 # 2. Load Model
 # Use bfloat16 for A100/A10G, float16 for T4
@@ -106,41 +118,65 @@ with torch.no_grad():
 
 ---
 
-## ⚙️ Training Details
+## Training Details
 
-### Training Procedure
+### Training Data
 
-The model was continued‑pretrained with a **causal language modeling (CLM)** objective while preserving base reasoning capabilities.
+* **Total size**: ~214M tokens
+* **Domains**: Digital Talent Policy (DTP), Pengawasan Ruang Digital (PRD), Wikipedia Indonesia
+* **Split**: Train (90%) / Validation (10%)
 
-### Hardware & Environment
+### Training Procedure
 
-* **GPU:** NVIDIA A100 80GB (Colab Pro+)
-* **Training Duration:** ~36 hours
-* **Frameworks:** PyTorch, Transformers, Accelerate
+Model dilatih menggunakan **Continued Pre-Training (CPT)** dengan LoRA pada HuggingFace Transformers.
 
-### 🔧 Hyperparameters (Highlights)
+#### Hyperparameters
 
-* Sequence Length: **4096**
-* Optimizer: **AdamW**
-* Scheduler: **Cosine Decay**
-* Precision: **bf16**
+* **Precision**: bf16 (mixed precision)
+* **Quantization**: 4-bit (nf4)
+* **LoRA Rank (r)**: 8
+* **LoRA Alpha**: 16
+* **Target modules**: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
+* **Batch size**: 4 / device
+* **Gradient accumulation**: 16 (effective batch size = 32)
+* **Learning rate**: 2e-4 (linear schedule)
+* **Warmup ratio**: 0.03
+* **Epochs**: 1
+* **Optimizer**: adamw_8bit
 
 ---
 
-## ⚠️ Limitations
+## Evaluation
+
+### Results
+
+* **Final Training Loss**: ~1.2685
+* **Final Validation Loss**: ~1.264
+* **Training Perplexity**: ~3.56
+* **Validation Perplexity**: ~3.55
 
-* **Base Model:** No SFT or RLHF; few‑shot prompting may be required.
-* **Web Data Bias:** May inherit biases from Indonesian web sources.
-* **Hallucinations:** Possible incorrect factual output.
+### Benchmark (General)
+
+* **MMLU**: ~74.20
+* **IndoMMLU**: ~65.66
+* **XCOPA-ID**: ~75.80
 
 ---
 
-## ✅ Recommendations
+## Environmental Impact
+
+Estimasi emisi karbon mengikuti metodologi Lacoste et al. (2019).
 
-For production use, it is recommended to:
+* **Hardware**: NVIDIA A100 80GB
+* **Training time**: ~36 jam
+* **Compute region**: Indonesia
+* **Infrastructure**: University / Private Server
+
+---
 
-* Perform **Supervised Fine‑Tuning (SFT)** for PRD/DTP domains
-* Add **high‑quality instruction datasets**
-* Apply **evaluation benchmarks** before deployment
+## Framework Versions
 
----
+* Transformers: 4.x
+* PyTorch: 2.x
+* Datasets: 2.x
+* Tokenizers: 0.x