| --- |
| language: |
| - ug |
| license: mit |
| library_name: kenlm |
| tags: |
| - uyghur |
| - input-method |
| - nlp |
| - character-level |
| datasets: |
| - custom |
| metrics: |
| - perplexity |
| --- |
| # Uyghur-Character-Level-KenLM-Input-Method (سىناق نۇسخىسى) |
|
|
|
|
|
|
| An intelligent input prediction engine specifically designed for the **Uyghur language**. It combines traditional corpus-based prefix searching with high-performance **KenLM (N-gram)** language models to achieve real-time mapping from Latin characters to Uyghur script with probabilistic ranking. |
|
|
| <p dir="rtl" align="right">ئۇيغۇرتىلى ئۈچۈن مەخسۇس لايىھەلەنگەن ئەقلىي كىرگۈزۈش سىناق تۈرى</p> |
|
|
| --- |
|
|
| ## 🌟 Why Character-Level? |
|
|
| Uyghur is a highly **Agglutinative Language**. A single word root can produce dozens of different forms through the addition of suffixes. |
|
|
| - **Example**: |
| - `مەك-تەپ` (School) |
| - `مەك-تەپ-لى-رى-مىز` (Our schools) |
|
|
| Traditional word-level N-grams often suffer from data sparsity and poor generalization in Uyghur. This project utilizes a **Character-level n-gram** approach: |
| - **Root & Suffix Learning**: Automatically learns the relationships between word roots and various suffixes. |
| - **Superior Generalization**: Handles "Out-of-Vocabulary" (OOV) words more effectively than word-level models. |
| - **Stability**: Provides more reliable completion results for the unique phonetic and morphological structure of Uyghur. |
| --- |
|
|
| ## 🧠 System Architecture |
|
|
| The project functions as a **Ranking/Scoring Engine** based on the formula: |
| $$P(candidate | context)$$ |
|
|
| 1. **Candidate Generation**: The system retrieves possible words from the 14M+ word dictionary (`dict.txt`) based on the user's input prefix. |
| 2. **KenLM Scoring**: The Character-level KenLM (`char.bin`) acts as the "Brain," scoring each candidate based on linguistic probability. |
| 3. **Sorting**: The most probable candidates are delivered to the user interface. |
| --- |
|
|
| ## 📂 Repository Contents |
|
|
| | File | Description | |
| | :---------- | :----------------------------------------------------------- | |
| | `char.bin` | **Core Model**: 407MB Binary KenLM model. | |
| | `dict.txt` | **Dictionary**: Massive corpus containing 14,416,068 Uyghur entries. | |
| | `server.py` | **Linux Server**: Flask API for remote scoring and prediction. | |
| | `main.py` | **Windows Client**: Desktop overlay for real-time typing. | |
| | `test.py` | **Testing Script**: CLI script to verify candidate scoring. | |
|
|
| --- |
|
|
| ## 📊 Performance & Case Studies |
|
|
| ### CLI Prediction Test |
| When a user types a prefix, the engine generates scored candidates instantly: |
|
|
| **Input: `مە`** |
| - مەكتەپتىكى (Score: -6.69) |
| - مەدەنىيلىكنىڭ (Score: -7.67) |
| - مەنپەئەتى (Score: -7.70) |
|
|
| **Input: `مەر`** |
| - مەردان (Score: -6.27) |
| - مەرھابانىڭ (Score: -7.18) |
| - مەركىزىدىكى (Score: -8.05) |
|
|
| --- |
|
|
| ## 🖼️ Input Method Preview (Screenshots) |
|
|
| Below are the test cases for the input method interface. It supports seamless rendering of RTL (Right-to-Left) Uyghur script. |
|
|
| <div align="center"> |
| <img src="img/test-1.jpg" height="100px" alt="Test 1"> |
| <img src="img/test-2.jpg" height="100px" alt="Test 2"> |
| <img src="img/test-3.jpg" height="100px" alt="Test 3"> |
| </div> |
| --- |
|
|
| ## 🚀 Deployment |
|
|
| ### 1. Backend (Linux Server) |
| The heavy-lifting scoring is done on Linux: |
| ```bash |
| pip install flask (https://github.com/kpu/kenlm/archive/master.zip) |
| python3 server.py |
| ``` |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
