SandLogicTechnologies/Phi-3-mini-4k-instruct-GGUF

Phi-3-Mini-4K-Instruct

Phi-3-Mini-4K-Instruct is a lightweight yet highly capable instruction-tuned large language model developed by Microsoft as part of the Phi-3 family. Designed for conversational AI, reasoning, and instruction-following tasks, the model supports up to 4K tokens of context. To enable efficient local and resource-constrained deployment, the model is provided in GGUF quantized formats, where Q4_K_M and Q5_K_M quantization reduce numerical precision from full precision to 4-bit and 5-bit representations. This significantly lowers memory usage and improves inference speed on CPUs and consumer-grade GPUs, while largely preserving the model’s response quality and reasoning ability.

---

Model Overview

• Model Name: Phi-3-Mini-4K-Instruct
• Base Model: microsoft/Phi-3-mini-4k-instruct
• Architecture: Decoder-only Transformer
• Parameters: ~3.8 Billion
• Context Length: 4K tokens
• Quantized Versions:
  • Q4_K_M (4-bit quantization)
  • Q5_K_M (5-bit quantization)
• Modalities: Text only
• Developer: Microsoft
• License: MIT

---

Quantization Details

Q4_K_M

• Approx. ~71% size reduction
• Very low memory footprint (2.23 GB)
• Optimized for CPU inference and low-VRAM environments
• Faster inference speeds
• Slight degradation in complex or multi-step reasoning tasks

Q5_K_M

• Approx. ~64% size reduction
• Better fidelity to the original FP16 model (2.64 GB)
• Improved coherence and reasoning consistency
• Recommended when slightly more memory is available

---

Training Details (Original Model)

Phi-3-Mini-4K-Instruct is trained using a multi-stage pipeline focused on high-quality reasoning and instruction alignment, optimized for strong performance at small scale.

---

Pretraining

• Trained on a curated mixture of high-quality publicly available data.
• Emphasis on reasoning-centric content, including:
  • Mathematics
  • Logic
  • Code
  • Scientific and technical text
• Uses autoregressive language modeling as the primary training objective.
• Designed to maximize reasoning efficiency per parameter.

Instruction Fine-Tuning

• Fine-tuned on diverse supervised instruction datasets.
• Further aligned using preference optimization techniques.
• Improves:
  • Instruction-following accuracy
  • Safety and response helpfulness
  • Multi-turn conversational coherence

---

Key Features

• Instruction-tuned chat model
  Designed to follow user instructions accurately and generate helpful, aligned responses.

• Compact and efficient
  Strong reasoning performance with a small parameter count, suitable for local and edge deployment.

• Strong reasoning capabilities
  Performs well on logical, mathematical, and analytical tasks relative to its size.

• Multi-turn dialogue support
  Maintains context across short-to-medium conversations.

• Efficient inference via GGUF
  Quantized GGUF formats enable fast, low-memory inference on CPUs and consumer GPUs.

• Safe and aligned outputs
  Fine-tuned to reduce harmful, misleading, or unsafe responses.

---

Usage

llama.cpp

./llama-cli \
  -m SandLogicTechnologies/Phi-3-mini-4k-instruct_Q4_K_M.gguf \
  -p "Explain transformers in simple terms."

Recommended Use Cases

• Local AI assistants Run lightweight, offline chat assistants on personal machines.

-Reasoning and Q&A tasks Perform logical analysis, explanations, and structured problem-solving.

• Developer tools Integrate into coding helpers, documentation assistants, or CLI tools.

• Edge and CPU inference Ideal for laptops, desktops, and low-resource environments.

• Privacy-preserving applications Keep inference fully local with no external data transmission.

Acknowledgments

These quantized models are based on the original work by Microsoft development team.

Special thanks to:

• The Microsoft team for developing and releasing the Phi-3-mini-4k-instruct model.

• Georgi Gerganov and the entire llama.cpp open-source community for enabling efficient model quantization and inference via the GGUF format.

---

Contact

For any inquiries or support, please contact us at support@sandlogic.com or visit our Website. ```