kappatune-lora-tinyllama-agnews

This model is a fine-tuned version of TinyLlama/TinyLlama-1.1B-Chat-v1.0 on the AG News dataset. It was trained to demonstrate the KappaTune fine-tuning methodology, which selectively adapts neural network layers based on their condition number to mitigate catastrophic forgetting.

Model description

This model is a LoRA (Low-Rank Adaptation) adapter applied to TinyLlama-1.1B. Unlike standard LoRA, which targets manually specified module types (e.g., all q_proj or v_proj layers), this adapter was trained using the KappaTune PEFT Integration. Before training, the KappaTune algorithm performed a Singular Value Decomposition (SVD) on all candidate weight matrices to calculate their Condition Number (kappa).

• High-kappa tensors (highly specialized, anisotropic weights containing pre-trained knowledge) were frozen.
• Low-kappa tensors (numerically stable, general-purpose weights of higher output entropy acting as a "raw marble block") were selected for LoRA adaptation.

This targeted approach ensures the model learns the new domain efficiently while preserving its foundational conversational capabilities.

• Paper: The Condition Number as a Scale-Invariant Proxy for Information Encoding in Neural Units (arXiv:2506.16289)
• Repository: oswaldoludwig/kappaTune

Intended uses & limitations

• Methodological Proof of Concept: This model serves primarily as a demonstration of the KappaTune automated layer-selection strategy.
• Domain Adaptation: It illustrates how a general-purpose chat model can be steered toward a specific formatting and vocabulary style (News Reporting) using a tiny fraction of trainable parameters.

Training and evaluation data

The model was fine-tuned on the AG News dataset, a curated collection of news articles categorized into four topics: World, Sports, Business, and Sci/Tech.

Training procedure

The model was trained using the Hugging Face Trainer API combined with the kappaTune optimizer wrapper. Although AG News is traditionally a classification benchmark, the data was formatted as a continuous text stream for this experiment. The model was trained on a Causal Language Modeling task (next-token prediction) to adapt its generation style to news reporting, rather than predicting a categorical label.

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-05
• train_batch_size: 4
• eval_batch_size: 8
• seed: 42
• optimizer: Use adamw_torch with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: linear
• num_epochs: 3

Training results

Framework versions

• PEFT 0.17.1
• Transformers 4.55.4
• Pytorch 2.6.0+cu124
• Datasets 4.1.1
• Tokenizers 0.21.4