LoRA Redundancy: A Problem Of The Past?(clickbait?)
The Problem
We all know that LoRA training ends with an echo chamber. You pass your corpus into the SFT, and the layers of the model you're fine-tuning come out with every layer learning to answer the prompt the same way. We throw more parameters at the problem, larger datasets. You can freeze layers and risk the reintroduction blowing up in your face. So how can we combat the redundancy issue? Entropy settles and finds the easiest way to answer naturally. It's a byproduct of high-dimensional optimization.
An Abstract Connection
I recently had the observation that my own high entropy speech patterns are unpredictable catalysts for driving models to hallucinations and opening vulnerabilities with frontier models. I don't try to open models up to weakness or hallucination, but inevitably, I end up correcting models and pointing out the drifting behaviour patterns to bring them back to center. I use shortened sentences that can be taken in multiple ways, I'll think out loud in my prompt, and sometimes I will copy and paste conversations I'm having with other models into the window, asking a coding model for advice on a topic I'm discussing with a context model. I treat the conversations like a casual chat between colleagues and often praise models for helping me solve a problem. This pattern is difficult to predict, and it brings different personality traits out that I can't predict either. This isn't a brag or a jailbreak walk-through; it was a realization that unpredictability makes unique outcomes with all models. So I decided to try to apply it to the redundancy problem.
I asked myself, what if I introduced unpredictability and chaos into fine-tuning? So I came up with chaos injection. Instead of adding to my training data and tainting the personality I was trying to introduce into the model, I made the training itself structurally unpredictable. I placed small neural networks at strategic points between the model's transformer layers, using them to introduce entropy into the forward pass, forcing the layers around them to develop their own specialized responses.
Chaos in Practice
With a Llama 3.2 3B Instruct model, there are 28 layers to consider. The first and final 2 layers handle initial prompt understanding and response translation. The middle layers are where the model makes its decisions, where personality lives. A 3B model is often underestimated, but with chaos injection, we can create a model that reconsiders its own decisions multiple times during a single forward pass. I think of it like folding the model back on itself and forcing it to challenge its own reasoning. A model that questions its own actions can be more powerful in both reasoning and personality. More internal deliberation enriches the output and creates less predictable, more genuine responses. The result was models with orthogonally unique layers that don't share semantic similarities, each genuinely processing the information being passed forward rather than echoing the same answer from different positions.
The full method is still under research and development. I wanted to share this early to ask the community what solutions they've tested to bring more unique personalities to your models!