Post 8: From Diagnosis to Guarded Repair

The Problem With "50/50" Merges

LoRA adapters are cheap to train, easy to share, and tempting to combine. In practice, that usually means taking two adapters — a math adapter and a code adapter, or a code adapter and a chat adapter — and merging them with a simple 50/50 linear average.

That baseline is convenient, but it fails in a predictable way. Two adapters can differ sharply in scale: one may have been trained longer, at higher rank, or on a larger or cleaner dataset. When that happens, a "50/50" merge is not really 50/50 at all. It is often just the larger adapter with the smaller one rattling around inside it.

Practitioners usually discover this only after loading the merge and running inference, when one side of the combination has effectively disappeared. By then, the GPU time is already gone.

Earlier in this series, we showed that these failures are not random. Merge difficulty is visible in the geometry of the adapters themselves: scale imbalance, overlap, and conflict leave structural fingerprints before inference begins.

This post is about the next step.

We built a recommendation engine that moves Gradience from diagnosis to intervention. It audits two adapters layer by layer, identifies where naive merging is likely to fail, and assigns a merge strategy accordingly. We then tested that engine in two stages:

1. Structural validation on public adapters — does the engine reduce domination in weight space?
2. Behavioral validation on downstream tasks — do those structural improvements translate into better model behavior?

The answer is clear.

Structurally, the engine works very well. Behaviorally, the bridge is partial. That matters because it tells us what Gradience is actually becoming: not a universal merge oracle, but a QA and preflight system for LoRA adapters and merges, with guarded intervention where structural risk is clearest.

What the Engine Actually Checks

The merge recommendation engine evaluates each corresponding layer pair along two dimensions.

1. Magnitude imbalance

The first check is the ratio of Frobenius norms between the two adapter updates at that layer. If one update is much larger than the other, equal merge coefficients will not produce equal contribution. The larger update will dominate the merged result.

When that ratio exceeds the default threshold of 5×, the engine assigns magnitude-aware coefficients:

coeff_strong = 1 / (1 + ratio)
coeff_weak   = ratio / (1 + ratio)

This suppresses the larger adapter and amplifies the smaller one. If the raw ratio is 19.7×, the coefficients are roughly 0.048 and 0.952. That looks extreme until you remember the weaker signal is almost twenty times smaller to begin with.

2. Subspace geometry

If magnitude is not the main issue, the engine evaluates geometric structure:

• overlap between the two adapter subspaces,
• conflict in signed directions,
• redundancy and related structural interactions.

From there, it chooses among a small set of merge behaviors:

• high overlap → TIES-style compression,
• high conflict → DARE-TIES-style dropout and sign election,
• low-risk / balanced layers → standard linear merge.

These two dimensions are independent. A pair can be geometrically benign and still be merge-catastrophic because one side is vastly larger than the other. In the current test set, magnitude imbalance is doing most of the practical work.

Stage 1: The Structural Result Is Strong

We first tested the engine on five pairs of publicly available LoRA adapters for Llama-2-7B, chosen to span several realistic merge regimes.

• Pair 01 — metamath-r16 × openwebmath-r16 — math × math. Same domain, same rank. Frobenius ratio 3.1× overall, with 32 of 224 layers above the 5× threshold.
• Pair 02 — metamath-r16 × magicoder-r16 — math × code. Different domains, same rank, well-balanced magnitudes. Frobenius ratio 2.1×. All 224 layers classified safe.
• Pair 03 — magicoder-r16 × btgenbot-r8 — code × chat. Different domains, different ranks. Frobenius ratio 8.4× overall, with individual layers reaching 61×. 217 of 224 layers classified imbalanced.
• Pair 04 — openwebmath-r64 × btgenbot-r8 — math × chat. Different domains, 8× rank mismatch. Frobenius ratio 19.7× overall, with individual layers reaching 875×. All 224 layers classified imbalanced.
• Pair 06 — catsubcat-r16 × btgenbot-r8 — chat × chat. Same domain, different ranks. Frobenius ratio 11.3× overall. 151 of 224 layers classified imbalanced.

Pair 05 (metamath × llm2vec) was excluded because the adapters targeted different module names and had no mergeable layer correspondence.

Each included pair was merged under two structural conditions:

• Naive: uniform 0.5 / 0.5 linear merge
• Recommended: Gradience's audit-aware, per-layer merge recommendation

We evaluated both conditions in weight space using structural retention, Q_min (worst-case structural retention), and D (dominance index). These are structural outcomes. They tell us whether the merge preserves both adapters in weight space, not whether the merged model is good on downstream tasks.

Structural Results

Taken on their own, these are strong structural results.

The clearest wins are the high-imbalance pairs: 03, 04, and 06. Under naive merging, all three are heavily dominated. The recommendation engine reverses that pattern.

Pair 04 is the clearest example. The naive merge behaves almost entirely like the larger source adapter. Under the recommended strategy, both sides are represented much more evenly, and dominance drops from 0.966 to 0.030.

Pair 03 shows the same pattern, with D falling from 0.928 to 0.002. Pair 06 is less extreme but still substantial, with D falling from 0.776 to 0.167.

Pair 02 is the control. Magnitudes are balanced, no layer exceeds the imbalance threshold, and the engine correctly leaves the pair alone. Structural outcomes are identical.

Pair 01 sits in the middle: modest overall ratio, some imbalanced layers, modest improvement.

The structural conclusion is straightforward:

Per-layer spectral auditing can identify dangerous merge asymmetries and reduce domination in weight space, especially in strongly imbalanced pairs.

That is a real result. But it is not yet the whole practical story.

Stage 2: The Behavioral Bridge Is Partial

Structural balance is not the same thing as downstream usefulness. A merge can look more balanced in weight space and still disappoint on actual inference. So we followed the structural study with a behavioral one: Study 16, an end-to-end perplexity validation.

We evaluated each pair under three merge conditions:

• Naive: uniform 0.5 / 0.5 merge
• Norm-equalized: global Frobenius-proportional reweighting, with no per-layer decision tree
• Recommended: full layerwise audit-aware merge

We also evaluated each source adapter individually and the bare base model.

Evaluation sets

• GSM8K for math
• MBPP for code
• OpenAssistant oasst2 for chat

Scoring rule

We used completion-only, token-weighted negative log-likelihood:

• prompt tokens were masked,
• only completion tokens contributed to loss,
• losses were weighted by scored token count.

That avoids the silly bug where a mountain of tiny examples distorts the whole metric.

First Problem: Some Source Adapters Are Weak

Before the merge results even matter, there is a more basic issue: several source adapters are weak on the evaluation sets.

Only metamath-r16 clearly beats the bare base model on its target benchmark. The others are roughly neutral or actively worse.

This matters because merge evaluation assumes the source adapters are worth preserving. If an adapter is already worse than base, then "preserving more of it" is not necessarily desirable. A structurally fair merge can still be behaviorally disappointing if one or both sides of the merge are simply bad candidates.

That is not a side issue. It is one of the main findings of the study.

The recommendation engine assumes both source adapters are worth carrying forward. Behavioral validation shows that this assumption does not always hold, which means source-adapter eligibility has to be screened before merge recommendation.

Three Different Kinds of "Success"

To interpret the behavioral results correctly, it helps to separate three objectives:

1. Structural balance — does the merged update retain both source adapters in weight space?
2. Worst-side preservation — does the merge prevent one adapter from being silently erased?
3. Absolute task quality — does the merged model actually perform well on downstream evaluation?

A recommendation can improve the first two and still fail on the third.

Where the Engine Helps

The strongest behavioral support appears in the most imbalanced pairs.

Pair 03 — magicoder × btgenbot

This pair shows a large structural improvement and a real behavioral tradeoff. The recommendation engine reduces domination and preserves more of the weaker side. In that sense, it succeeds.

But the downstream effect is not "better at everything." On the code side, naive and recommended are effectively similar. On the chat side, naive scores better — precisely because under naive merging the pair is behaving more like the dominant side rather than as a balanced merge.

That is not the same objective as maximizing absolute performance on one benchmark. The recommended merge redistributes influence instead of letting one side wash the other out.

Pair 04 — openwebmath-r64 × btgenbot

This is the clearest case of domination mitigation. Structurally, the naive merge is almost entirely one-sided. The recommended merge restores balance.

Behaviorally, the same tradeoff appears. The naive merge looks better on the dominant side's evaluation because it is effectively acting as a near-copy of the dominant adapter. The recommended merge gives the weaker side representation again, which necessarily reduces the stronger side's monopoly over the result.

Together, Pairs 03 and 04 support the strongest defensible behavioral claim in the study:

In high-imbalance pairs, spectral auditing can reduce catastrophic one-sided domination and improve preservation of the otherwise drowned side.

That is narrower than "general merge optimization," but it is real.

Where the Engine Does Almost Nothing

Pair 02 — metamath × magicoder

This is the control case. Magnitudes are balanced, the spectral engine finds no imbalance to correct, and all three merge strategies behave almost identically. That is exactly what we want from a conservative recommender: when there is nothing obvious to fix, it does not invent heroics.

Pair 01 — metamath × openwebmath

This pair has only a modest structural signal and correspondingly small task-level differences. A minority of layers are rebalanced, but the behavioral effect is minimal. There simply was not much to fix.

The Surprise: Norm-Equalized Is Doing a Lot

We added a third condition — global norm-equalized merging — to separate simple magnitude correction from the full layerwise decision tree.

That comparison turns out to be one of the most informative parts of the study.

• On Pair 01, the norm-equalized baseline performs better than both naive and recommended on GSM8K.
• On Pairs 03 and 04, norm-equalized and recommended are nearly identical across evaluation sets.
• On Pair 06, norm-equalized sits between naive and recommended.

This suggests that, in the current test set, much of the practical benefit comes from magnitude-aware reweighting itself, not from the full geometric branch logic.

That does not invalidate the broader recommendation engine. It clarifies what is presently validated:

The strongest demonstrated lever is norm-aware correction of imbalanced pairs.

The richer geometric branches may still matter in other adapter regimes, but this study does not yet show strong behavioral evidence for them.

Pair 06: The Boundary Case

Pair 06 is the most instructive boundary case in the study.

Structurally, the engine detects a real problem and fixes it in the way it was designed to:

• Frobenius ratio: 11.3×
• Imbalanced layers: 151 / 224
• δQ_min: +0.434
• D: 0.776 → 0.167

Behaviorally, the key lesson is more precise than our first draft.

The recommended merge is not worse than the available merge baselines. It is the best of the three tested merge conditions for this pair:

• Naive: PPL 5.437
• Norm-equalized: PPL 5.158
• Recommended: PPL 5.092
• btgenbot alone: PPL 5.474
• catsubcat alone: PPL 6.813
• Base model: PPL 4.659

So Pair 06 is not the case where a geometrically correct intervention made behavior worse. It is the case where the engine made the best of a bad situation.

The deeper problem is upstream. Both source adapters are worse than the base model on the evaluation set. That means the merge problem itself is ill-posed: the recommendation engine is being asked how to preserve two sources that are not, in deployment terms, worth preserving.

This clarifies the real limitation:

The engine can improve a merge locally while still operating on a globally bad candidate set.

In Pair 06, the engine correctly suppresses the larger and worse adapter, improves perplexity relative to naive and norm-equalized baselines, and reduces domination substantially. But the globally best decision is still not to use either adapter at all. In that sense, the optimal coefficient for this pair is not 0.5 / 0.5 or 0.102 / 0.898. It is effectively 0.0 / 0.0.

That is the real lesson of Pair 06. The recommendation engine knows how to ask:

• Is this merge imbalanced?
• Is one side being erased?
• How can I reduce domination?

It does not yet know how to ask:

• Is either source adapter worth preserving in the first place?

That missing question is not a merge-strategy problem. It is a source-eligibility problem.

The Correlations Are Weak — The Pattern Is Not

With five pairs and two eval sides each, the behavioral sample is small. We therefore treat pooled correlations as descriptive rather than inferential.

The observed rank correlations were:

• Q_min vs worst normalized retention: ρ = 0.36, bootstrap CI [−0.38, +0.70]
• D vs worst loss delta: ρ = −0.18, bootstrap CI [−0.90, +0.62]

The confidence intervals cross zero. At this sample size, that is not surprising.

The point is not that the pooled correlations are grandly conclusive. The more meaningful signal is the case pattern:

• strong help in catastrophic imbalance pairs,
• near-zero change in balanced control cases,
• and a clear demonstration that source eligibility matters before merge strategy can mean what we want it to mean.

That is why the right classification for this work is not "full validation" and not "proxy collapse."

It is partial validation.

What Actually Survives Validation

The combined structural and behavioral results support a narrower but more defensible claim than the strongest early version of the merge thesis.

What is supported

• Spectral auditing can identify structurally dangerous merges.
• The recommendation engine can mitigate one-sided domination in strongly imbalanced pairs.
• Norm-aware correction is a real and practically important intervention.
• The tool behaves conservatively in balanced cases and usually leaves low-risk pairs alone.

What is not supported

• Structural metrics do not yet justify general claims of downstream quality optimization.
• High structural balance does not guarantee high behavioral quality.
• The current recommendation engine should not be treated as a replacement for source-adapter evaluation.
• Source eligibility and merge strategy are distinct decisions; this study validates the need for the first, but does not yet solve it.

So the practical role of Gradience is now clearer:

Gradience is a structural guardrail and merge-risk tool, not a universal merge-quality oracle.

That is not a retreat into vagueness. It is a stronger and more honest product definition.

The Product Architecture This Forces

Before asking whether two adapters should be merged, we need to ask whether each adapter is worth preserving at all.

That means the validated workflow is:

1. screen source adapters first,
2. decide whether each adapter is eligible for preservation at all,
3. audit pairwise merge risk next,
4. apply guarded rebalancing to catastrophic pairs,
5. treat downstream eval as the final quality check, not something structural metrics can replace.

That makes Gradience less like a universal optimizer and more like a layered QA system:

• single-adapter audit,
• source-eligibility screening,
• merge-risk screening,
• guarded structural intervention.

That is a sturdier result to stand on, because it reflects the actual order of decisions practitioners need to make.

What Comes Next

The next phase is now clearer.

1. Source-adapter QA becomes first-class

The most immediate product need is not more merge cleverness. It is a way to decide whether an adapter should even enter the merge candidate set.

2. Norm-aware baselines now matter

The simplest validated intervention is now a serious part of the system, not a side note. Future work has to beat it honestly.

3. Compress-then-merge moves up the queue

If dead dimensions are contaminating merges, then auditing effective rank before merging may produce a cleaner workflow than trying to rescue everything at full nominal rank.

4. The geometric branches need harder tests

Overlap, conflict, and redundancy may still matter a great deal in the right adapter regimes. But they need stronger behavioral validation than this first test set provided.

Conclusion

This post began with a practical claim: naive LoRA merges fail in predictable ways, and those failures are often visible before inference.

That claim survives.

We built a recommendation engine that turns spectral diagnosis into structural intervention. On public adapters, it substantially reduces domination and improves worst-case structural retention. On downstream behavioral evaluation, the result is more constrained but still useful: the engine helps most in high-imbalance cases, behaves conservatively in balanced cases, and exposes a critical upstream problem when source adapters are not worth preserving in the first place.

So the perplexity bridge holds — but only partially.

That is enough to support guardrail language, failure-prevention language, and merge-risk screening. It is not enough to support broad claims of behavioral optimization.