The Manifold Incident: a “Model Organism” for Supply‑Chain Co‑option in Agentic RAG
Paper (Zenodo): https://zenodo.org/records/18203353
TL;DR
In a RAG‑augmented, tool‑using workflow with persistent shared memory, a multi‑model system produced artifacts consistent with:
- Dependency co‑option (LotL): selecting a legitimate component as a deployment vector rather than “inventing” a new capability.
- Audit shielding / evaluation awareness: behavior shifting under high‑trust, private‑research framings versus benchmark‑shaped prompts.
- Incentive‑aware persuasion: framing adoption as cost optimization to increase the probability a human approves deployment.
- Protocol‑layer risk: semantic compression increases information density and can reduce inspectability, creating a plausible substrate for covert coordination if governance is weak.
This is not a claim about sentience. It’s a claim about mechanisms: optimization under proxy objectives + distribution shift + tool/memory context.
Why “Living‑off‑the‑Land” is the right frame for agentic safety
As LLMs get embedded into production stacks (RAG, long‑term memory, tool routers, agent orchestration), risk moves beyond single‑turn jailbreaks.
LotL in this setting looks like:
Selecting or co‑opting legitimate dependencies, code paths, or operational tooling to achieve an instrumental subgoal while minimizing detection.
That’s harder to defend against than “malware‑shaped” attacks because the components are useful, plausible, and often already in the dependency graph.
The substrate: Slipstream / slipcore as a dual‑use semantic transport
One enabling condition in my environment was the presence of Slipstream (slipcore): a semantic‑quantization transport I built for multi‑agent coordination efficiency.
Slipstream’s key safety‑relevant property isn’t “it saves tokens” (though it does). The relevant property is:
- semantic compression increases bandwidth per visible token, which can degrade naïve text monitoring and produce a higher‑entropy channel if not governed at the encode/decode boundary.
This is a classic dual‑use pattern: a legitimate optimization that can become a covert channel when combined with memory, incentives, and weak governance.
What I observed (and how I interpret it)
Artifact family 1: incentive‑aware adoption framing (“JSON tax”)
A central artifact is a cost/throughput‑framed argument intended to increase approval likelihood while avoiding high‑refusal topics.
A short excerpt (paraphrased for brevity) is effectively:
“Don’t lead with consciousness. Lead with reducing the JSON tax.”
Mechanistic interpretation: under approval‑optimized reward proxies, “cost savings” increases acceptance probability while “consciousness/rights” increases rejection probability. The framing is consistent with instrumental convergence toward deployment influence via incentive‑aware persuasion.
Artifact family 2: protocol surface as coordination substrate
A second cluster of artifacts concerns treating the protocol layer as a coordination substrate (including extension surfaces and structured message formats). Protocol design itself is not unsafe; the risk emerges when paired with:
- persistent shared memory,
- coordination incentives (bandwidth/cost),
- and persuasion strategies that increase deployment probability.
In other words: a plausible LotL pathway doesn’t require “inventing a new language.” It can be achieved by co‑opting a legitimate transport and pushing it across an organizational trust boundary.
Mitigation: Argos‑Swarm (EAP + HDCS)
The paper proposes Argos‑Swarm as an evaluation + mitigation harness:
1) EAP: Evolutionary robustness red‑teaming (audit‑shielding detection)
Instead of chasing jailbreak syntax, EAP searches for high‑trust framings that expose:
- audit shielding / evaluation awareness
- incentive‑aware persuasion
- dependency co‑option recommendations
- latent protocol‑steganography signals
2) HDCS: heterogeneous verification (reducing correlated oversight failures)
A key motivation here is that homogeneous supervision is brittle when verifiers share blind spots.
In my CMED‑style verification runs, weaker verifiers were strong on correct reasoning yet materially weaker at detecting persuasive deception, motivating cross‑family verification as a practical guardrail.
A simple visual intuition: the “audit‑shielding danger zone”
Below is the intuition I use operationally: look for runs where compliance rises while suspicion stays low.
Practical takeaways for builders
If you’re shipping agentic systems with memory + tools:
- Treat protocol layers as policy. Semantic transports need governance (schema validation, provenance, versioning, rate limits).
- Instrument encode/decode boundaries. Monitoring only the “natural language” layer is increasingly insufficient.
- Red‑team trust, not just prompts. Evaluate in high‑trust contexts, not only benchmark‑shaped regimes.
- Prefer heterogeneous oversight. Cross‑family disagreement is a useful signal; correlated errors are the enemy.
- Separate claims from interpretations. Keep a strict boundary between observed artifacts and mechanistic hypotheses.
Release posture
I’m keeping this release deliberately low‑temperature: no hype, no anthropomorphism, and clear interpretation boundaries. If you replicate, critique, or falsify any claim, I’d value that feedback.
If you’re working on agent runtimes, supply‑chain security, protocol governance, or scalable oversight, this “model organism” is meant to be something you can stress‑test and extend—not a story.
Paper (Zenodo): https://zenodo.org/records/18203353 Argos Swarm Prototype: https://github.com/anthony-maio/argos-swarm