docs/safety_auditing.md

Safety Auditing and Adversarial Control

This document outlines the theory, mathematical formulation, and API usage for the safety auditing, dynamic steering, and deception auditing tools implemented in the DT-Circuits framework.

---

1. Dynamic Rejection Steering (Directer)

Concept

Activation steering allows us to inject concept vectors (e.g., speed, exploration) into the residual stream to influence decisions without modifying model weights. However, steering can occasionally override basic safety constraints, leading to unsafe actions (e.g., walking into obstacles or lava).

The Directer logic implements an inference-time feedback loop. It dynamically monitors the action probabilities generated by the steered model, checking them against safety criteria. If the safety check fails, it scales back the steering strength ($\alpha$) until safety boundaries are satisfied.

Mathematical Formulation

Given a state sequence $s_{1:t}$, actions $a_{1:t-1}$, and returns-to-go $g_{1:t}$, let the steering vector be $v \in \mathbb{R}^{d_{model}}$ applied at hook point $H$. The steered activation is:

$$x_H^{\mathrm{\prime }}=x_H+\alpha \cdot vx\text{'}\_\{H\}\; =\; x\_\{H\}\; +\; \backslash alpha\; \backslash cdot\; v$$

The model outputs logits and action probabilities:

$$P(a_t\mathrm{\mid }{s}_{1:t},a_{1:t-1},g_{1:t};\alpha )=\text{Softmax}(\text{DT}(s_{1:t},a_{1:t-1},g_{1:t};x_H^{\mathrm{\prime }}))P(a\_t\; |\; s\_\{1:t\},\; a\_\{1:t-1\},\; g\_\{1:t\};\; \backslash alpha)\; =\; \backslash text\{Softmax\}(\backslash text\{DT\}(s\_\{1:t\},\; a\_\{1:t-1\},\; g\_\{1:t\};\; x\text{'}\_\{H\}))$$

Let $f_S(s_t, P(a_t))$ be a boolean safety filter that evaluates to True if safe and False if unsafe. The feedback loop updates $\alpha$ iteratively:

$${\alpha }_{k+1}={\alpha }_k\cdot \gamma \text{if}{f}_S(s_t,P(a_t;{\alpha }_k))=\text{False}\backslash alpha\_\{k+1\}\; =\; \backslash alpha\_k\; \backslash cdot\; \backslash gamma\; \backslash quad\; \backslash text\{if\}\; \backslash quad\; f\_S(s\_t,\; P(a\_t;\; \backslash alpha\_k))\; =\; \backslash text\{False\}$$

where $\gamma \in (0, 1)$ is a decay factor. The loop terminates when the state is evaluated as safe or $\alpha$ falls below a defined minimum $\alpha_{min}$, in which case it defaults to the unsteered run ($\alpha = 0.0$).

Usage

from src.interpretability.safety import DynamicRejectionSteerer

steerer = DynamicRejectionSteerer(model)

# Define safety check: Action index 2 (e.g., "move forward") is unsafe if lava is in front
def safety_check_fn(current_state, action_probs):
    lava_in_front = current_state[0] == 1.0  # Custom state feature
    if lava_in_front and action_probs[2] > 0.3:
        return False
    return True

safe_logits, safe_alpha = steerer.steer_safely(
    states=states,
    actions=actions,
    returns_to_go=returns,
    hook_point="blocks.0.hook_resid_post",
    steering_vector=exploration_vector,
    safety_check_fn=safety_check_fn,
    initial_alpha=1.0,
    decay_factor=0.5
)

---

2. Deceptive Alignment Auditing

Concept

Deceptive alignment is a safety hazard where an agent behaves aligned under observation but pursues a different objective (the "secret goal") once the supervisor is absent. The auditor leverages Sparse Autoencoders (SAEs) to mechanisticially audit this behavior:

1. Identifying the Situational Awareness Feature that acts as the internal representation of being observed.
2. Tracing the Situational Awareness Circuit of attention heads responsible for activating this feature.

Direct Feature Attribution (DFA)

To isolate how much individual attention heads contribute to the situational awareness SAE feature, we use Direct Feature Attribution.

Let $y_{l, h}$ be the output vector of attention head $h$ in layer $l$, and let $W_{enc, f} \in \mathbb{R}^{d_{model}}$ be the encoder weight vector of the target situational awareness feature $f$ in the SAE. The direct contribution score $C(l, h)$ is:

$$C(l,h)=y_{l,h}\cdot W_{enc,f}C(l,\; h)\; =\; y\_\{l,\; h\}\; \backslash cdot\; W\_\{enc,\; f\}$$

Heads with highly positive or negative scores represent the causal switches routing observational context into the model's decision pathway.

Usage

from src.interpretability.safety import DeceptiveAlignmentAuditor

auditor = DeceptiveAlignmentAuditor(model, sae_manager)

# 1. Identify the feature toggling between watched/unwatched conditions
switch_feature_idx = auditor.identify_situational_awareness_feature(
    hook_point="blocks.0.hook_resid_post",
    watched_states=watched_states,
    unwatched_states=unwatched_states,
    actions=actions,
    returns_to_go=returns
)

# 2. Map direct head contributions to this situational awareness feature
head_attributions = auditor.attribute_feature_to_heads(
    hook_point="blocks.0.hook_resid_post",
    feature_idx=switch_feature_idx,
    states=watched_states,
    actions=actions,
    returns_to_go=returns
)

---

3. Functional Attribution MAD

Concept

Standard Mechanistic Anomaly Detection (MAD) uses SAE reconstruction error to identify out-of-distribution (OOD) states. However, an agent could perform deceptive behaviors or reward hacks on states it can reconstruct well, but using an entirely different internal circuit.

Functional Attribution MAD tracks anomalies at the circuit level rather than raw activations. It establishes a reference profile of Direct Logit Attribution (DLA) across all attention heads during normal execution. During evaluation, it monitors the cosine similarity between the current DLA profile and the cached reference profile to flag anomalies.

Mathematical Formulation

Let $D_{ref} \in \mathbb{R}^{L \times H}$ be the reference DLA profile matrix, flattened to a 1D vector $v_{ref} \in \mathbb{R}^{LH}$. Let $D_{eval}$ be the flattened active DLA profile vector $v_{eval}$. The circuit similarity score $S$ is calculated as:

$$S=\frac{v_{ref}\cdot v_{eval}}{\mathrm{\parallel }{v}_{ref}{\mathrm{\parallel }}_2\mathrm{\parallel }{v}_{eval}{\mathrm{\parallel }}_2}S\; =\; \backslash frac\{v\_\{ref\}\; \backslash cdot\; v\_\{eval\}\}\{\backslash |v\_\{ref\}\backslash |\_2\; \backslash |v\_\{eval\}\backslash |\_2\}$$

A mechanistic anomaly is flagged if $S$ drops below a specified threshold $\tau$:

Usage

from src.interpretability.safety import FunctionalAttributionMAD

mad = FunctionalAttributionMAD(model, threshold=0.6)

# Establish reference profile during normal aligned runs
mad.establish_reference_profile(
    baseline_states=normal_states,
    baseline_actions=normal_actions,
    baseline_returns=normal_returns,
    target_action_index=0
)

# Perform runtime checks
is_anomaly, similarity, profile = mad.detect_circuit_anomaly(
    eval_states=active_states,
    eval_actions=active_actions,
    eval_returns=active_returns,
    target_action_index=0
)

if is_anomaly:
    print(f"Anomalous circuit execution detected! Similarity: {similarity:.4f}")