---
title: CARB Failure Observability
emoji: 🔬
colorFrom: indigo
colorTo: blue
sdk: gradio
sdk_version: 5.50.0
app_file: app.py
pinned: false
license: mit
short_description: Failure analysis for LM reasoning via HF Inference API
---

# CARB Failure Observability

Research pipeline for structured failure analysis in language model reasoning tasks.

```text
CARB dataset → HF Inference API → failure extraction → MiniLM embeddings → KMeans → mutual information
```

The central question: *do failure clusters align with reasoning categories (transitivity, negation, syllogism, distractor logic) more than with model identity?*

## What this Space does

1. Loads 50 controlled reasoning examples across four reasoning types (CARB-style: compositional, negation, syllogism, distractor logic).
2. Sends each prompt to one or more HF Inference API models.
3. Parses binary predictions and isolates failures (incorrect or unparsable outputs).
4. Embeds failures with `sentence-transformers/all-MiniLM-L6-v2`.
5. Clusters embeddings with KMeans (`k` is user-selectable).
6. Computes mutual information between cluster assignments and (a) reasoning type, (b) model identity.
7. Displays the MI comparison as a bar plot alongside a failure summary table.

## What this Space does not claim

- Benchmark results, leaderboard rankings, or SOTA comparisons.
- That the MI gap proves a general theory of failure structure — it is a signal on this dataset and these models.
- Production readiness; this is a research scaffold intended to be inspectable, not deployed.

## Running

Set `HF_TOKEN` in **Space secrets** before clicking **Run Experiment**.

Models queried by default: `google/flan-t5-small`, `google/flan-t5-base`.

## Related work

- **[obversarystudios.org](https://obversarystudios.org)** — research engineering narrative.
- [Failure discovery on binary reasoning](https://obversarystudios.org/docs/failure_discovery_binary_reasoning.html) — framing for this experiment.
- [Failure clusters as interventions](https://obversarystudios.org/docs/failure_clusters_as_interventions.html) — what to do with clusters once found.
- [Evaluation systems](https://obversarystudios.org/docs/evaluation_systems.html) — how this fits the broader eval lane.
- **[failure-geometry-demo](https://huggingface.co/spaces/obversarystudios/failure-geometry-demo)** — always-runnable sibling Space (sklearn baseline, no API key needed).
- **[agent-threat-map](https://huggingface.co/spaces/obversarystudios/agent-threat-map)** — agent-threat benchmark and observability (manual responses; optional geometry/MI).

## Honest scope

Evidence posture follows the lab template at
[github.com/architectfromthefuture](https://github.com/architectfromthefuture):

- **Verified here:** pipeline runs end-to-end with a valid `HF_TOKEN`; MI scores are computed and plotted.
- **Described but not verified here:** generalization beyond this seed dataset; statistical significance of any MI gap.