| --- |
| title: Failure Geometry Demo |
| emoji: 🧩 |
| colorFrom: purple |
| colorTo: indigo |
| sdk: gradio |
| sdk_version: 5.50.0 |
| app_file: app.py |
| pinned: false |
| license: mit |
| short_description: Failure structure discovery on CARB reasoning |
| --- |
| |
| # Failure Geometry Demo |
|
|
| Self-contained research demo for failure structure analysis on compositional reasoning. |
| **No API key required.** Runs entirely with scikit-learn. |
|
|
| ```text |
| CARB dataset → weak baselines → failure extraction → TF-IDF + SVD embeddings → KMeans → MI comparison |
| ``` |
|
|
| ## What this demonstrates |
|
|
| Two deliberately weak baselines expose different failure geometries on the same dataset: |
|
|
| | Baseline | Failure pattern | |
| |----------|----------------| |
| | `always_1` | Systematic bias — fails on every false-labeled item | |
| | `keyword_heuristic` | Negation-sensitive — fails on affirmative-false and negated-true items | |
|
|
| Pooling failures from both lets the demo ask: |
|
|
| > **Are failure clusters organised by reasoning category, by which baseline failed, or both?** |
|
|
| Mutual information over cluster assignments answers this. The accuracy-by-type chart shows |
| per-baseline slice performance. The 2-D SVD scatter shows whether clusters are visually separable. |
|
|
| ## Dataset |
|
|
| 50 controlled reasoning examples across four types: |
|
|
| - **Transitivity** — chained relational premises (`All A → B; All B → C; therefore All A → C`) |
| - **Negation** — rule-state-conclusion triples where the state negates the conclusion |
| - **Syllogism** — classical universal / particular forms with valid and invalid conclusions |
| - **Distractor logic** — valid causal rules with irrelevant sentences injected as distractors |
|
|
| Dataset schema matches `failure-induced-benchmarks` CARB conventions: `x`, `y`, `reasoning_type`. |
|
|
| ## What this does not claim |
|
|
| - The baselines are **deliberately weak** to surface failures; they are not representative of production models. |
| - MI scores on 50 items are **indicative, not statistically conclusive**. |
| - This is a scaffold for the pipeline, not a benchmark result. |
|
|
| ## Related |
|
|
| - **[obversarystudios.org](https://obversarystudios.org)** — research engineering narrative. |
| - [Failure discovery on binary reasoning](https://obversarystudios.org/docs/failure_discovery_binary_reasoning.html) — research framing. |
| - [Failure clusters as interventions](https://obversarystudios.org/docs/failure_clusters_as_interventions.html) — what clusters imply for system changes. |
| - **[carb-observability-space](https://huggingface.co/spaces/obversarystudios/carb-observability-space)** — live HF Inference API version (requires `HF_TOKEN`). |
| - **[agent-threat-map](https://huggingface.co/spaces/obversarystudios/agent-threat-map)** — agent fragility benchmark (metrics + embedding/cluster/MI on scored probes). |
| - **[github.com/architectfromthefuture/failure-induced-benchmarks](https://github.com/architectfromthefuture/failure-induced-benchmarks)** — CARB generator and failure geometry library. |
|
|
| ## Honest scope |
|
|
| - **Verified here:** pipeline runs end-to-end on the seed dataset; MI scores are computed correctly. |
| - **Described but not verified here:** statistical significance of MI gaps; generalization beyond this seed set. |
|
|
