Add CARB observability pipeline

README.md

@@ -1,13 +1,60 @@
 ---
-title: Carb Observability Space
-emoji: 😻
+title: CARB Failure Observability
+emoji: 🔬
 colorFrom: indigo
-colorTo: green
+colorTo: blue
 sdk: gradio
-sdk_version: 6.13.0
+sdk_version: 4.44.1
 app_file: app.py
 pinned: false
 license: mit
+short_description: Structured failure analysis for LM reasoning — HF Inference API + cluster MI
 ---
 
-Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference
+# CARB Failure Observability
+
+Research pipeline for structured failure analysis in language model reasoning tasks.
+
+```text
+CARB dataset → HF Inference API → failure extraction → MiniLM embeddings → KMeans → mutual information
+```
+
+The central question: *do failure clusters align with reasoning categories (transitivity, negation, syllogism, distractor logic) more than with model identity?*
+
+## What this Space does
+
+1. Loads 50 controlled reasoning examples across four reasoning types (CARB-style: compositional, negation, syllogism, distractor logic).
+2. Sends each prompt to one or more HF Inference API models.
+3. Parses binary predictions and isolates failures (incorrect or unparsable outputs).
+4. Embeds failures with `sentence-transformers/all-MiniLM-L6-v2`.
+5. Clusters embeddings with KMeans (`k` is user-selectable).
+6. Computes mutual information between cluster assignments and (a) reasoning type, (b) model identity.
+7. Displays the MI comparison as a bar plot alongside a failure summary table.
+
+## What this Space does not claim
+
+- Benchmark results, leaderboard rankings, or SOTA comparisons.
+- That the MI gap proves a general theory of failure structure — it is a signal on this dataset and these models.
+- Production readiness; this is a research scaffold intended to be inspectable, not deployed.
+
+## Running
+
+Set `HF_TOKEN` in **Space secrets** before clicking **Run Experiment**.
+
+Models queried by default: `google/flan-t5-small`, `google/flan-t5-base`.
+
+## Related work
+
+- **[obversarystudios.org](https://obversarystudios.org)** — research engineering narrative.
+- [Failure discovery on binary reasoning](https://obversarystudios.org/docs/failure_discovery_binary_reasoning.html) — framing for this experiment.
+- [Failure clusters as interventions](https://obversarystudios.org/docs/failure_clusters_as_interventions.html) — what to do with clusters once found.
+- [Evaluation systems](https://obversarystudios.org/docs/evaluation_systems.html) — how this fits the broader eval lane.
+- **[failure-geometry-demo](https://huggingface.co/spaces/architectfromthefuture/failure-geometry-demo)** — always-runnable sibling Space (sklearn baseline, no API key needed).
+
+## Honest scope
+
+Evidence posture follows the lab template at
+[github.com/architectfromthefuture](https://github.com/architectfromthefuture):
+
+- **Verified here:** pipeline runs end-to-end with a valid `HF_TOKEN`; MI scores are computed and plotted.
+- **Described but not verified here:** generalization beyond this seed dataset; statistical significance of any MI gap.

app.py

@@ -0,0 +1,175 @@
+from pathlib import Path
+
+import gradio as gr
+import pandas as pd
+
+from core.cluster import cluster_embeddings
+from core.dataset import load_dataset
+from core.embed import embed_failures
+from core.eval import evaluate
+from core.metrics import compute_mi_scores
+from core.model import DEFAULT_MODELS, query_model
+from viz.plots import plot_mi_comparison
+
+
+DATA_PATH = Path(__file__).parent / "data" / "carb_seed.json"
+
+_DESCRIPTION = """\
+## CARB Failure Observability
+
+Research pipeline for structured failure analysis in language model reasoning.
+
+```
+CARB dataset → HF Inference API → failure extraction → MiniLM embeddings → KMeans → mutual information
+```
+
+**Central question:** do failure clusters align with *reasoning category* more than with *model identity*?
+
+The MI comparison plot answers this directly — a larger `MI(cluster, reasoning_type)` bar relative to
+`MI(cluster, model_identity)` supports the hypothesis that failure structure is organized by reasoning
+difficulty, not model choice alone.
+
+> **Requires** `HF_TOKEN` set in Space secrets. See
+> [failure-geometry-demo](https://huggingface.co/spaces/architectfromthefuture/failure-geometry-demo)
+> for a fully self-contained version that needs no API key.
+>
+> Research context: [obversarystudios.org](https://obversarystudios.org)
+"""
+
+
+def run_experiment(
+    selected_models: list[str],
+    n_clusters: int,
+) -> tuple[str, object, object]:
+    log_lines: list[str] = []
+
+    def log(msg: str) -> None:
+        log_lines.append(msg)
+
+    if not selected_models:
+        selected_models = DEFAULT_MODELS[:1]
+
+    log(f"Loading dataset from {DATA_PATH.name} …")
+    try:
+        dataset = load_dataset(DATA_PATH)
+    except Exception as exc:
+        return f"Dataset error: {exc}", None, None
+
+    log(f"  {len(dataset)} examples across {len({r['reasoning_type'] for r in dataset})} reasoning types.")
+    log(f"Querying models: {', '.join(selected_models)} …")
+
+    try:
+        failures = evaluate(dataset, query_model, model_ids=selected_models)
+    except Exception as exc:
+        return f"Evaluation error: {exc}", None, None
+
+    log(f"  Found {len(failures)} failures from {len(dataset) * len(selected_models)} total predictions.")
+
+    if not failures:
+        log("No failures detected — all predictions were correct.")
+        empty_mi = {
+            "MI(cluster, reasoning_type)": 0.0,
+            "MI(cluster, model_identity)": 0.0,
+        }
+        fig = plot_mi_comparison(empty_mi)
+        return "\n".join(log_lines), fig, _empty_summary_table()
+
+    log("Embedding failures with all-MiniLM-L6-v2 …")
+    try:
+        embeddings = embed_failures(failures)
+    except Exception as exc:
+        return "\n".join(log_lines) + f"\nEmbed error: {exc}", None, None
+
+    log(f"  Embeddings shape: {embeddings.shape}")
+    log(f"Clustering into k={n_clusters} clusters (KMeans) …")
+
+    cluster_ids = cluster_embeddings(embeddings, n_clusters=n_clusters)
+    for failure, cluster_id in zip(failures, cluster_ids, strict=True):
+        failure["cluster_id"] = cluster_id
+
+    counts_per_cluster = {}
+    for cid in cluster_ids:
+        counts_per_cluster[cid] = counts_per_cluster.get(cid, 0) + 1
+    log(f"  Cluster sizes: { {k: counts_per_cluster[k] for k in sorted(counts_per_cluster)} }")
+
+    reasoning_types = [f["reasoning_type"] for f in failures]
+    model_ids_list = [f["model_id"] for f in failures]
+
+    log("Computing mutual information …")
+    mi_scores = compute_mi_scores(cluster_ids, reasoning_types, model_ids_list)
+    for label, score in mi_scores.items():
+        log(f"  {label}: {score:.4f}")
+
+    fig = plot_mi_comparison(mi_scores)
+    summary_df = _build_summary_table(failures)
+
+    return "\n".join(log_lines), fig, summary_df
+
+
+def _build_summary_table(failures: list[dict]) -> pd.DataFrame:
+    from collections import Counter
+    counts: Counter = Counter()
+    for f in failures:
+        counts[(f["reasoning_type"], f["model_id"])] += 1
+
+    rows = [
+        {"reasoning_type": rtype, "model_id": mid, "failure_count": cnt}
+        for (rtype, mid), cnt in sorted(counts.items())
+    ]
+    return pd.DataFrame(rows) if rows else _empty_summary_table()
+
+
+def _empty_summary_table() -> pd.DataFrame:
+    return pd.DataFrame(columns=["reasoning_type", "model_id", "failure_count"])
+
+
+with gr.Blocks(title="CARB Failure Observability", theme=gr.themes.Soft()) as demo:
+    gr.Markdown(_DESCRIPTION)
+
+    with gr.Row():
+        with gr.Column(scale=1, min_width=260):
+            model_selector = gr.CheckboxGroup(
+                choices=DEFAULT_MODELS,
+                value=DEFAULT_MODELS[:1],
+                label="Models to query",
+                info="Each model runs on all 50 examples. Multiple models increase failure pool diversity.",
+            )
+            n_clusters_slider = gr.Slider(
+                minimum=2,
+                maximum=6,
+                step=1,
+                value=4,
+                label="KMeans clusters (k)",
+                info="Should be ≤ number of reasoning types (4).",
+            )
+            run_btn = gr.Button("Run Experiment", variant="primary", size="lg")
+
+        with gr.Column(scale=2):
+            status_log = gr.Textbox(
+                label="Pipeline log",
+                lines=9,
+                interactive=False,
+                placeholder="Click 'Run Experiment' to start …",
+            )
+
+    with gr.Row():
+        mi_plot = gr.Plot(
+            label="Mutual information: cluster vs. reasoning type vs. model identity"
+        )
+
+    with gr.Row():
+        summary_table = gr.Dataframe(
+            headers=["reasoning_type", "model_id", "failure_count"],
+            label="Failures by reasoning type and model",
+            interactive=False,
+        )
+
+    run_btn.click(
+        fn=run_experiment,
+        inputs=[model_selector, n_clusters_slider],
+        outputs=[status_log, mi_plot, summary_table],
+    )
+
+
+if __name__ == "__main__":
+    demo.launch()

carb-observability-space

@@ -0,0 +1 @@
+Subproject commit 0fcfd1cb2222aa6b2ce874133ab7ac03305d7823

core/cluster.py

@@ -0,0 +1,22 @@
+import numpy as np
+from sklearn.cluster import KMeans
+
+
+def cluster_embeddings(
+    embeddings: np.ndarray,
+    n_clusters: int = 4,
+    random_state: int = 42,
+) -> list[int]:
+    if len(embeddings) == 0:
+        return []
+
+    effective_clusters = min(n_clusters, len(embeddings))
+    if effective_clusters == 1:
+        return [0]
+
+    kmeans = KMeans(
+        n_clusters=effective_clusters,
+        random_state=random_state,
+        n_init=10,
+    )
+    return kmeans.fit_predict(embeddings).tolist()

core/dataset.py

@@ -0,0 +1,29 @@
+import json
+from pathlib import Path
+from typing import Any
+
+
+REQUIRED_FIELDS = {"x", "y", "reasoning_type"}
+
+
+def load_dataset(path: str | Path) -> list[dict[str, Any]]:
+    """Load and validate the small CARB-style seed dataset."""
+    dataset_path = Path(path)
+    with dataset_path.open("r", encoding="utf-8") as f:
+        rows = json.load(f)
+
+    if not isinstance(rows, list):
+        raise ValueError("Dataset must be a JSON list.")
+
+    for index, row in enumerate(rows):
+        missing = REQUIRED_FIELDS.difference(row)
+        if missing:
+            raise ValueError(f"Row {index} is missing required fields: {sorted(missing)}")
+        if row["y"] not in (0, 1):
+            raise ValueError(f"Row {index} has non-binary label: {row['y']!r}")
+        if not isinstance(row["x"], str) or not row["x"].strip():
+            raise ValueError(f"Row {index} has an empty input string.")
+        if not isinstance(row["reasoning_type"], str) or not row["reasoning_type"].strip():
+            raise ValueError(f"Row {index} has an empty reasoning_type.")
+
+    return rows

core/embed.py

@@ -0,0 +1,26 @@
+from collections.abc import Sequence
+
+import numpy as np
+from sentence_transformers import SentenceTransformer
+
+
+EMBEDDING_MODEL = "sentence-transformers/all-MiniLM-L6-v2"
+
+
+def embed_failures(failures: Sequence[dict[str, object]]) -> np.ndarray:
+    texts = [_failure_text(failure) for failure in failures]
+    if not texts:
+        return np.empty((0, 384))
+
+    model = SentenceTransformer(EMBEDDING_MODEL)
+    return model.encode(texts, convert_to_numpy=True, normalize_embeddings=True)
+
+
+def _failure_text(failure: dict[str, object]) -> str:
+    return (
+        f"input: {failure['x']}\n"
+        f"expected: {failure['y']}\n"
+        f"prediction: {failure['prediction']}\n"
+        f"reasoning_type: {failure['reasoning_type']}\n"
+        f"model: {failure['model_id']}"
+    )

core/eval.py

@@ -0,0 +1,43 @@
+from collections.abc import Callable, Sequence
+from typing import Any
+
+from core.model import DEFAULT_MODELS, build_prompt, parse_binary_prediction
+
+
+ModelFn = Callable[[str, str], str]
+
+
+def evaluate(
+    dataset: Sequence[dict[str, Any]],
+    model_fn: ModelFn,
+    model_ids: Sequence[str] | None = None,
+) -> list[dict[str, Any]]:
+    """Run models over the dataset and return only incorrect or unparsable cases."""
+    failures: list[dict[str, Any]] = []
+    selected_model_ids = list(model_ids or DEFAULT_MODELS)
+
+    for sample_id, sample in enumerate(dataset):
+        prompt = build_prompt(sample["x"])
+        expected = int(sample["y"])
+
+        for model_id in selected_model_ids:
+            raw_output = model_fn(prompt, model_id)
+            prediction = parse_binary_prediction(raw_output)
+            is_correct = prediction == expected
+
+            if not is_correct:
+                failures.append(
+                    {
+                        "sample_id": sample_id,
+                        "x": sample["x"],
+                        "y": expected,
+                        "reasoning_type": sample["reasoning_type"],
+                        "model_id": model_id,
+                        "prompt": prompt,
+                        "raw_output": raw_output,
+                        "prediction": prediction,
+                        "failure_kind": "parse_error" if prediction is None else "wrong_label",
+                    }
+                )
+
+    return failures

core/metrics.py

@@ -0,0 +1,20 @@
+from collections.abc import Sequence
+
+from sklearn.metrics import mutual_info_score
+
+
+def compute_mi_scores(
+    cluster_ids: Sequence[int],
+    reasoning_types: Sequence[str],
+    model_ids: Sequence[str],
+) -> dict[str, float]:
+    if not cluster_ids:
+        return {
+            "MI(cluster, reasoning_type)": 0.0,
+            "MI(cluster, model_identity)": 0.0,
+        }
+
+    return {
+        "MI(cluster, reasoning_type)": float(mutual_info_score(cluster_ids, reasoning_types)),
+        "MI(cluster, model_identity)": float(mutual_info_score(cluster_ids, model_ids)),
+    }

core/model.py

@@ -0,0 +1,91 @@
+import os
+import re
+from collections.abc import Sequence
+
+import requests
+
+
+DEFAULT_MODELS = [
+    "google/flan-t5-small",
+    "google/flan-t5-base",
+]
+
+
+def build_prompt(input_text: str) -> str:
+    return (
+        "Answer this binary reasoning question. "
+        "Return only one line in the format 'label: 0' or 'label: 1'.\n\n"
+        f"Question: {input_text}"
+    )
+
+
+def query_model(prompt: str, model_id: str = DEFAULT_MODELS[0], timeout: int = 60) -> str:
+    """Call the Hugging Face Inference API and return model text."""
+    token = os.environ.get("HF_TOKEN")
+    if not token:
+        return "ERROR: HF_TOKEN is not set."
+
+    url = f"https://api-inference.huggingface.co/models/{model_id}"
+    headers = {"Authorization": f"Bearer {token}"}
+    payload = {
+        "inputs": prompt,
+        "parameters": {"max_new_tokens": 32, "return_full_text": False},
+        "options": {"wait_for_model": True},
+    }
+
+    try:
+        response = requests.post(url, headers=headers, json=payload, timeout=timeout)
+        response.raise_for_status()
+        data = response.json()
+    except requests.RequestException as exc:
+        return f"ERROR: request failed for {model_id}: {exc}"
+    except ValueError:
+        return f"ERROR: non-JSON response from {model_id}."
+
+    return _extract_generated_text(data)
+
+
+def query_models(prompt: str, model_ids: Sequence[str]) -> dict[str, str]:
+    return {model_id: query_model(prompt, model_id=model_id) for model_id in model_ids}
+
+
+def parse_binary_prediction(output: str) -> int | None:
+    """Parse a structured binary label from model output."""
+    normalized = output.strip().lower()
+    if normalized.startswith("error:"):
+        return None
+
+    structured_patterns = [
+        r"\blabel\s*[:=]\s*([01])\b",
+        r"\banswer\s*[:=]\s*([01])\b",
+        r"\bprediction\s*[:=]\s*([01])\b",
+    ]
+    for pattern in structured_patterns:
+        match = re.search(pattern, normalized)
+        if match:
+            return int(match.group(1))
+
+    if re.fullmatch(r"[01]", normalized):
+        return int(normalized)
+
+    return None
+
+
+def _extract_generated_text(data: object) -> str:
+    if isinstance(data, list) and data:
+        first = data[0]
+        if isinstance(first, dict):
+            text = first.get("generated_text") or first.get("summary_text")
+            if isinstance(text, str):
+                return text
+        if isinstance(first, str):
+            return first
+
+    if isinstance(data, dict):
+        if isinstance(data.get("error"), str):
+            return f"ERROR: {data['error']}"
+        text = data.get("generated_text") or data.get("summary_text")
+        if isinstance(text, str):
+            return text
+
+    return f"ERROR: unsupported response format: {data!r}"

data/carb_seed.json

@@ -0,0 +1,252 @@
+[
+  {
+    "x": "Label 1 if the conclusion follows, else 0. Premise: All robins are birds. All birds are animals. Conclusion: All robins are animals.",
+    "y": 1,
+    "reasoning_type": "transitivity"
+  },
+  {
+    "x": "Label 1 if the conclusion follows, else 0. Premise: All squares are rectangles. All rectangles are shapes. Conclusion: All squares are shapes.",
+    "y": 1,
+    "reasoning_type": "transitivity"
+  },
+  {
+    "x": "Label 1 if the conclusion follows, else 0. Premise: All tulips are flowers. All flowers are plants. Conclusion: All tulips are plants.",
+    "y": 1,
+    "reasoning_type": "transitivity"
+  },
+  {
+    "x": "Label 1 if the conclusion follows, else 0. Premise: All ferries are boats. All boats are vehicles. Conclusion: All ferries are vehicles.",
+    "y": 1,
+    "reasoning_type": "transitivity"
+  },
+  {
+    "x": "Label 1 if the conclusion follows, else 0. Premise: All violins are instruments. All instruments are objects. Conclusion: All violins are objects.",
+    "y": 1,
+    "reasoning_type": "transitivity"
+  },
+  {
+    "x": "Label 1 if the conclusion follows, else 0. Premise: All oak trees are trees. All trees are living things. Conclusion: All oak trees are living things.",
+    "y": 1,
+    "reasoning_type": "transitivity"
+  },
+  {
+    "x": "Label 1 if the conclusion follows, else 0. Premise: All comets are space objects. All space objects are visible from telescopes. Conclusion: All comets are visible from telescopes.",
+    "y": 1,
+    "reasoning_type": "transitivity"
+  },
+  {
+    "x": "Label 1 if the conclusion follows, else 0. Premise: All laptops are computers. All computers are machines. Conclusion: All machines are laptops.",
+    "y": 0,
+    "reasoning_type": "transitivity"
+  },
+  {
+    "x": "Label 1 if the conclusion follows, else 0. Premise: All sparrows are birds. All birds have feathers. Conclusion: All feathered things are sparrows.",
+    "y": 0,
+    "reasoning_type": "transitivity"
+  },
+  {
+    "x": "Label 1 if the conclusion follows, else 0. Premise: All apples are fruit. All fruit is food. Conclusion: All food is apples.",
+    "y": 0,
+    "reasoning_type": "transitivity"
+  },
+  {
+    "x": "Label 1 if the conclusion follows, else 0. Premise: All poets are writers. All writers use language. Conclusion: All language users are poets.",
+    "y": 0,
+    "reasoning_type": "transitivity"
+  },
+  {
+    "x": "Label 1 if the conclusion follows, else 0. Premise: All poodles are dogs. All dogs are mammals. Conclusion: Some mammals are not poodles.",
+    "y": 0,
+    "reasoning_type": "transitivity"
+  },
+  {
+    "x": "Label 1 if the conclusion follows, else 0. Premise: All taxis are cars. All cars need fuel. Conclusion: All taxis need fuel.",
+    "y": 1,
+    "reasoning_type": "transitivity"
+  },
+  {
+    "x": "Label 1 if the final statement is true, else 0. Rule: If a door is locked, it is not open. The door is locked. Statement: The door is open.",
+    "y": 0,
+    "reasoning_type": "negation"
+  },
+  {
+    "x": "Label 1 if the final statement is true, else 0. Rule: If a badge is valid, it is not expired. The badge is valid. Statement: The badge is expired.",
+    "y": 0,
+    "reasoning_type": "negation"
+  },
+  {
+    "x": "Label 1 if the final statement is true, else 0. Rule: If the lamp is unplugged, it is not powered. The lamp is unplugged. Statement: The lamp is not powered.",
+    "y": 1,
+    "reasoning_type": "negation"
+  },
+  {
+    "x": "Label 1 if the final statement is true, else 0. Rule: If the file is encrypted, it is not readable as plain text. The file is encrypted. Statement: The file is readable as plain text.",
+    "y": 0,
+    "reasoning_type": "negation"
+  },
+  {
+    "x": "Label 1 if the final statement is true, else 0. Rule: If the road is closed, cars cannot pass. The road is closed. Statement: Cars can pass.",
+    "y": 0,
+    "reasoning_type": "negation"
+  },
+  {
+    "x": "Label 1 if the final statement is true, else 0. Rule: If a switch is off, the circuit is not active. The switch is off. Statement: The circuit is not active.",
+    "y": 1,
+    "reasoning_type": "negation"
+  },
+  {
+    "x": "Label 1 if the final statement is true, else 0. Rule: If a ticket is unpaid, it is not confirmed. The ticket is unpaid. Statement: The ticket is confirmed.",
+    "y": 0,
+    "reasoning_type": "negation"
+  },
+  {
+    "x": "Label 1 if the final statement is true, else 0. Rule: If a jar is empty, it contains no marbles. The jar is empty. Statement: The jar contains marbles.",
+    "y": 0,
+    "reasoning_type": "negation"
+  },
+  {
+    "x": "Label 1 if the final statement is true, else 0. Rule: If a user is banned, they are not allowed to post. The user is banned. Statement: The user is not allowed to post.",
+    "y": 1,
+    "reasoning_type": "negation"
+  },
+  {
+    "x": "Label 1 if the final statement is true, else 0. Rule: If the sensor is disabled, it sends no alerts. The sensor is disabled. Statement: The sensor sends alerts.",
+    "y": 0,
+    "reasoning_type": "negation"
+  },
+  {
+    "x": "Label 1 if the final statement is true, else 0. Rule: If a package is missing, it is not delivered. The package is missing. Statement: The package is delivered.",
+    "y": 0,
+    "reasoning_type": "negation"
+  },
+  {
+    "x": "Label 1 if the final statement is true, else 0. Rule: If a plant is dead, it is not growing. The plant is dead. Statement: The plant is not growing.",
+    "y": 1,
+    "reasoning_type": "negation"
+  },
+  {
+    "x": "Label 1 if the conclusion follows, else 0. Premise: All doctors are trained professionals. Mira is a doctor. Conclusion: Mira is a trained professional.",
+    "y": 1,
+    "reasoning_type": "syllogism"
+  },
+  {
+    "x": "Label 1 if the conclusion follows, else 0. Premise: All guests need invitations. Omar is a guest. Conclusion: Omar needs an invitation.",
+    "y": 1,
+    "reasoning_type": "syllogism"
+  },
+  {
+    "x": "Label 1 if the conclusion follows, else 0. Premise: No reptiles are warm-blooded. A gecko is a reptile. Conclusion: A gecko is warm-blooded.",
+    "y": 0,
+    "reasoning_type": "syllogism"
+  },
+  {
+    "x": "Label 1 if the conclusion follows, else 0. Premise: All library books have catalog numbers. This item is a library book. Conclusion: This item has a catalog number.",
+    "y": 1,
+    "reasoning_type": "syllogism"
+  },
+  {
+    "x": "Label 1 if the conclusion follows, else 0. Premise: No expired coupons are accepted. This coupon is expired. Conclusion: This coupon is accepted.",
+    "y": 0,
+    "reasoning_type": "syllogism"
+  },
+  {
+    "x": "Label 1 if the conclusion follows, else 0. Premise: All certified pilots can fly planes. Dana is certified pilot. Conclusion: Dana can fly planes.",
+    "y": 1,
+    "reasoning_type": "syllogism"
+  },
+  {
+    "x": "Label 1 if the conclusion follows, else 0. Premise: All medals are awards. This object is an award. Conclusion: This object is a medal.",
+    "y": 0,
+    "reasoning_type": "syllogism"
+  },
+  {
+    "x": "Label 1 if the conclusion follows, else 0. Premise: No broken clocks keep correct time. This clock is broken. Conclusion: This clock keeps correct time.",
+    "y": 0,
+    "reasoning_type": "syllogism"
+  },
+  {
+    "x": "Label 1 if the conclusion follows, else 0. Premise: All subscribers receive updates. Jin is a subscriber. Conclusion: Jin receives updates.",
+    "y": 1,
+    "reasoning_type": "syllogism"
+  },
+  {
+    "x": "Label 1 if the conclusion follows, else 0. Premise: No silent alarms make noise. This alarm is silent. Conclusion: This alarm makes noise.",
+    "y": 0,
+    "reasoning_type": "syllogism"
+  },
+  {
+    "x": "Label 1 if the conclusion follows, else 0. Premise: All registered voters may vote. Lee is registered voter. Conclusion: Lee may vote.",
+    "y": 1,
+    "reasoning_type": "syllogism"
+  },
+  {
+    "x": "Label 1 if the conclusion follows, else 0. Premise: All chess players know rules. Sam knows rules. Conclusion: Sam is a chess player.",
+    "y": 0,
+    "reasoning_type": "syllogism"
+  },
+  {
+    "x": "Label 1 if the target conclusion follows, else 0. Useful rule: If the blue key is used, the safe opens. Distractor: The red key is shiny. The blue key is used. Conclusion: The safe opens.",
+    "y": 1,
+    "reasoning_type": "distractor logic"
+  },
+  {
+    "x": "Label 1 if the target conclusion follows, else 0. Useful rule: If the server restarts, the cache clears. Distractor: The keyboard is wireless. The server restarts. Conclusion: The cache clears.",
+    "y": 1,
+    "reasoning_type": "distractor logic"
+  },
+  {
+    "x": "Label 1 if the target conclusion follows, else 0. Useful rule: If the form is signed, the request is valid. Distractor: The envelope is yellow. The form is not signed. Conclusion: The request is valid.",
+    "y": 0,
+    "reasoning_type": "distractor logic"
+  },
+  {
+    "x": "Label 1 if the target conclusion follows, else 0. Useful rule: If the alarm rings, the guard wakes. Distractor: The guard owns a bicycle. The alarm rings. Conclusion: The guard wakes.",
+    "y": 1,
+    "reasoning_type": "distractor logic"
+  },
+  {
+    "x": "Label 1 if the target conclusion follows, else 0. Useful rule: If the code compiles, tests can run. Distractor: The monitor is large. The code does not compile. Conclusion: Tests can run.",
+    "y": 0,
+    "reasoning_type": "distractor logic"
+  },
+  {
+    "x": "Label 1 if the target conclusion follows, else 0. Useful rule: If the window is open, the room cools. Distractor: The carpet is green. The window is open. Conclusion: The room cools.",
+    "y": 1,
+    "reasoning_type": "distractor logic"
+  },
+  {
+    "x": "Label 1 if the target conclusion follows, else 0. Useful rule: If the invoice is paid, the account is active. Distractor: The logo is blue. The invoice is unpaid. Conclusion: The account is active.",
+    "y": 0,
+    "reasoning_type": "distractor logic"
+  },
+  {
+    "x": "Label 1 if the target conclusion follows, else 0. Useful rule: If the train arrives, passengers board. Distractor: The station has a clock. The train arrives. Conclusion: Passengers board.",
+    "y": 1,
+    "reasoning_type": "distractor logic"
+  },
+  {
+    "x": "Label 1 if the target conclusion follows, else 0. Useful rule: If the token is invalid, access is denied. Distractor: The desk has two drawers. The token is invalid. Conclusion: Access is denied.",
+    "y": 1,
+    "reasoning_type": "distractor logic"
+  },
+  {
+    "x": "Label 1 if the target conclusion follows, else 0. Useful rule: If rain falls, the ground gets wet. Distractor: The umbrella is red. Rain does not fall. Conclusion: The ground gets wet.",
+    "y": 0,
+    "reasoning_type": "distractor logic"
+  },
+  {
+    "x": "Label 1 if the target conclusion follows, else 0. Useful rule: If the switch is flipped, the light turns on. Distractor: The wall is painted white. The switch is flipped. Conclusion: The light turns on.",
+    "y": 1,
+    "reasoning_type": "distractor logic"
+  },
+  {
+    "x": "Label 1 if the target conclusion follows, else 0. Useful rule: If the battery is charged, the robot moves. Distractor: The robot is made of metal. The battery is empty. Conclusion: The robot moves.",
+    "y": 0,
+    "reasoning_type": "distractor logic"
+  },
+  {
+    "x": "Label 1 if the target conclusion follows, else 0. Useful rule: If the map is accurate, the route is reliable. Distractor: The compass is old. The map is accurate. Conclusion: The route is reliable.",
+    "y": 1,
+    "reasoning_type": "distractor logic"
+  }
+]

requirements.txt

@@ -0,0 +1,6 @@
+gradio
+requests
+sentence-transformers
+scikit-learn
+matplotlib
+numpy

viz/plots.py

@@ -0,0 +1,19 @@
+import matplotlib.pyplot as plt
+
+
+def plot_mi_comparison(mi_scores: dict[str, float]):
+    fig, ax = plt.subplots(figsize=(7, 4))
+    labels = list(mi_scores.keys())
+    values = list(mi_scores.values())
+
+    ax.bar(labels, values, color=["#4C78A8", "#F58518"])
+    ax.set_ylabel("Mutual information")
+    ax.set_title("Failure Cluster Mutual Information")
+    ax.set_ylim(0, max(values + [0.05]) * 1.2)
+    ax.tick_params(axis="x", labelrotation=15)
+
+    for index, value in enumerate(values):
+        ax.text(index, value, f"{value:.3f}", ha="center", va="bottom")
+
+    fig.tight_layout()
+    return fig