heavyhelium/tfidf-logreg-logic-fallacy

TF-IDF + Logistic Regression for logical fallacy classification

Classical baseline for 13-way logical fallacy classification on the LOGIC dataset from Jin et al. 2022, Logical Fallacy Detection (arXiv:2202.13758). Fit on the original zhijin train split (1849 examples).

This is an sklearn Pipeline: a TfidfVectorizer feeding a LogisticRegression. Useful as a cheap, CPU-only baseline — the paper does not include classical baselines, so this fills in the "surface-feature floor" for comparison against transformer fine-tuning.

Labels

ad hominem, ad populum, appeal to emotion, circular reasoning, equivocation,
fallacy of credibility, fallacy of extension, fallacy of logic,
fallacy of relevance, false causality, false dilemma, faulty generalization,
intentional

Configuration

Vectorizer: TfidfVectorizer(ngram_range=(1,2), min_df=2, stop_words='english', sublinear_tf=True, lowercase=True) — ~3.2k vocab features.

Classifier: LogisticRegression(C=4.0, class_weight='balanced', max_iter=2000, random_state=42).

Fit with sklearn==1.8.0.

Evaluation

In-domain (LOGIC test, n=300)

metric	value
accuracy	0.443
macro-F1	0.398
weighted-F1	0.443

Zero-shot transfer (LOGICCLIMATE all, n=1312)

metric	value
accuracy	0.097
macro-F1	0.082

Near random (13 classes → ~0.077 uniform). A pure-vocabulary model has essentially no transferable signal between the educational LOGIC domain and LOGICCLIMATE climate-change news. The fine-tuned electra-base-logic-fallacy retains ~0.18 macro-F1 under the same transfer, ~2× better, suggesting pretraining carries some domain-general signal that TF-IDF cannot.

Usage

from huggingface_hub import hf_hub_download
import joblib

path = hf_hub_download(
    repo_id="heavyhelium/tfidf-logreg-logic-fallacy",
    filename="pipeline.joblib",
)
pipe = joblib.load(path)

texts = [
    "Everyone I know drives a Toyota, so Toyotas must be the best cars.",
    "You are wrong because you are ugly.",
]
print(pipe.predict(texts))
# -> ['ad populum' 'ad hominem']

Limitations

• Pure vocabulary model. No semantic or structural understanding; succeeds only when a fallacy class has distinctive lexical cues (e.g. "everyone", "all") and fails catastrophically out of domain.
• Random-level transfer performance. Do not use on text that differs in topic or register from short educational examples.
• Imbalanced-class artifacts. equivocation and fallacy of logic have <0.25 F1 due to scarce training data and vocabulary overlap with other classes.

Citation

@inproceedings{jin-etal-2022-logical,
    title = "Logical Fallacy Detection",
    author = "Jin, Zhijing and Lalwani, Abhinav and Vaidhya, Tejas and Shen, Xiaoyu and Ding, Yiwen and Lyu, Zhiheng and Sachan, Mrinmaya and Mihalcea, Rada and Sch{\"o}lkopf, Bernhard",
    booktitle = "Findings of EMNLP 2022",
    year = "2022",
    url = "https://arxiv.org/abs/2202.13758",
}