meichifan/Qwen3-8B_lora_syllogism_fewshot_prompt

Qwen3-8B LoRA for Syllogism Validity (SemEval 2026 Task 11)

This repository contains a LoRA adapter on top of Qwen/Qwen3-8B fine-tuned to judge whether an English syllogism is logically valid or invalid, focusing on formal logical structure rather than real-world plausibility.

The model was developed for SemEval 2026 Task 11: Disentangling Content and Formal Reasoning in Language Models, as part of a course shared task project.

---

Model Details

Model Description

• Developed by: Eric Zhanyu Chen (University of Groningen – Hábrók cluster)
• Model type: Causal language model + LoRA adapter (PEFT)
• Base model: Qwen/Qwen3-8B
• Adapter type: LoRA (r=32, alpha=64, rank-decomposed attention/MLP)
• Language(s): English (trained on English syllogisms)
• Task: Binary classification of logical validity for syllogistic arguments
• License (adapter): MIT (see the License field of this repo)

  ⚠️ Using this model in practice also requires compliance with the base model’s license (Qwen/Qwen3-8B).

• Finetuned from: Qwen/Qwen3-8B with PEFT / LoRA

The adapter is relatively small (~87M trainable parameters, ≈1.05% of the 8B base model), so it is easy to load on top of the original Qwen3-8B weights.

Model Sources

• Code & training setup: https://github.com/SemHuis/group6-shared-task
  (LoRA fine-tuning and evaluation scripts)
• Base model: https://huggingface.co/Qwen/Qwen3-8B

---

Uses

Direct Use

This adapter is intended to be used as:

• A reasoning helper that classifies whether a given English syllogism is logically valid or invalid.
• A research tool to study content effects in reasoning:
  • Intra-plausibility content effect (valid vs. invalid within same plausibility)
  • Cross-plausibility content effect (plausible vs. implausible within same validity)

Typical use:

• Provide a short prompt that describes the task
• Insert the syllogism
• Ask the model to answer with “valid” or “invalid”
  (the adapter was trained with that exact target format).

The model is not a general-purpose logical theorem prover; it is tuned for syllogisms similar to those in the SemEval 2026 Task 11 data.

Downstream Use

Possible downstream uses include:

• Evaluating or analyzing other models’ reasoning via few-shot prompting of this LoRA model.
• Using it as a component in a larger pipeline that:
  • Generates candidate syllogisms, then
  • Filters them by formal validity.
• Classroom / educational demos about:
  • Formal vs. informal fallacies
  • The difference between logical validity and content plausibility

Out-of-Scope Use

This model should not be used for:

• Any high-stakes decisions (medical, legal, safety-critical, etc.).
• General factual QA or open-domain reasoning (it is only adapted for simple syllogisms).
• Automated grading or assessment of students without human oversight.
• As a substitute for professional expertise in logic, philosophy, or any domain it is applied to.

---

Bias, Risks, and Limitations

• The model is trained on a small, synthetic dataset (SemEval 2026 Task 11 training data – balanced valid/invalid).
• It is explicitly optimized to reduce content bias (plausibility effects), but:
  • It may still be influenced by real-world plausibility in unseen domains.
  • It is not guaranteed to be robust on long or non-standard arguments.
• It only sees the argument as a short English text, so:
  • Subtle scope ambiguities or unusual quantifiers may not be handled correctly.
  • It assumes “normal” natural language formulation similar to the training data.

Recommendations

• Treat outputs as suggestions, not ground truth.
• For research on reasoning:
  • Report both overall accuracy and content-effect metrics (see Evaluation section below).
• For teaching / demos:
  • Use the model as an assistant to generate examples, but always double-check results manually.
• Do not deploy this model as a standalone decision system in any sensitive domain.

---

How to Get Started with the Model

You need to load the base model and then the LoRA adapter.

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

BASE_MODEL = "Qwen/Qwen3-8B"
ADAPTER_ID = "meichifan/Qwen3_lora_syllogism"

# 1. Load tokenizer from the base model
tokenizer = AutoTokenizer.from_pretrained(BASE_MODEL, trust_remote_code=True)

# 2. Load base Qwen3-8B
base_model = AutoModelForCausalLM.from_pretrained(
    BASE_MODEL,
    torch_dtype="auto",
    device_map="auto",
    trust_remote_code=True,
)

# 3. Attach LoRA adapter
model = PeftModel.from_pretrained(base_model, ADAPTER_ID)
model.eval()