mariklolik228/AraCompose-Qwen3-8B-adapters

AraCompose: Zero-Shot Arabic Task Transfer via Cross-Lingual Adapter Composition

Paper: "AraCompose: Zero-Shot Arabic Task Transfer via Cross-Lingual Adapter Composition with FLARE and TIES Merging" Authors: Mark Kashirskiy, Artiom Lipinski, Ilya Makarov Base model: Qwen/Qwen3-8B

Overview

AraCompose decomposes Arabic task performance into English task adapters + Arabic language adapter, then composes them via AdaMergeX. The result is Arabic task performance with zero Arabic task labels.

Method

1. Train English task adapters (QA, Math, Instruction, Safety) on SQuAD/GSM8K/Alpaca/SafetyBench
2. Train Arabic language adapter (CLM on Arabic Wikipedia)
3. Apply AdaMergeX formula: A_composed = A_ar + t * (A_en_task - A_en_wiki)
4. Optionally compose multiple task adapters via TIES merging

Adapter Variants

Adapter	Description	Size
task_adapters/qa	English QA (SQuAD) LoRA	170 MB
task_adapters/math	English Math (GSM8K) LoRA	170 MB
task_adapters/instruction	English Instruction (Alpaca) LoRA	170 MB
task_adapters/safety	English Safety LoRA	170 MB
full_adapters/ar_clm	Arabic language adapter (Wikipedia CLM)	170 MB
full_adapters/en_clm	English language adapter (Wikipedia CLM)	170 MB
full_adapters/adamergex_qa	AdaMergeX(QA + Arabic) — zero-shot Arabic QA	89 MB
full_adapters/adamergex_math	AdaMergeX(Math + Arabic) — zero-shot Arabic Math	89 MB
full_adapters/adamergex_instruction	AdaMergeX(Instruction + Arabic) — zero-shot Arabic Instruction	89 MB
full_adapters/adamergex_safety	AdaMergeX(Safety + Arabic) — zero-shot Arabic Safety	89 MB
full_adapters/ties_adamergex_all	TIES of all 4 AdaMergeX adapters	89 MB
full_adapters/ties_task_all	TIES of all 4 English task adapters	89 MB
full_adapters/ties_qa_arwiki	TIES of QA + Arabic CLM	89 MB
full_adapters/dialect_gulf	Gulf Arabic dialect adapter	7 MB
full_adapters/dialect_egyptian	Egyptian Arabic dialect adapter	5 MB
full_adapters/dialect_levantine	Levantine Arabic dialect adapter	5 MB
full_adapters/dialect_maghrebi	Maghrebi Arabic dialect adapter	5 MB

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
import torch

base_model = "Qwen/Qwen3-8B"
adapter_path = "mariklolik228/AraCompose-Qwen3-8B-adapters/full_adapters/adamergex_qa"

tokenizer = AutoTokenizer.from_pretrained(base_model)
model = AutoModelForCausalLM.from_pretrained(
    base_model,
    load_in_4bit=True,
    device_map={"": 0}
)
model = PeftModel.from_pretrained(model, adapter_path)

# Generate Arabic answer to an English QA question (zero-shot transfer)
prompt = "أجب على السؤال التالي: ما هي عاصمة مصر؟"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
output = model.generate(**inputs, max_new_tokens=100)
print(tokenizer.decode(output[0], skip_special_tokens=True))

Citation

@article{kashirskiy2026aracompose,
  title={AraCompose: Zero-Shot Arabic Task Transfer via Cross-Lingual Adapter Composition},
  author={Kashirskiy, Mark and Lipinski, Artiom and Makarov, Ilya},
  year={2026}
}