Add model card for HAC (#1)
Browse files- Add model card for HAC (6abc905e0fee1090d8c044dcfc70f516b144b20f)
Co-authored-by: Niels Rogge <nielsr@users.noreply.huggingface.co>
README.md
ADDED
|
@@ -0,0 +1,51 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
pipeline_tag: image-text-to-text
|
| 3 |
+
---
|
| 4 |
+
|
| 5 |
+
# HAC: Parameter-Efficient Hyperbolic Adaptation of CLIP for Zero-Shot VQA
|
| 6 |
+
|
| 7 |
+
HAC (Hyperbolic Adaptation of CLIP) is a parameter-efficient framework that enables pretrained CLIP models to transition into hyperbolic space via lightweight fine-tuning. This approach captures hierarchical structures more effectively than traditional Euclidean embeddings, specifically for tasks like zero-shot Visual Question Answering (VQA).
|
| 8 |
+
|
| 9 |
+
This repository contains the weights for **HAC-B w/ LoRA**.
|
| 10 |
+
|
| 11 |
+
- **Paper:** [HAC: Parameter-Efficient Hyperbolic Adaptation of CLIP for Zero-Shot VQA](https://huggingface.co/papers/2604.23665)
|
| 12 |
+
- **GitHub Repository:** [https://github.com/fdibiton/HAC](https://github.com/fdibiton/HAC)
|
| 13 |
+
|
| 14 |
+
## Environment Setup
|
| 15 |
+
|
| 16 |
+
Create and configure the environment using Conda:
|
| 17 |
+
|
| 18 |
+
```bash
|
| 19 |
+
git clone https://github.com/fdibiton/HAC.git
|
| 20 |
+
cd HAC
|
| 21 |
+
conda create -n hac python=3.9 --yes
|
| 22 |
+
conda activate hac
|
| 23 |
+
|
| 24 |
+
# Install dependencies
|
| 25 |
+
python -m pip install --pre timm
|
| 26 |
+
python -m pip install -r requirements.txt
|
| 27 |
+
```
|
| 28 |
+
|
| 29 |
+
## Evaluation
|
| 30 |
+
|
| 31 |
+
To run zero-shot VQA evaluation with the HAC-B w/ LoRA model, place the `hac_vit_b_lora.pth` file in the `./checkpoints` directory and run:
|
| 32 |
+
|
| 33 |
+
```bash
|
| 34 |
+
python scripts/evaluate.py \
|
| 35 |
+
--config configs/eval_vqa_all_categories.py \
|
| 36 |
+
--train-config configs/train_hac_vit_b_lora.py \
|
| 37 |
+
--checkpoint-path checkpoints/hac_vit_b_lora.pth
|
| 38 |
+
```
|
| 39 |
+
|
| 40 |
+
> **Note:** The VQA evaluation datasets need to be downloaded and arranged beforehand. Please refer to the instructions in the [GitHub repository](https://github.com/fdibiton/HAC) for details.
|
| 41 |
+
|
| 42 |
+
## Citation
|
| 43 |
+
|
| 44 |
+
```bibtex
|
| 45 |
+
@inproceedings{dibiton2026hac,
|
| 46 |
+
title={HAC: Parameter-Efficient Hyperbolic Adaptation of CLIP for Zero-Shot VQA},
|
| 47 |
+
author={Dibitonto, Francesco and Beyan, Cigdem and Murino, Vittorio},
|
| 48 |
+
booktitle={International Conference on Pattern Recognition (ICPR)},
|
| 49 |
+
year={2026}
|
| 50 |
+
}
|
| 51 |
+
```
|