File size: 1,898 Bytes
6abc905 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 | ---
pipeline_tag: image-text-to-text
---
# HAC: Parameter-Efficient Hyperbolic Adaptation of CLIP for Zero-Shot VQA
HAC (Hyperbolic Adaptation of CLIP) is a parameter-efficient framework that enables pretrained CLIP models to transition into hyperbolic space via lightweight fine-tuning. This approach captures hierarchical structures more effectively than traditional Euclidean embeddings, specifically for tasks like zero-shot Visual Question Answering (VQA).
This repository contains the weights for **HAC-B w/ LoRA**.
- **Paper:** [HAC: Parameter-Efficient Hyperbolic Adaptation of CLIP for Zero-Shot VQA](https://huggingface.co/papers/2604.23665)
- **GitHub Repository:** [https://github.com/fdibiton/HAC](https://github.com/fdibiton/HAC)
## Environment Setup
Create and configure the environment using Conda:
```bash
git clone https://github.com/fdibiton/HAC.git
cd HAC
conda create -n hac python=3.9 --yes
conda activate hac
# Install dependencies
python -m pip install --pre timm
python -m pip install -r requirements.txt
```
## Evaluation
To run zero-shot VQA evaluation with the HAC-B w/ LoRA model, place the `hac_vit_b_lora.pth` file in the `./checkpoints` directory and run:
```bash
python scripts/evaluate.py \
--config configs/eval_vqa_all_categories.py \
--train-config configs/train_hac_vit_b_lora.py \
--checkpoint-path checkpoints/hac_vit_b_lora.pth
```
> **Note:** The VQA evaluation datasets need to be downloaded and arranged beforehand. Please refer to the instructions in the [GitHub repository](https://github.com/fdibiton/HAC) for details.
## Citation
```bibtex
@inproceedings{dibiton2026hac,
title={HAC: Parameter-Efficient Hyperbolic Adaptation of CLIP for Zero-Shot VQA},
author={Dibitonto, Francesco and Beyan, Cigdem and Murino, Vittorio},
booktitle={International Conference on Pattern Recognition (ICPR)},
year={2026}
}
``` |