Image Classification
vision
ternary
quantization
vit
szymonrucinski commited on
Commit
d273128
·
verified ·
1 Parent(s): 8cb4b19

Add README with results table and download links

Browse files
Files changed (1) hide show
  1. README.md +57 -0
README.md ADDED
@@ -0,0 +1,57 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: apache-2.0
3
+ tags:
4
+ - vision
5
+ - image-classification
6
+ - ternary
7
+ - quantization
8
+ - vit
9
+ datasets:
10
+ - imagenet-1k
11
+ - cifar10
12
+ - cifar100
13
+ ---
14
+
15
+ # FTerViT: Fully Ternary Vision Transformer
16
+
17
+ Pretrained checkpoints for **FTerViT** — the first fully ternary Vision Transformer where *all* weight matrices and normalization parameters are constrained to {-1, 0, +1}.
18
+
19
+ **Paper:** [FTerViT: Fully Ternary Vision Transformer](https://arxiv.org/abs/XXXX.XXXXX) (NeurIPS 2026 submission)
20
+ **Code:** [github.com/szymonrucinski/FTerViT](https://github.com/szymonrucinski/FTerViT)
21
+
22
+ ## Key Results
23
+
24
+ All models use **W2A8** (2-bit weights, 8-bit activations) with 100% ternary coverage — including patch embedding, LayerNorm, and classifier head.
25
+
26
+ ### ImageNet-1K
27
+
28
+ | Model | Phase | Epochs | Top-1 (%) | Size (MB) | Compression | Checkpoint |
29
+ |-------|-------|--------|-----------|-----------|-------------|------------|
30
+ | DeiT-Small | Phase 1 | 250 | 75.05 | 5.81 | 15.2× | [download](https://huggingface.co/szymonrucinski/FTerViT/resolve/main/imagenet1k/phase1_ep250_acc75.05_deit_small_224.pth) |
31
+ | DeiT-III-Small | Phase 1 | 250 | 76.78 | 5.81 | 15.2× | [download](https://huggingface.co/szymonrucinski/FTerViT/resolve/main/imagenet1k/phase1_ep250_acc76.78_deit3_small_224.pth) |
32
+ | DeiT-Small | Phase 2 | +10 | **77.47** | 5.81 | 15.2× | [download](https://huggingface.co/szymonrucinski/FTerViT/resolve/main/imagenet1k/phase2_ep010_acc77.47_deit_small_224.pth) |
33
+ | DeiT-III-Small | Phase 2 | +10 | **79.64** | 5.81 | 15.2× | [download](https://huggingface.co/szymonrucinski/FTerViT/resolve/main/imagenet1k/phase2_ep010_acc79.64_deit3_small_224.pth) |
34
+
35
+ ### CIFAR-10 / CIFAR-100
36
+
37
+ | Model | Dataset | Top-1 (%) | FP32 Baseline | Size (MB) | Checkpoint |
38
+ |-------|---------|-----------|---------------|-----------|------------|
39
+ | DeiT-Tiny | CIFAR-10 | **97.43** | 97.52 | 1.53 | [download](https://huggingface.co/szymonrucinski/FTerViT/resolve/main/cifar10/phase2_ep010_acc97.43_deit_tiny_224.pth) |
40
+ | DeiT-Tiny | CIFAR-100 | **86.01** | 86.54 | 1.53 | [download](https://huggingface.co/szymonrucinski/FTerViT/resolve/main/cifar100/phase2_ep010_acc86.01_deit_tiny_224.pth) |
41
+
42
+ ## Training Protocol
43
+
44
+ Training uses a two-phase knowledge distillation approach:
45
+
46
+ - **Phase 1:** QAD with frozen FP32 teacher, KL-only loss, lr=1e-4 cosine decay, 250 epochs
47
+ - **Phase 2:** Low-lr recovery fine-tuning, lr=1e-5 cosine decay, 10 epochs
48
+
49
+ See the paper for full details.
50
+
51
+ ## Loading a Checkpoint
52
+
53
+
54
+
55
+ ## Citation
56
+
57
+