Update README.md

README.md

@@ -1,3 +1,84 @@
----
-license: apache-2.0
----
+---
+license: apache-2.0
+base_model:
+- Zyphra/ZAYA1-base
+pipeline_tag: image-text-to-text
+---
+
+# ZAYA1-8B-VL
+
+
+
+
+## Performance
+
+ZAYA1-8B-VL performs extremely strongly against models of a comparable size and inference flops including outperforming several strong larger models. 
+
+![scores_vs_params-1](https://cdn-uploads.huggingface.co/production/uploads/65c05e75c084467acab2f84a/ivOwANwAmAzFDsa-JveP6.png)
+
+### Key Capabilities:
+
+* **Strong OCR and Document Understanding**: Excels at reading and understanding complex documents
+
+* **Powerful Gruonding and Spatial Perception**: Possesses strong grounding and bounding-box capabilities especially for high-resolution images.
+
+* **Multimodal Reasoning**: Capable of powerful and detailed mathematical and scientific reasoning about images.
+
+* **GUI and Computer-Use**: Highly proficient at reading websites and phone apps, pointing to correct buttons, and general computer use interactions.
+
+
+### Model Architecture
+
+ZAYA1-8B-VL builds upon and uses our [ZAYA1-7B LLM](https://huggingface.co/Zyphra/ZAYA1-base) as its base text encoder. We also use the [Qwen2.5-VL vision encoder](https://huggingface.co/docs/transformers/model_doc/qwen2_5_vl) for the ViT. ZAYA1-8B-VL introduces two novel architectural innovations:
+
+* **Vision-specific LoRA parameters**: ZAYA1-8B-VL utilizes specialized LoRA parameters on its MLPs and CCA weights which are only activated on vision tokens. We find that adding vision-specific parameters substantially improves model performance since the model has the option to devote specific parameters solely to visual processing. We train these LoRA parameters alongside the main model parameters during training.
+
+* **Bidirectional Attention for image tokens**: ZAYA1-8B-VL processes all image token inputs with a bidirectional attention mask, meaning attention is not causal across an image. We find that this improves performance by not imposing an arbitrary causal order to image tokens which are naturally non-causal.
+
+
+![smoe_vl_arch-1](https://cdn-uploads.huggingface.co/production/uploads/65c05e75c084467acab2f84a/wFuVyNIAzcaehBG1RiMjg.png)
+
+ZAYA1-8B-VL is trained only upon open data. Detailed dataset descriptions can be found in the accompanying technical report.
+
+## Quick start
+
+### Prerequisites
+
+TODO
+
+To use ZAYA1, install `zaya` branch from our fork of `transformers` library, which is based on the v4.57.1 of `transformers`:  
+```bash
+pip install "transformers @ git+https://github.com/Zyphra/transformers.git@zaya"
+```
+
+The command above relies on requirements for `transformers v4.57.1` being installed in your environment. If you're installing in a fresh Python environment, you might want to specify a specific extra, like `[dev-torch]`, to install all the dependencies:  
+```bash
+pip install "transformers[dev-torch] @ git+https://github.com/Zyphra/transformers.git@zaya"
+```
+
+
+### Inference
+
+TODO
+
+```python
+from transformers import AutoTokenizer, AutoModelForCausalLM
+import torch
+
+tokenizer = AutoTokenizer.from_pretrained("Zyphra/ZAYA1-base")
+model = AutoModelForCausalLM.from_pretrained("Zyphra/ZAYA1-base", device_map="cuda", dtype=torch.bfloat16)
+
+input_text = "What factors contributed to the fall of the Roman Empire?"
+input_ids = tokenizer(input_text, return_tensors="pt").to("cuda")
+
+outputs = model.generate(**input_ids, max_new_tokens=100)
+print(tokenizer.decode(outputs[0]))
+```
+
+### Citation
+
+If you find our work helpful, feel free to give us a cite.
+
+```
+TODO
+```