Update README.md

README.md

@@ -1,72 +1,161 @@
 ---
-library_name: transformers
-license: other
-base_model: Qwen/Qwen3-1.7B-Base
+license: apache-2.0
+language:
+- en
 tags:
-- llama-factory
-- full
-- generated_from_trainer
-model-index:
-- name: cpt_1p7b_biom_cpt_ds_64gpus_z0_resume
-  results: []
+- biology
+- chemistry
+- molecule
+- protein
+- multimodal
+- foundation-model
+- pretrained
+pipeline_tag: text-generation
+base_model: Qwen/Qwen3-1.7B
+library_name: transformers
 ---
 
-<!-- This model card has been generated automatically according to the information the Trainer had access to. You
-should probably proofread and complete it, then remove this comment. -->
+# BioMatrix-1.7B-Base
+
+**BioMatrix** is a multimodal biological foundation model that natively integrates **1D sequences**, **3D structures**, and **natural language** for both **molecules** and **proteins** within a single decoder-only architecture.
+
+This is the **1.7B-parameter Base model**, obtained via **multimodal continual pretraining** of Qwen3-1.7B on 304.4 billion tokens spanning text, molecular and protein 1D/3D data, and cross-modal corpora. This base checkpoint is intended for further fine-tuning on downstream tasks. For an instruction-tuned model ready for inference, see [BioMatrix-1.7B-SFT](https://huggingface.co/QizhiPei/BioMatrix-1.7B-SFT). For a larger model, see [BioMatrix-4B-Base](https://huggingface.co/QizhiPei/BioMatrix-4B-Base).
+
+- 📄 **Paper**: [BioMatrix: Towards a Comprehensive Biological Foundation Model Spanning the Modality Matrix of Sequences, Structures, and Language](https://arxiv.org/abs/xxxx.xxxxx)
+- 💻 **Code**: [https://github.com/QizhiPei/biomatrix](https://github.com/QizhiPei/biomatrix)
+- 🤗 **Model & Data Collection**: [https://huggingface.co/collections/QizhiPei/biomatrix](https://huggingface.co/collections/QizhiPei/biomatrix)
+
+## Model Overview
+
+BioMatrix maps **all biological modalities into a shared discrete token space** via a unified tokenization scheme:
+
+- **Molecular 1D sequences** (both SMILES and SELFIES notations)
+- **Molecular 3D structures** (via MolStrucTok with branch-decoupled decoder)
+- **Protein 1D sequences** (residue-level tokens)
+- **Protein 3D structures** (via GCP-VQVAE backbone tokenizer)
+- **Natural language** (inherited from Qwen3 tokenizer)
+
+All modalities are consumed and produced uniformly under a **single next-token prediction objective**—without external encoders, projection adapters, or modality-specific output heads.
+
+| Model | Molecule 1D | Molecule 3D | Protein 1D | Protein 3D | Natural Language |
+|-------|:-----------:|:-----------:|:----------:|:----------:|:----------------:|
+| ESM3 | ✗ | ✗ | ✓ | ✓ | ✓ |
+| 3D-MoLM | ✓ | ✓ | ✗ | ✗ | ✓ |
+| AlphaFold3 | ✓ | ✓ | ✓ | ✓ | ✗ |
+| BioT5/BioT5+ | ✓ | ✗ | ✓ | ✗ | ✓ |
+| BioMedGPT | ✓ | ✗ | ✓ | ✗ | ✓ |
+| **BioMatrix** | **✓** | **✓** | **✓** | **✓** | **✓** |
 
-# cpt_1p7b_biom_cpt_ds_64gpus_z0_resume
+## Model Details
 
-This model is a fine-tuned version of [Qwen/Qwen3-1.7B-Base](https://huggingface.co/Qwen/Qwen3-1.7B-Base) on the biom_cpt_ds dataset.
-It achieves the following results on the evaluation set:
-- Loss: 1.5401
+- **Base Architecture**: Qwen3-1.7B
+- **Parameters**: 1.7B
+- **Training Stage**: Multimodal Continual Pretraining only (not instruction-tuned)
+- **Training Tokens**: 304.4B
+- **Context Length**: 8,192 tokens
+- **Tokenizer**: Extended Qwen3 vocabulary with:
+  - 11,294 joint molecular 3D tokens (composed from SELFIES atom × MolStrucTok codes)
+  - 4,096 protein 3D tokens (GCP-VQVAE codebook)
+  - 26 protein 1D tokens (amino acids + non-standard/unknown)
+  - SELFIES atom tokens and modality-specific control tokens
 
-## Model description
+### Embedding Initialization
 
-More information needed
+New vocabulary entries are initialized via a **description-based scheme**: each new token is grounded in the pretrained Qwen3 embedding space by averaging the embeddings of the subword tokens of a short natural-language description (e.g., `<A_W>` → "Tryptophan"), plus a small isotropic Gaussian perturbation to break symmetry. This provides a more stable starting point than random initialization.
 
-## Intended uses & limitations
+## Pretraining Corpus (304.4B tokens)
 
-More information needed
+| Category | Tokens | Sources |
+|----------|--------|---------|
+| **Text** (105.3B) | General: 25.6B | FineWeb-Edu |
+| | Scientific: 79.7B | FineFineWeb (biology/chemistry/medical/health), PubMed Full Articles |
+| **Molecule** (73.7B) | 1D: 36.0B | PubChem, MolTextNet |
+| | 3D: 17.6B | PubChem, PCQM4Mv2, PubChemQC |
+| | Other: 24.0B | (text descriptions, properties, IUPAC names) |
+| **Protein** (77.4B) | 1D: 17.1B | UniRef50 |
+| | 3D: 38.5B | RCSB PDB, AlphaFold DB |
+| | Other: 19.5B | Swiss-Prot, TrEMBL annotations |
+| | Other (additional): 2.9B | |
+| **Cross-entity** (48.0B) | Interleaved Text: 17.1B | PubMed, bioRxiv, S2ORC, USPTO |
+| | 3D: 11.4B | CrossDocked, PPIRef |
+| | Other: 19.5B | BindingDB, STITCH, jglaser, AlphaSeq |
 
-## Training and evaluation data
+### Training Configuration
 
-More information needed
+- **Framework**: LLaMA-Factory
+- **Hardware**: 64 NVIDIA H100 GPUs
+- **Global Batch Size**: 1,024
+- **Maximum Sequence Length**: 8,192 tokens
+- **Optimizer**: AdamW
+- **Peak Learning Rate**: 2.0 × 10⁻⁴ (cosine schedule)
+- **Warmup Steps**: 2,000
+- **Total Steps**: ~36.4K (1 epoch over the full 304.4B-token corpus)
 
-## Training procedure
+## Intended Use
 
-### Training hyperparameters
+This **Base model is not instruction-tuned**. It is suitable for:
 
-The following hyperparameters were used during training:
-- learning_rate: 0.0002
-- train_batch_size: 2
-- eval_batch_size: 2
-- seed: 0
-- distributed_type: multi-GPU
-- num_devices: 64
-- gradient_accumulation_steps: 4
-- total_train_batch_size: 512
-- total_eval_batch_size: 128
-- optimizer: Use adamw_torch with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
-- lr_scheduler_type: cosine_with_min_lr
-- lr_scheduler_warmup_steps: 2000
-- num_epochs: 1
+- **Further fine-tuning** on custom biological tasks
+- **Continued pretraining** on domain-specific corpora
+- **Research on representation learning** across biomolecular modalities
+- **Embedding extraction** for downstream classification/regression tasks
 
-### Training results
+For ready-to-use instruction-following capabilities (e.g., molecule captioning, protein design, property prediction), please use the [SFT variant](https://huggingface.co/QizhiPei/BioMatrix-1.7B-SFT).
 
-| Training Loss | Epoch  | Step  | Validation Loss |
-|:-------------:|:------:|:-----:|:---------------:|
-| 1.8156        | 0.1373 | 10000 | 1.8109          |
-| 1.7268        | 0.2746 | 20000 | 1.7244          |
-| 1.6801        | 0.4119 | 30000 | 1.6486          |
-| 1.6083        | 0.5492 | 40000 | 1.6094          |
-| 1.5921        | 0.6865 | 50000 | 1.5779          |
-| 1.5569        | 0.8238 | 60000 | 1.5552          |
-| 1.5478        | 0.9611 | 70000 | 1.5421          |
+## Quick Start
 
+```python
+from transformers import AutoModelForCausalLM, AutoTokenizer
 
-### Framework versions
+model_name = "QizhiPei/BioMatrix-1.7B-Base"
+tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
+model = AutoModelForCausalLM.from_pretrained(
+    model_name,
+    torch_dtype="auto",
+    device_map="auto",
+    trust_remote_code=True
+)
+
+# Example: Continue a SMILES sequence
+prompt = "<|mol_smi_start|>CC(=O)"
+inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
+outputs = model.generate(**inputs, max_new_tokens=64)
+print(tokenizer.decode(outputs[0], skip_special_tokens=False))
+```
+
+## Modality Wrapping
+
+When constructing inputs, biomolecular content must be wrapped with the corresponding control tokens:
+
+| Modality | Wrapping Example |
+|----------|------------------|
+| Molecule SMILES | `<\|mol_smi_start\|>CC#CC#N<\|mol_smi_end\|>` |
+| Molecule SELFIES | `<\|mol_sfi_start\|>[C][#C][C][#N]<\|mol_sfi_end\|>` |
+| Molecule 3D | `<\|mol_3d_start\|>[H_3][C_0][#C_6]...<\|mol_3d_end\|>` |
+| Protein 1D | `<\|prot_aa_start\|><A_M><A_R><A_A>...<\|prot_aa_end\|>` |
+| Protein 3D | `<\|prot_3d_start\|><S_4012><S_153>...<\|prot_3d_end\|>` |
+
+Natural language text is left unwrapped and serves as the default carrier modality.
+
+## Limitations
+
+- This model is **not instruction-tuned** and is unlikely to follow natural-language instructions out-of-the-box. Use the SFT variant for instruction-following.
+- Molecular and protein 3D structures are tokenized in **disjoint geometric reference frames**, so the model cannot natively represent biomolecular complexes (e.g., docking poses).
+- Heavy domain specialization may erode some general-purpose language capabilities of the underlying Qwen3 backbone.
+- Coverage is limited to **small molecules and proteins**; nucleic acids, carbohydrates, and lipids are not currently supported.
+
+## Citation
+
+If you find BioMatrix useful, please cite:
+
+```bibtex
+@article{pei2026biomatrix,
+  title={BioMatrix: Towards a Comprehensive Biological Foundation Model Spanning the Modality Matrix of Sequences, Structures, and Language},
+  author={Pei, Qizhi and Zhou, Zhimeng and Duan, Yi and Zhao, Yiyang and He, Liang and Hsieh, Chang-Yu and He, Conghui and Yan, Rui and Wu, Lijun},
+  year={2026}
+}
+```
+
+## License
 
-- Transformers 4.51.3
-- Pytorch 2.6.0+cu124
-- Datasets 3.5.0
-- Tokenizers 0.21.1
+This model is released under the Apache 2.0 license. The base model (Qwen3-1.7B) is subject to its own license terms.