| --- |
| license: apache-2.0 |
| language: |
| - multilingual |
| - en |
| - ru |
| - es |
| - fr |
| - de |
| - it |
| - pt |
| - pl |
| - nl |
| - vi |
| - tr |
| - sv |
| - id |
| - ro |
| - cs |
| - zh |
| - hu |
| - ja |
| - th |
| - fi |
| - fa |
| - uk |
| - da |
| - el |
| - "no" |
| - bg |
| - sk |
| - ko |
| - ar |
| - lt |
| - ca |
| - sl |
| - he |
| - et |
| - lv |
| - hi |
| - sq |
| - ms |
| - az |
| - sr |
| - ta |
| - hr |
| - kk |
| - is |
| - ml |
| - mr |
| - te |
| - af |
| - gl |
| - fil |
| - be |
| - mk |
| - eu |
| - bn |
| - ka |
| - mn |
| - bs |
| - uz |
| - ur |
| - sw |
| - yue |
| - ne |
| - kn |
| - kaa |
| - gu |
| - si |
| - cy |
| - eo |
| - la |
| - hy |
| - ky |
| - tg |
| - ga |
| - mt |
| - my |
| - km |
| - tt |
| - so |
| - ku |
| - ps |
| - pa |
| - rw |
| - lo |
| - ha |
| - dv |
| - fy |
| - lb |
| - ckb |
| - mg |
| - gd |
| - am |
| - ug |
| - ht |
| - grc |
| - hmn |
| - sd |
| - jv |
| - mi |
| - tk |
| - ceb |
| - yi |
| - ba |
| - fo |
| - or |
| - xh |
| - su |
| - kl |
| - ny |
| - sm |
| - sn |
| - co |
| - zu |
| - ig |
| - yo |
| - pap |
| - st |
| - haw |
| - as |
| - oc |
| - cv |
| - lus |
| - tet |
| - gsw |
| - sah |
| - br |
| - rm |
| - sa |
| - bo |
| - om |
| - se |
| - ce |
| - cnh |
| - ilo |
| - hil |
| - udm |
| - os |
| - lg |
| - ti |
| - vec |
| - ts |
| - tyv |
| - kbd |
| - ee |
| - iba |
| - av |
| - kha |
| - to |
| - tn |
| - nso |
| - fj |
| - zza |
| - ak |
| - ada |
| - otq |
| - dz |
| - bua |
| - cfm |
| - ln |
| - chm |
| - gn |
| - krc |
| - wa |
| - hif |
| - yua |
| - srn |
| - war |
| - rom |
| - bik |
| - pam |
| - sg |
| - lu |
| - ady |
| - kbp |
| - syr |
| - ltg |
| - myv |
| - iso |
| - kac |
| - bho |
| - ay |
| - kum |
| - qu |
| - za |
| - pag |
| - ngu |
| - ve |
| - pck |
| - zap |
| - tyz |
| - hui |
| - bbc |
| - tzo |
| - tiv |
| - ksd |
| - gom |
| - min |
| - ang |
| - nhe |
| - bgp |
| - nzi |
| - nnb |
| - nv |
| - zxx |
| - bci |
| - kv |
| - new |
| - mps |
| - alt |
| - meu |
| - bew |
| - fon |
| - iu |
| - abt |
| - mgh |
| - mnw |
| - tvl |
| - dov |
| - tlh |
| - ho |
| - kw |
| - mrj |
| - meo |
| - crh |
| - mbt |
| - emp |
| - ace |
| - ium |
| - mam |
| - gym |
| - mai |
| - crs |
| - pon |
| - ubu |
| - fip |
| - quc |
| - gv |
| - kj |
| - btx |
| - ape |
| - chk |
| - rcf |
| - shn |
| - tzh |
| - mdf |
| - ppk |
| - ss |
| - gag |
| - cab |
| - kri |
| - seh |
| - ibb |
| - tbz |
| - bru |
| - enq |
| - ach |
| - cuk |
| - kmb |
| - wo |
| - kek |
| - qub |
| - tab |
| - bts |
| - kos |
| - rwo |
| - cak |
| - tuc |
| - bum |
| - cjk |
| - gil |
| - stq |
| - tsg |
| - quh |
| - mak |
| - arn |
| - ban |
| - jiv |
| - sja |
| - yap |
| - tcy |
| - toj |
| - twu |
| - xal |
| - amu |
| - rmc |
| - hus |
| - nia |
| - kjh |
| - bm |
| - guh |
| - mas |
| - acf |
| - dtp |
| - ksw |
| - bzj |
| - din |
| - zne |
| - mad |
| - msi |
| - mag |
| - mkn |
| - kg |
| - lhu |
| - ch |
| - qvi |
| - mh |
| - djk |
| - sus |
| - mfe |
| - srm |
| - dyu |
| - ctu |
| - gui |
| - pau |
| - inb |
| - bi |
| - mni |
| - guc |
| - jam |
| - wal |
| - jac |
| - bas |
| - gor |
| - skr |
| - nyu |
| - noa |
| - sda |
| - gub |
| - nog |
| - cni |
| - teo |
| - tdx |
| - sxn |
| - rki |
| - nr |
| - frp |
| - alz |
| - taj |
| - lrc |
| - cce |
| - rn |
| - jvn |
| - hvn |
| - nij |
| - dwr |
| - izz |
| - msm |
| - bus |
| - ktu |
| - chr |
| - maz |
| - tzj |
| - suz |
| - knj |
| - bim |
| - gvl |
| - bqc |
| - tca |
| - pis |
| - prk |
| - laj |
| - mel |
| - qxr |
| - niq |
| - ahk |
| - shp |
| - hne |
| - spp |
| - koi |
| - krj |
| - quf |
| - luz |
| - agr |
| - tsc |
| - mqy |
| - gof |
| - gbm |
| - miq |
| - dje |
| - awa |
| - bjj |
| - qvz |
| - sjp |
| - tll |
| - raj |
| - kjg |
| - bgz |
| - quy |
| - cbk |
| - akb |
| - oj |
| - ify |
| - mey |
| - ks |
| - cac |
| - brx |
| - qup |
| - syl |
| - jax |
| - ff |
| - ber |
| - tks |
| - trp |
| - mrw |
| - adh |
| - smt |
| - srr |
| - ffm |
| - qvc |
| - mtr |
| - ann |
| - kaa |
| - aa |
| - noe |
| - nut |
| - gyn |
| - kwi |
| - xmm |
| - msb |
| library_name: transformers |
| tags: |
| - text2text-generation |
| - text-generation-inference |
| datasets: |
| - allenai/MADLAD-400 |
| pipeline_tag: translation |
|
|
| --- |
| |
| # Model Card for MADLAD-400-3B-MT |
|
|
| # Table of Contents |
|
|
| 0. [TL;DR](#TL;DR) |
| 1. [Model Details](#model-details) |
| 2. [Usage](#usage) |
| 3. [Uses](#uses) |
| 4. [Bias, Risks, and Limitations](#bias-risks-and-limitations) |
| 5. [Training Details](#training-details) |
| 6. [Evaluation](#evaluation) |
| 7. [Environmental Impact](#environmental-impact) |
| 8. [Citation](#citation) |
|
|
| # TL;DR |
|
|
| MADLAD-400-3B-MT is a multilingual machine translation model based on the T5 architecture that was |
| trained on 1 trillion tokens covering over 450 languages using publicly available data. |
| It is competitive with models that are significantly larger. |
|
|
| **Disclaimer**: [Juarez Bochi](https://huggingface.co/jbochi), who was not involved in this research, converted |
| the original weights and wrote the contents of this model card based on the original paper and Flan-T5. |
|
|
| # Model Details |
|
|
| ## Model Description |
|
|
| - **Model type:** Language model |
| - **Language(s) (NLP):** Multilingual (400+ languages) |
| - **License:** Apache 2.0 |
| - **Related Models:** [All MADLAD-400 Checkpoints](https://huggingface.co/models?search=madlad) |
| - **Original Checkpoints:** [All Original MADLAD-400 Checkpoints](https://github.com/google-research/google-research/tree/master/madlad_400) |
| - **Resources for more information:** |
| - [Research paper](https://arxiv.org/abs/2309.04662) |
| - [GitHub Repo](https://github.com/google-research/t5x) |
| - [Hugging Face MADLAD-400 Docs (Similar to T5) ](https://huggingface.co/docs/transformers/model_doc/MADLAD-400) - [Pending PR](https://github.com/huggingface/transformers/pull/27471) |
|
|
| # Usage |
|
|
| Find below some example scripts on how to use the model: |
|
|
| ## Using the Pytorch model with `transformers` |
|
|
| ### Running the model on a CPU or GPU |
|
|
| <details> |
| <summary> Click to expand </summary> |
|
|
| First, install the Python packages that are required: |
|
|
| `pip install transformers accelerate sentencepiece` |
|
|
| ```python |
| from transformers import T5ForConditionalGeneration, T5Tokenizer |
| |
| model_name = 'jbochi/madlad400-3b-mt' |
| model = T5ForConditionalGeneration.from_pretrained(model_name, device_map="auto") |
| tokenizer = T5Tokenizer.from_pretrained(model_name) |
| |
| text = "<2pt> I love pizza!" |
| input_ids = tokenizer(text, return_tensors="pt").input_ids.to(model.device) |
| outputs = model.generate(input_ids=input_ids) |
| |
| tokenizer.decode(outputs[0], skip_special_tokens=True) |
| # Eu adoro pizza! |
| ``` |
|
|
| </details> |
|
|
| ## Running the model with Candle |
|
|
| <details> |
| <summary> Click to expand </summary> |
|
|
| Usage with [candle](https://github.com/huggingface/candle): |
|
|
| ```bash |
| $ cargo run --example t5 --release -- \ |
| --model-id "jbochi/madlad400-3b-mt" \ |
| --prompt "<2de> How are you, my friend?" \ |
| --decode --temperature 0 |
| ``` |
|
|
| We also provide a quantized model (1.65 GB vs the original 11.8 GB file): |
|
|
| ``` |
| cargo run --example quantized-t5 --release -- \ |
| --model-id "jbochi/madlad400-3b-mt" --weight-file "model-q4k.gguf" \ |
| --prompt "<2de> How are you, my friend?" \ |
| --temperature 0 |
| ... |
| Wie geht es dir, mein Freund? |
| ``` |
|
|
| </details> |
|
|
|
|
| # Uses |
|
|
| ## Direct Use and Downstream Use |
|
|
| > Primary intended uses: Machine Translation and multilingual NLP tasks on over 400 languages. |
| > Primary intended users: Research community. |
|
|
| ## Out-of-Scope Use |
|
|
| > These models are trained on general domain data and are therefore not meant to |
| > work on domain-specific models out-of-the box. Moreover, these research models have not been assessed |
| > for production usecases. |
|
|
| # Bias, Risks, and Limitations |
|
|
| > We note that we evaluate on only 204 of the languages supported by these models and on machine translation |
| > and few-shot machine translation tasks. Users must consider use of this model carefully for their own |
| > usecase. |
|
|
| ## Ethical considerations and risks |
|
|
| > We trained these models with MADLAD-400 and publicly available data to create baseline models that |
| > support NLP for over 400 languages, with a focus on languages underrepresented in large-scale corpora. |
| > Given that these models were trained with web-crawled datasets that may contain sensitive, offensive or |
| > otherwise low-quality content despite extensive preprocessing, it is still possible that these issues to the |
| > underlying training data may cause differences in model performance and toxic (or otherwise problematic) |
| > output for certain domains. Moreover, large models are dual use technologies that have specific risks |
| > associated with their use and development. We point the reader to surveys such as those written by |
| > Weidinger et al. or Bommasani et al. for a more detailed discussion of these risks, and to Liebling |
| > et al. for a thorough discussion of the risks of machine translation systems. |
|
|
| ## Known Limitations |
|
|
| More information needed |
|
|
| ## Sensitive Use: |
|
|
| More information needed |
|
|
| # Training Details |
|
|
| > We train models of various sizes: a 3B, 32-layer parameter model, |
| > a 7.2B 48-layer parameter model and a 10.7B 32-layer parameter model. |
| > We share all parameters of the model across language pairs, |
| > and use a Sentence Piece Model with 256k tokens shared on both the encoder and decoder |
| > side. Each input sentence has a <2xx> token prepended to the source sentence to indicate the target |
| > language. |
|
|
| See the [research paper](https://arxiv.org/pdf/2309.04662.pdf) for further details. |
|
|
| ## Training Data |
|
|
| > For both the machine translation and language model, MADLAD-400 is used. For the machine translation |
| > model, a combination of parallel datasources covering 157 languages is also used. Further details are |
| > described in the [paper](https://arxiv.org/pdf/2309.04662.pdf). |
|
|
| ## Training Procedure |
|
|
| See the [research paper](https://arxiv.org/pdf/2309.04662.pdf) for further details. |
|
|
| # Evaluation |
|
|
| ## Testing Data, Factors & Metrics |
|
|
| > For evaluation, we used WMT, NTREX, Flores-200 and Gatones datasets as described in Section 4.3 in the [paper](https://arxiv.org/pdf/2309.04662.pdf). |
|
|
| > The translation quality of this model varies based on language, as seen in the paper, and likely varies on |
| > domain, though we have not assessed this. |
|
|
| ## Results |
|
|
|  |
|
|
|  |
|
|
|  |
|
|
| See the [research paper](https://arxiv.org/pdf/2309.04662.pdf) for further details. |
|
|
| # Environmental Impact |
|
|
| More information needed |
|
|
| # Citation |
|
|
| **BibTeX:** |
|
|
| ```bibtex |
| @misc{kudugunta2023madlad400, |
| title={MADLAD-400: A Multilingual And Document-Level Large Audited Dataset}, |
| author={Sneha Kudugunta and Isaac Caswell and Biao Zhang and Xavier Garcia and Christopher A. Choquette-Choo and Katherine Lee and Derrick Xin and Aditya Kusupati and Romi Stella and Ankur Bapna and Orhan Firat}, |
| year={2023}, |
| eprint={2309.04662}, |
| archivePrefix={arXiv}, |
| primaryClass={cs.CL} |
| } |
| ``` |
|
|
|
|
