| ---
|
| license: mit
|
| base_model: hpcai-tech/openmoe-base
|
| tags:
|
| - finance
|
| - mixture-of-experts
|
| - openmoe
|
| - umt5
|
| language:
|
| - en
|
| ---
|
|
|
| # Meridian.AI
|
|
|
| Meridian.AI is an experimental finance-focused sparse Mixture-of-Experts (MoE) causal language model trained via continual updates.
|
|
|
| ## Intended use
|
|
|
| Use this model for:
|
|
|
| - Financial Q&A style prompting
|
| - Basic quantitative finance explanations
|
| - Summarization/classification-style finance text tasks
|
|
|
| ## Base model + tokenizer
|
|
|
| - **Base model weights**: `hpcai-tech/openmoe-base`
|
| - **Tokenizer**: `google/umt5-small` (256k vocab, SentencePiece/umT5)
|
|
|
| This repo includes a working umT5 tokenizer at the root so `AutoTokenizer.from_pretrained("MeridianAlgo/MeridianAI")` works.
|
|
|
| ## How to use (Transformers)
|
|
|
| The model weights are stored under the `checkpoint/` subfolder.
|
|
|
| ```python
|
| import torch
|
| from transformers import AutoModelForCausalLM, AutoTokenizer
|
|
|
| repo_id = "MeridianAlgo/MeridianAI"
|
|
|
| tokenizer = AutoTokenizer.from_pretrained(repo_id)
|
| model = AutoModelForCausalLM.from_pretrained(
|
| repo_id,
|
| subfolder="checkpoint",
|
| trust_remote_code=True,
|
| torch_dtype=torch.float32,
|
| low_cpu_mem_usage=True,
|
| ignore_mismatched_sizes=True,
|
| )
|
| model.eval()
|
|
|
| prompt = """### Instruction:
|
| Explain what a P/E ratio is and how it is used.
|
|
|
| ### Response:
|
| """
|
|
|
| inputs = tokenizer(prompt, return_tensors="pt")
|
| out = model.generate(
|
| **inputs,
|
| max_new_tokens=128,
|
| do_sample=True,
|
| temperature=0.85,
|
| top_p=0.92,
|
| repetition_penalty=1.25,
|
| no_repeat_ngram_size=3,
|
| pad_token_id=tokenizer.pad_token_id,
|
| eos_token_id=tokenizer.eos_token_id,
|
| )
|
|
|
| print(tokenizer.decode(out[0], skip_special_tokens=True))
|
| ```
|
|
|
| ## Training data
|
|
|
| Training uses streaming mixes of finance datasets (FinanceMTEB family) plus optional larger corpora depending on environment configuration.
|
|
|
| ## Limitations
|
|
|
| - This is a continually trained experimental model and may exhibit repetition.
|
| - Not financial advice.
|
| - Outputs may be incorrect or outdated.
|
|
|
| ## Source code
|
|
|
| Training pipeline and scripts live in the GitHub repo: https://github.com/MeridianAlgo/FinAI
|
|
|
