| --- |
| license: mit |
| datasets: |
| - Harley-ml/HFMC |
| language: |
| - en |
| tags: |
| - mcod |
| - configgen |
| - model-config-generation |
| - json |
| - small |
| - small-language-model |
| - config-generation |
| - json-generation |
| - harley-ml |
| --- |
| |
| # MCOD |
|
|
| MCOD, which stands for "Model Configs on Drugs," is a 4.7M parameter model trained on 7.1M tokens of Hugging Face model configs. |
|
|
| We are well aware that 7.1M tokens is below the Chinchilla optimal target, but including more tokens wouldn't improve diversity. For example, after cleaning the full 90M token dataset, we were left with 7.1M tokens after deduplication (over 13k docs) and filtering (by language and length). |
|
|
| MCOD generates plausible-looking configs with the correct hyperparameters per model family. |
|
|
| ## Architecture |
|
|
| | Parameter | Value | |
| |-------------------------|-------| |
| | hidden_size | 256 | |
| | num_hidden_layers | 4 | |
| | num_attention_heads | 4 | |
| | num_key_value_heads | 4 | |
| | intermediate_size | 1024 | |
| | max_position_embeddings | 1024 | |
| | rope_theta | 100000.0 | |
| | tie_word_embeddings | true | |
|
|
| MCOD uses the Qwen3 architecture. |
|
|
| ## Training |
|
|
| MCOD was trained on 18k entries, 7.1M tokens, and 1M words. |
|
|
| ### Hardware |
|
|
| MCOD was trained on one NVIDIA RTX 2060 6GB for 3 epochs with a batch size of 8. |
|
|
| ### Training Results |
| | Step | Epoch | Train Loss | Train PPL | Eval Loss | Eval PPL | |
| |------|-------|------------|-----------|-----------|----------| |
| | 200 | 0.44 | 4.1022 | 60.53 | 1.8218 | 6.18 | |
| | 400 | 0.88 | 1.0227 | 2.78 | 0.5671 | 1.76 | |
| | 600 | 1.33 | 0.5434 | 1.72 | 0.3560 | 1.43 | |
| | 800 | 1.77 | 0.3978 | 1.49 | 0.2939 | 1.34 | |
| | 1000 | 2.21 | 0.3486 | 1.42 | 0.2514 | 1.29 | |
| | 1200 | 2.65 | 0.2944 | 1.34 | 0.2259 | 1.25 | |
|
|
|  |
|
|
| ## Generations |
|
|
| Prompt: `{"model_type": "gpt2"` |
|
|
| Output: |
| ``` |
| , "n_ctx": 1024, "n_embd": 768, "n_head": 12, "n_inner": null, "n_layer": 6, "n_positions": 1024, "reorder_and_upcast_attn": false, "resid_pdrop": 0.1, "scale_attn_by_inverse_layer_idx": false, "scale_attn_weights": true, "summary_activation": null, "summary_first_dropout": 0.1, "summary_proj_to_labels": true, "summary_type": "cls_index", "summary_use_proj": true, "task_specific_params": {"text-generation": {"do_sample": true, "max_length": 50}}, "torch_dtype": "float32", "transformers_version": "4.25.1", "use_cache": true, "vocab_size": 50257} |
| ``` |
|
|
| Prompt: `{` |
|
|
| Output: |
| ``` |
| "architectures": ["RobertaForSequenceClassification"], "attention_probs_dropout_prob": 0.1, "bos_token_id": 0, "classifier_dropout": null, "eos_token_id": 2, "hidden_act": "gelu", "hidden_dropout_prob": 0.1, "hidden_size": 768, "id2label": {"0": "LABEL_0"}, "initializer_range": 0.02, "intermediate_size": 3072, "label2id": {"LABEL_0": 0}, "layer_norm_eps": 1e-05, "max_position_embeddings": 514, "model_type": "roberta", "num_attention_heads": 12, "num_hidden_layers": 6, "pad_token_id": 1, "position_embedding_type": "absolute", "problem_type": "single_label_classification", "torch_dtype": "float32", "transformers_version": "4.11.3", "type_vocab_size": 1, "use_cache": true, "vocab_size": 50265} |
| ``` |
|
|
| Prompt: `The cat crossed` |
|
|
| Output: |
| ``` |
| , "architectures": ["BertForSequenceClassification"], "attention_probs_dropout_prob": 0.1, "classifier_dropout": null, "hidden_act": "gelu", "hidden_dropout_prob": 0.1, "hidden_size": 768, "id2label": {"0": "anger", "1": "fear", "2": "surprise"}, "initializer_range": 0.02, "intermediate_size": 3072, "label2id": {"sadness": 0, "neutral": 1, "joy": 2}, "layer_norm_eps": 1e-12, "max_position_embeddings": 512, "model_type": "bert", "num_attention_heads": 12, "num_hidden_layers": 12, "pad_token_id": 0, "position_embedding_type": "absolute", "problem_type": "single_label_classification", "torch_dtype": "float32", "transformers_version": "4.11.3", "type_vocab_size": 2, "use_cache": true, "vocab_size": 30522} |
| ``` |
|
|
| ## Limitations |
|
|
| 1. Only generates model configs |
| 2. Cannot converse or reason |
| 3. Most unconditionally generated configs are BERT- or BART-centered |
|
|
| ## Use Cases |
|
|
| 1. Educational research |
| 2. JSON modeling |
| 3. Generating synthetic configs for pretraining or fine-tuning datasets (be careful; the model hallucinates a lot) |
| 4. Or, more simply, for fun. |
|
|
| ## Inference |
|
|
| ```python |
| # ============================================================================= |
| # Inference |
| # ============================================================================= |
| |
| MODEL_DIR = "Harley-ml/MCOD-4.7M" # path |
| TOKENIZER_PATH = MODEL_DIR |
| |
| # --- Generation settings --- |
| PROMPT = "{" # prompt |
| MAX_NEW_TOKENS = 1024 |
| TEMPERATURE = 0.7 |
| TOP_P = 0.95 |
| TOP_K = 50 |
| REPETITION_PENALTY = 1.1 |
| DO_SAMPLE = True |
| |
| # ============================================================================= |
| |
| import torch |
| from pathlib import Path |
| from transformers import ( |
| AutoModelForCausalLM, |
| PreTrainedTokenizerFast, |
| AddedToken, |
| ) |
| |
| # --------------------------------------------------------------------------- |
| # Device |
| # --------------------------------------------------------------------------- |
| |
| device = ( |
| "cuda" if torch.cuda.is_available() else |
| "mps" if torch.backends.mps.is_available() else |
| "cpu" |
| ) |
| print(f"Device : {device}") |
| |
| # --------------------------------------------------------------------------- |
| # Tokenizer (mirrors training setup) |
| # --------------------------------------------------------------------------- |
| |
| def load_tokenizer(path: str): |
| p = Path(path).resolve() |
| if not p.exists(): |
| raise FileNotFoundError(f"Tokenizer not found: {p}") |
| tok = PreTrainedTokenizerFast(tokenizer_file=str(p)) |
| specials = {} |
| if tok.bos_token is None: specials["bos_token"] = AddedToken("<|bos|>", special=True) |
| if tok.eos_token is None: specials["eos_token"] = AddedToken("<|eos|>", special=True) |
| if tok.unk_token is None: specials["unk_token"] = AddedToken("<|unk|>", special=True) |
| if tok.pad_token is None: |
| if tok.eos_token is not None: |
| tok.pad_token = tok.eos_token |
| else: |
| specials["pad_token"] = AddedToken("<|pad|>", special=True) |
| if specials: |
| tok.add_special_tokens(specials) |
| tok.padding_side = "left" # left-pad for batched generation |
| return tok |
| |
| print("Loading tokenizer...") |
| tokenizer = load_tokenizer(TOKENIZER_PATH) |
| print(f" Vocab size : {tokenizer.vocab_size}") |
| print(f" BOS : {tokenizer.bos_token!r}") |
| print(f" EOS : {tokenizer.eos_token!r}") |
| print(f" PAD : {tokenizer.pad_token!r} (id={tokenizer.pad_token_id})") |
| |
| # --------------------------------------------------------------------------- |
| # Model |
| # --------------------------------------------------------------------------- |
| |
| print(f"\nLoading model from {MODEL_DIR} ...") |
| model = AutoModelForCausalLM.from_pretrained( |
| MODEL_DIR, |
| dtype=torch.float16 if device == "cuda" else torch.float32, |
| low_cpu_mem_usage=True, |
| ) |
| model.eval() |
| model.to(device) |
| |
| total_params = sum(p.numel() for p in model.parameters()) |
| print(f" Parameters : {total_params:,}") |
| |
| # --------------------------------------------------------------------------- |
| # Generation helper |
| # --------------------------------------------------------------------------- |
| |
| def generate( |
| prompt: str = PROMPT, |
| max_new_tokens: int = MAX_NEW_TOKENS, |
| temperature: float = TEMPERATURE, |
| top_p: float = TOP_P, |
| top_k: int = TOP_K, |
| repetition_penalty: float = REPETITION_PENALTY, |
| do_sample: bool = DO_SAMPLE, |
| ) -> str: |
| |
| bos = tokenizer.bos_token or "" |
| full_prompt = bos + prompt |
| |
| inputs = tokenizer( |
| full_prompt, |
| return_tensors="pt", |
| add_special_tokens=False, |
| ).to(device) |
| inputs.pop("token_type_ids", None) # Qwen3 doesn't use this |
| |
| gen_kwargs = dict( |
| max_new_tokens = max_new_tokens, |
| do_sample = do_sample, |
| repetition_penalty = repetition_penalty, |
| eos_token_id = tokenizer.eos_token_id, |
| pad_token_id = tokenizer.pad_token_id, |
| ) |
| if do_sample: |
| gen_kwargs["temperature"] = temperature |
| gen_kwargs["top_p"] = top_p |
| gen_kwargs["top_k"] = top_k |
| |
| with torch.inference_mode(): |
| output_ids = model.generate(**inputs, **gen_kwargs) |
| |
| # Strip the prompt tokens so we only return what was generated |
| prompt_len = inputs["input_ids"].shape[-1] |
| new_ids = output_ids[0][prompt_len:] |
| return tokenizer.decode(new_ids, skip_special_tokens=True) |
| |
| |
| # --------------------------------------------------------------------------- |
| # Run |
| # --------------------------------------------------------------------------- |
| |
| if __name__ == "__main__": |
| print(f"\nPrompt : {PROMPT!r}") |
| print("-" * 60) |
| |
| output = generate(PROMPT) |
| |
| print("Generated:") |
| print(output) |
| ``` |
|
|
| ## Citation |
|
|
| ```bibtex |
| @misc{mcod-4.7m, |
| title = {MCOD-4.7M: Low Entropy Training; Hugging Face Model Configs}, |
| author = {Harley-ml}, |
| year = {2026}, |
| url = {https://huggingface.co/Harley-ml/MCOD-4.7M} |
| } |
| ``` |
