--- license: mit datasets: - Harley-ml/HFMC language: - en tags: - mcod - configgen - model-config-generation - json - small - small-language-model - config-generation - json-generation - harley-ml --- # MCOD MCOD, which stands for "Model Configs on Drugs," is a 4.7M parameter model trained on 7.1M tokens of Hugging Face model configs. We are well aware that 7.1M tokens is below the Chinchilla optimal target, but including more tokens wouldn't improve diversity. For example, after cleaning the full 90M token dataset, we were left with 7.1M tokens after deduplication (over 13k docs) and filtering (by language and length). MCOD generates plausible-looking configs with the correct hyperparameters per model family. ## Architecture | Parameter | Value | |-------------------------|-------| | hidden_size | 256 | | num_hidden_layers | 4 | | num_attention_heads | 4 | | num_key_value_heads | 4 | | intermediate_size | 1024 | | max_position_embeddings | 1024 | | rope_theta | 100000.0 | | tie_word_embeddings | true | MCOD uses the Qwen3 architecture. ## Training MCOD was trained on 18k entries, 7.1M tokens, and 1M words. ### Hardware MCOD was trained on one NVIDIA RTX 2060 6GB for 3 epochs with a batch size of 8. ### Training Results | Step | Epoch | Train Loss | Train PPL | Eval Loss | Eval PPL | |------|-------|------------|-----------|-----------|----------| | 200 | 0.44 | 4.1022 | 60.53 | 1.8218 | 6.18 | | 400 | 0.88 | 1.0227 | 2.78 | 0.5671 | 1.76 | | 600 | 1.33 | 0.5434 | 1.72 | 0.3560 | 1.43 | | 800 | 1.77 | 0.3978 | 1.49 | 0.2939 | 1.34 | | 1000 | 2.21 | 0.3486 | 1.42 | 0.2514 | 1.29 | | 1200 | 2.65 | 0.2944 | 1.34 | 0.2259 | 1.25 | ![Training and Evaluation Curves](images/training_graph.png) ## Generations Prompt: `{"model_type": "gpt2"` Output: ``` , "n_ctx": 1024, "n_embd": 768, "n_head": 12, "n_inner": null, "n_layer": 6, "n_positions": 1024, "reorder_and_upcast_attn": false, "resid_pdrop": 0.1, "scale_attn_by_inverse_layer_idx": false, "scale_attn_weights": true, "summary_activation": null, "summary_first_dropout": 0.1, "summary_proj_to_labels": true, "summary_type": "cls_index", "summary_use_proj": true, "task_specific_params": {"text-generation": {"do_sample": true, "max_length": 50}}, "torch_dtype": "float32", "transformers_version": "4.25.1", "use_cache": true, "vocab_size": 50257} ``` Prompt: `{` Output: ``` "architectures": ["RobertaForSequenceClassification"], "attention_probs_dropout_prob": 0.1, "bos_token_id": 0, "classifier_dropout": null, "eos_token_id": 2, "hidden_act": "gelu", "hidden_dropout_prob": 0.1, "hidden_size": 768, "id2label": {"0": "LABEL_0"}, "initializer_range": 0.02, "intermediate_size": 3072, "label2id": {"LABEL_0": 0}, "layer_norm_eps": 1e-05, "max_position_embeddings": 514, "model_type": "roberta", "num_attention_heads": 12, "num_hidden_layers": 6, "pad_token_id": 1, "position_embedding_type": "absolute", "problem_type": "single_label_classification", "torch_dtype": "float32", "transformers_version": "4.11.3", "type_vocab_size": 1, "use_cache": true, "vocab_size": 50265} ``` Prompt: `The cat crossed` Output: ``` , "architectures": ["BertForSequenceClassification"], "attention_probs_dropout_prob": 0.1, "classifier_dropout": null, "hidden_act": "gelu", "hidden_dropout_prob": 0.1, "hidden_size": 768, "id2label": {"0": "anger", "1": "fear", "2": "surprise"}, "initializer_range": 0.02, "intermediate_size": 3072, "label2id": {"sadness": 0, "neutral": 1, "joy": 2}, "layer_norm_eps": 1e-12, "max_position_embeddings": 512, "model_type": "bert", "num_attention_heads": 12, "num_hidden_layers": 12, "pad_token_id": 0, "position_embedding_type": "absolute", "problem_type": "single_label_classification", "torch_dtype": "float32", "transformers_version": "4.11.3", "type_vocab_size": 2, "use_cache": true, "vocab_size": 30522} ``` ## Limitations 1. Only generates model configs 2. Cannot converse or reason 3. Most unconditionally generated configs are BERT- or BART-centered ## Use Cases 1. Educational research 2. JSON modeling 3. Generating synthetic configs for pretraining or fine-tuning datasets (be careful; the model hallucinates a lot) 4. Or, more simply, for fun. ## Inference ```python # ============================================================================= # Inference # ============================================================================= MODEL_DIR = "Harley-ml/MCOD-4.7M" # path TOKENIZER_PATH = MODEL_DIR # --- Generation settings --- PROMPT = "{" # prompt MAX_NEW_TOKENS = 1024 TEMPERATURE = 0.7 TOP_P = 0.95 TOP_K = 50 REPETITION_PENALTY = 1.1 DO_SAMPLE = True # ============================================================================= import torch from pathlib import Path from transformers import ( AutoModelForCausalLM, PreTrainedTokenizerFast, AddedToken, ) # --------------------------------------------------------------------------- # Device # --------------------------------------------------------------------------- device = ( "cuda" if torch.cuda.is_available() else "mps" if torch.backends.mps.is_available() else "cpu" ) print(f"Device : {device}") # --------------------------------------------------------------------------- # Tokenizer (mirrors training setup) # --------------------------------------------------------------------------- def load_tokenizer(path: str): p = Path(path).resolve() if not p.exists(): raise FileNotFoundError(f"Tokenizer not found: {p}") tok = PreTrainedTokenizerFast(tokenizer_file=str(p)) specials = {} if tok.bos_token is None: specials["bos_token"] = AddedToken("<|bos|>", special=True) if tok.eos_token is None: specials["eos_token"] = AddedToken("<|eos|>", special=True) if tok.unk_token is None: specials["unk_token"] = AddedToken("<|unk|>", special=True) if tok.pad_token is None: if tok.eos_token is not None: tok.pad_token = tok.eos_token else: specials["pad_token"] = AddedToken("<|pad|>", special=True) if specials: tok.add_special_tokens(specials) tok.padding_side = "left" # left-pad for batched generation return tok print("Loading tokenizer...") tokenizer = load_tokenizer(TOKENIZER_PATH) print(f" Vocab size : {tokenizer.vocab_size}") print(f" BOS : {tokenizer.bos_token!r}") print(f" EOS : {tokenizer.eos_token!r}") print(f" PAD : {tokenizer.pad_token!r} (id={tokenizer.pad_token_id})") # --------------------------------------------------------------------------- # Model # --------------------------------------------------------------------------- print(f"\nLoading model from {MODEL_DIR} ...") model = AutoModelForCausalLM.from_pretrained( MODEL_DIR, dtype=torch.float16 if device == "cuda" else torch.float32, low_cpu_mem_usage=True, ) model.eval() model.to(device) total_params = sum(p.numel() for p in model.parameters()) print(f" Parameters : {total_params:,}") # --------------------------------------------------------------------------- # Generation helper # --------------------------------------------------------------------------- def generate( prompt: str = PROMPT, max_new_tokens: int = MAX_NEW_TOKENS, temperature: float = TEMPERATURE, top_p: float = TOP_P, top_k: int = TOP_K, repetition_penalty: float = REPETITION_PENALTY, do_sample: bool = DO_SAMPLE, ) -> str: bos = tokenizer.bos_token or "" full_prompt = bos + prompt inputs = tokenizer( full_prompt, return_tensors="pt", add_special_tokens=False, ).to(device) inputs.pop("token_type_ids", None) # Qwen3 doesn't use this gen_kwargs = dict( max_new_tokens = max_new_tokens, do_sample = do_sample, repetition_penalty = repetition_penalty, eos_token_id = tokenizer.eos_token_id, pad_token_id = tokenizer.pad_token_id, ) if do_sample: gen_kwargs["temperature"] = temperature gen_kwargs["top_p"] = top_p gen_kwargs["top_k"] = top_k with torch.inference_mode(): output_ids = model.generate(**inputs, **gen_kwargs) # Strip the prompt tokens so we only return what was generated prompt_len = inputs["input_ids"].shape[-1] new_ids = output_ids[0][prompt_len:] return tokenizer.decode(new_ids, skip_special_tokens=True) # --------------------------------------------------------------------------- # Run # --------------------------------------------------------------------------- if __name__ == "__main__": print(f"\nPrompt : {PROMPT!r}") print("-" * 60) output = generate(PROMPT) print("Generated:") print(output) ``` ## Citation ```bibtex @misc{mcod-4.7m, title = {MCOD-4.7M: Low Entropy Training; Hugging Face Model Configs}, author = {Harley-ml}, year = {2026}, url = {https://huggingface.co/Harley-ml/MCOD-4.7M} } ```