| # MuMath: Multi-perspective Data Augmentation for Mathematical Reasoning in Large Language Models |
|
|
| ## Introduction |
|
|
| We have amalgamated and further refined these strengths while broadening the scope of augmentation methods to construct a multi-perspective augmentation dataset for mathematics—termed [MuMath (μ-Math) Dataset](https://huggingface.co/datasets/weihao1/MuMath). Subsequently, we finetune LLaMA-2 on the MuMath dataset to derive the MuMath model. |
|
|
|
|
| | Model | Size | GSM8k | MATH | |
| |---|---|---|---| |
| | WizardMath-7B | 7B | 54.9 | 10.7 | |
| | MetaMath-7B | 7B | 66.3 | 19.7 | |
| | MuggleMath-7B | 7B | 68.4 | - | |
| | [MuMath-7B](https://huggingface.co/weihao1/MuMath-7B) | 7B | **79.1** | **30.0** | |
| || |
| | WizardMath-13B | 13B | 63.9 | 14 | |
| | MetaMath-13B | 13B | 72.3 | 22.4 | |
| | MuggleMath-13B | 13B | 74 | - |s |
| | [MuMath-13B](https://huggingface.co/weihao1/MuMath-13B) | 13B | **83.6** | **33.3** | |
| || |
| | WizardMath-70B | 70B | 81.6 | 22.7 | |
| | MetaMath-70B | 70B | 82.3 | 26.6 | |
| | MuggleMath-70B | 70B | 82.3 | - | |
| | [MuMath-70B](https://huggingface.co/weihao1/MuMath-70B) | 70B | **88.5** | **41.2** | |
|
|
| > The best results are bolded. |
|
|
|
|
| ## Augmentation Methods |
| <p align="center"> |
| <img src="./images/MuMath.png" width="800"> |
| <br> |
| <em>Overview of the augmentation methods our MuMath employs, which can be divided into four categories: (1) Data Reformulation includes solution reorganization and question rephrasing; (2) Backward Creation includes Backward-Forward Transformation (BF-Trans) and FOBAR; (3) Question Alteration includes expression replacement and difficulty enhancement; (4) Nested Multi-task construction includes data of the auxiliary tasks, i.e., Problem Outline and Solution Plan.</em> |
| </p> |
| |
|
|
| ### Setup |
|
|
| We recommend using [Conda](https://docs.conda.io/projects/miniconda) to manage your environment. We use [vLLM](https://github.com/vllm-project/vllm) to accelerate inference. Run the following commands to setup your environment: |
|
|
| ```sh |
| conda create -n mumath python=3.10 |
| conda activate mumath |
| cd MuMath-src |
| pip install -r requirements.txt |
| ``` |
|
|
|
|
| ## Training |
|
|
| We also open [MuMath Dataset](https://huggingface.co/datasets/weihao1/MuMath) for the training stage. |
|
|
| To train a model, after specifying `MODEL_PATH`, `SAVE_PATH`, `DATA_PATH`, the conda environment and so on, run the following command: |
|
|
| ```sh |
| |
| # 7B or 13B |
| bash train_7b.sh |
| |
| # 34B |
| bash train_13b_70b.sh |
| |
| ``` |
|
|
| ### Inference and Evaluation |
|
|
| We provide scripts for inference and evaluation, which are called in `train_7b.sh` and `train_13b_70b.sh` as mentioned above. |
|
|
| ```sh |
| python eval_gsm8k.py --model $SAVE_PATH --data_file ./data/test/GSM8K_test.jsonl |
| python eval_math.py --model $SAVE_PATH --data_path ./data/test/MATH_test.jsonl |
| ``` |
|
|
| ## Citation |
| Please cite the paper if you refer to our model, code, data or paper from MuMath. |
|
|
| ``` |
| @inproceedings{you-etal-2024-mumath, |
| title = "{M}u{M}ath: Multi-perspective Data Augmentation for Mathematical Reasoning in Large Language Models", |
| author = "You, Weihao and Yin, Shuo and Zhao, Xudong and Ji, Zhilong and Zhong, Guoqiang and Bai, Jinfeng", |
| booktitle = "Findings of the Association for Computational Linguistics: NAACL 2024", |
| month = jun, |
| year = "2024", |
| pages = "2932--2958", |
| } |
| ``` |
|
|
| ## Credits |
|
|
| This project has adopted the [MeteMath](https://github.com/meta-math/MetaMath) and [MuggleMath](https://github.com/OFA-Sys/gsm8k-ScRel). |
|
|
|
|
