Fedifehmi/Llama3-Ultra-Instruct-ChatQA-1.5-8B

models

This is a merge of pre-trained language models created using mergekit.

Merge Details

Merge Method

This model was merged using the DARE TIES merge method using C:\Users\fedi\Desktop\mentalllm\static\mergekittest\nvidia-llama3 as a base.

Models Merged

The following models were included in the merge:

• C:\Users\fedi\Desktop\mentalllm\static\mergekittest\Llama-3-8B-Ultra-Instruct C:\Users\fedi\Desktop\mentalllm\static\mergekittest\nvidia-llama3-ChatQA-1.5-8B

  on huggingface:

• nvidia/Llama3-ChatQA-1.5-8B

• Dampfinchen/Llama-3-8B-Ultra-Instruct

Llama-3-8B-Ultra-Instruct//

This is a merge of pre-trained language models created using mergekit.

Hello everyone this is Dampf, creator of the Destroyer series!

This time, I'm introducing you to 8B-Ultra-Instruct. It is a small general purpose model that combines the most powerful instruct models with enticing roleplaying models. It will introduce better RAG capabilities in the form of Bagel to Llama 3 8B Instruct as well as German multilanguage, higher general intelligence and vision support. A model focused on Biology adds knowledge in the medical field.

As for roleplay, it features two of the hottest models right now. Those are known to be high quality and for being uncensored. So this model might put out harmful responses. We are not responsible for what you do with this model and please take everything the model says with a huge grain of salt. Lastly, you might notice I'm conversative with the weight values in the final merge. This is because I believe L8B Instruct is a very dense model that's already great and doesn't need a lot more data. So instead of reaching the weight value of 1 in a ties merge, I'm only using a total of 0,65. This is to preserve Llama Instruct's intelligence and knowledge, while adding a little bit of the aforementioned models as salt in the soup.

A huge thank you for all the creators of the datasets. Those include Undi95, Jon Durbin, Aaditya, VAGOsolutions, Teknium, Camel and many more. They deserve all the credit. And of course, thank you Elinas for providing the compute.

Quants GGUF

exllama2

Merge Details

Merge Method

This model was merged using the DARE TIES merge method using Undi95/Meta-Llama-3-8B-Instruct-hf as a base.

Models Merged

• The following models were included in the merge:

Undi95/Meta-Llama-3-8B-Instruct-hf Undi95/Llama-3-LewdPlay-8B-evo jondurbin/bagel-8b-v1.0 Weyaxi/Einstein-v6.1-Llama3-8B VAGOsolutions/Llama-3-SauerkrautLM-8b-Instruct aaditya/OpenBioLLM-Llama3-8B

Open LLM Leaderboard Evaluation Results

Detailed results can be found here

Metric Value Avg. 69.11 AI2 Reasoning Challenge (25-Shot) 64.59 HellaSwag (10-Shot) 81.63 MMLU (5-Shot) 68.32 TruthfulQA (0-shot) 52.80 Winogrande (5-shot) 76.95 GSM8k (5-shot) 70.36

nvidia/Llama3-ChatQA-1.5-8B //

Model Details

• We introduce Llama3-ChatQA-1.5, which excels at conversational question answering (QA) and retrieval-augmented generation (RAG). Llama3-ChatQA-1.5 is developed using an improved training recipe from ChatQA paper, and it is built on top of Llama-3 base model. Specifically, we incorporate more conversational QA data to enhance its tabular and arithmetic calculation capability. Llama3-ChatQA-1.5 has two variants: Llama3-ChatQA-1.5-8B and Llama3-ChatQA-1.5-70B. Both models were originally trained using Megatron-LM, we converted the checkpoints to Hugging Face format. For more information about ChatQA, check the website!

Benchmark Results

Results in ChatRAG Bench are as follows:

ChatQA-1.0-7B Command-R-Plus Llama3-instruct-70b GPT-4-0613 GPT-4-Turbo ChatQA-1.0-70B ChatQA-1.5-8B ChatQA-1.5-70B Doc2Dial 37.88 33.51 37.88 34.16 35.35 38.90 39.33 41.26 QuAC 29.69 34.16 36.96 40.29 40.10 41.82 39.73 38.82 QReCC 46.97 49.77 51.34 52.01 51.46 48.05 49.03 51.40 CoQA 76.61 69.71 76.98 77.42 77.73 78.57 76.46 78.44 DoQA 41.57 40.67 41.24 43.39 41.60 51.94 49.60 50.67 ConvFinQA 51.61 71.21 76.6 81.28 84.16 73.69 78.46 81.88 SQA 61.87 74.07 69.61 79.21 79.98 69.14 73.28 83.82 TopioCQA 45.45 53.77 49.72 45.09 48.32 50.98 49.96 55.63 HybriDial* 54.51 46.7 48.59 49.81 47.86 56.44 65.76 68.27 INSCIT 30.96 35.76 36.23 36.34 33.75 31.90 30.10 32.31 Average (all) 47.71 50.93 52.52 53.90 54.03 54.14 55.17 58.25 Average (exclude HybriDial) 46.96 51.40 52.95 54.35 54.72 53.89 53.99 57.14 Note that ChatQA-1.5 is built based on Llama-3 base model, and ChatQA-1.0 is built based on Llama-2 base model. ChatQA-1.5 models use HybriDial training dataset. To ensure fair comparison, we also compare average scores excluding HybriDial. The data and evaluation scripts for ChatRAG Bench can be found here.

The content of the system's turn (i.e., {System}) for both scenarios is as follows:

This is a chat between a user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions based on the context. The assistant should also indicate when the answer cannot be found in the context.

Note that our ChatQA-1.5 models are optimized for the capability with context, e.g., over documents or retrieved context.

Configuration

The following YAML configuration was used to produce this model:

models:
  - model: C:\Users\fedi\Desktop\mentalllm\static\mergekittest\nvidia-llama3
  - model: C:\Users\fedi\Desktop\mentalllm\static\mergekittest\Llama-3-8B-Ultra-Instruct
    parameters:
      density: 0.55
      weight: 0.2
merge_method: dare_ties
base_model: C:\Users\fedi\Desktop\mentalllm\static\mergekittest\nvidia-llama3
parameters:
  int8_mask: true
dtype: float16