Llama-3.1-8B-INBioCR-sp-DAPT-ft-INBioCR_plants_QA

This model is a supervised fine tuning of mariamoracrossitcr/Llama-3.1-8B-INBioCR-sp-DAPT for Spanish question answering on biodiversity data related to Costa Rican plants.

It is intended as a domain adapted assistant for answering plant related biodiversity questions in Spanish, using concise and factual responses grounded in curated structured data transformed into question-answer pairs.

Model lineage

This model follows the lineage:

meta-llama/Llama-3.1-8B-Instruct
→ mariamoracrossitcr/Llama-3.1-8B-INBioCR-sp-DAPT
→ Llama-3.1-8B-INBioCR-sp-DAPT-ft-INBioCR_plants_QA

The immediate parent for this fine tuning stage is:

• mariamoracrossitcr/Llama-3.1-8B-INBioCR-sp-DAPT

Intended use

This model is designed for:

• Spanish biodiversity question answering
• Costa Rican flora related queries
• educational and research oriented biodiversity assistance
• domain specific response generation from plant knowledge

Training data

The model was fine tuned using the dataset:

• mariamoracrossitcr/INBioCR_plants_QA

The training script formats each example as a chat conversation with:

• a system instruction in Spanish
• a user question
• an assistant answer

The system message used in training was:

Eres un asistente experto en biodiversidad y conservación en Costa Rica. Responde en español claro, preciso y basado en datos. Si la pregunta no se puede responder con la información disponible, dilo explícitamente.

Training procedure

This model was trained with PEFT LoRA using trl.SFTTrainer. Hardware: Kabre.cenat.ac.cr NVIDIA L40 48GB

Base model loading

• Base model: mariamoracrossitcr/Llama-3.1-8B-INBioCR-sp-DAPT
• Tokenizer: meta-llama/Llama-3.1-8B-Instruct
• Precision: bfloat16

LoRA configuration

• Rank r = 32
• lora_alpha = 32
• lora_dropout = 0.05
• Bias: none
• Task type: CAUSAL_LM

Target modules:

• o_proj
• q_proj
• v_proj
• k_proj
• gate_proj
• up_proj
• down_proj

Training arguments

• Per device train batch size: 2
• Gradient accumulation steps: 8
• Effective batch accumulation used for memory efficient fine tuning
• Number of epochs: 3
• Learning rate: 1e-4
• Scheduler: cosine
• Warmup ratio: 0.1
• Weight decay: 0.05
• Evaluation strategy: every 200 steps
• Save strategy: every 200 steps
• Logging steps: 20
• bf16 = True
• Gradient checkpointing enabled
• load_best_model_at_end = True
• Best model selected by lowest validation loss

Early stopping

• Patience: 5
• Threshold: 0.01

Training results

The training log shows a steady reduction in validation loss during fine tuning.

Observed validation losses from the provided log include:

• Epoch 0.15: 1.4776
• Epoch 0.30: 1.4007
• Epoch 0.45: 1.1877
• Epoch 0.60: 1.0944
• Epoch 0.75: 0.9866

---

license: apache-2.0 language:

• es base_model:
• meta-llama/Llama-3.1-8B-Instruct pipeline_tag: question-answering ---- Epoch 0.90: 0.9541
• Epoch 1.05: 0.7356
• Epoch 1.20: 0.6595
• Epoch 1.35: 0.5773
• Epoch 1.50: 0.4970
• Epoch 1.65: 0.4441
• Epoch 1.80: 0.3958

Best validation loss observed in the provided output:

• eval_loss = 0.3958 at epoch 1.80

Prompt format

This model was trained with chat formatted examples. A typical prompt structure is:

<system>
Eres un asistente experto en biodiversidad y conservación en Costa Rica. Responde en español claro, preciso y basado en datos. Si la pregunta no se puede responder con la información disponible, dilo explícitamente.

<user>
¿En qué tipo de bosques se puede encontrar con frecuencia Perrottetia longistylis?

<assistant>
...


Limitations
The model is specialized for the training domain and language.
It may hallucinate when the dataset does not provide enough evidence.
It should not be treated as a substitute for expert taxonomic or conservation judgment.
Performance may degrade for out of domain species or questions not represented in training.