README.md

---
license: apache-2.0
language:
- en
- zh
base_model:
- Qwen/Qwen3-8B
pipeline_tag: text-generation
library_name: transformers
tags:
- qwen3
- qwen3-8b
- lora
- qlora
- sft
- rag
- faiss
- dense-retrieval
- agent
- ppo
- rlhf
- rule-reward
- harness-engineering
- um-handbook
- question-answering
- chatbot
- education
- tensor-talk
---

# TensorTalk: UM Handbook Qwen3-8B SFT + RAG + Agent + PPO + Harness Engineering

TensorTalk is a staged LLM engineering project built for **Universiti Malaya Faculty of Computer Science and Information Technology handbook question answering**. The system is designed to answer undergraduate, postgraduate, and general faculty handbook questions using a controlled progression of three experimental stages:

1. **Baseline 1 — Closed-book SFT Qwen3-8B**
2. **Baseline 2 — SFT Qwen3-8B + Metadata-aware RAG + Official Web Agent + Harness Engineering**
3. **Improved Model — Rule-reward PPO post-training + RAG + Agent + Harness Engineering**

The project is not just a simple chatbot. It is a controlled comparison of how an LLM system improves when moving from memorized supervised fine-tuning, to retrieval-grounded answering, and finally to rule-reward post-training with a guarded agentic runtime.

The main idea is:

> Baseline 1 tests whether a fine-tuned model can answer handbook questions from parameters alone.  
> Baseline 2 keeps the same base model family but adds retrieval and harnessed evidence control.  
> The Improved Model keeps the RAG + Agent + Harness runtime and further adds PPO post-training to make the model better aligned with the desired answer behavior.

---

# 1. Project Goal

The goal of this project is to build a reliable and traceable UM Handbook assistant that can answer questions about:

- Faculty objectives, vision, mission, history, facilities, and academic calendar
- Undergraduate programme details
- Postgraduate programme details
- Candidature requirements
- Grading and academic rules
- Industrial training
- Academic project requirements
- Supervision policy
- Thesis/dissertation requirements
- Academic integrity and plagiarism
- Facilities and labs
- Official UM/FSKTM web information when handbook knowledge is insufficient or time-sensitive

The project also aims to demonstrate a complete LLM system development path:

```text
Closed-book SFT
→ RAG-augmented SFT
→ Metadata-aware retrieval
→ Official-source web agent
→ Harness Engineering guardrails
→ PPO rule-reward post-training
→ Strict artifact verification
→ Traceable TensorTalk UI
```

---

# 2. High-level System Overview

The final TensorTalk system contains several layers.

```text
User Question
    ↓
TensorTalk UI
    ↓
Planning / Thinking Display Layer
    ↓
Local Handbook RAG
    ↓
Official UM / FSKTM Web Agent
    ↓
Harness Engineering Guardrails
    ↓
Evidence Judge / Retry / Fallback
    ↓
PPO-trained Qwen3-8B Actor
    ↓
Answer Grounding Judge
    ↓
Completeness Guard
    ↓
Final Answer + Trace Panels
```

The final model is not used alone. It is wrapped inside a runtime harness that controls:

- where the system can search
- which sources it can trust
- whether web evidence is useful
- whether retrieved evidence supports the answer
- whether the model produced fake URLs
- whether the answer leaked internal reasoning
- whether fallback to local handbook RAG is needed
- whether the final answer is grounded enough to show

This is why the final stage is better described as:

> **A PPO-aligned RAG agent system with Harness Engineering**, rather than only a fine-tuned model.

---

# 3. Dataset Design

## 3.1 Source Domain

The dataset is built around UM FSKTM undergraduate and postgraduate handbook content. The data is organized into:

- SFT question-answer dataset
- hidden metadata
- RAG knowledge base
- RAG evaluation dataset
- PPO preference dataset

The project separates **model-visible training text** from **metadata used for retrieval, evaluation, and analysis**.

This distinction is important:

- Baseline 1 intentionally trains on question-answer text without forcing explicit metadata labels into the model-visible answer.
- Baseline 2 uses metadata-aware retrieval to reduce scope confusion.
- Stage 3 PPO uses preference pairs and reward functions to shape answer behavior.

---

## 3.2 Baseline 1 SFT Dataset

Baseline 1 uses:

```text
SFT_QA_Training_Ready.jsonl
```

The notebook validates:

```text
Total examples: 1000
Train examples: 800
Validation examples: 100
Test examples: 100
Split ratio: 8:1:1
Duplicate question groups: 0
Duplicate question rows: 0
```

Each example follows a supervised chat-style format:

```json
{
  "prompt": [
    {
      "role": "system",
      "content": "You are an academic assistant for the Faculty of Computer Science and Information Technology, Universiti Malaya..."
    },
    {
      "role": "user",
      "content": "What are the faculty objectives?"
    }
  ],
  "completion": [
    {
      "role": "assistant",
      "content": "The faculty objectives are..."
    }
  ],
  "question": "...",
  "answer": "..."
}
```

This stage teaches the model to imitate handbook-style answers directly.

---

## 3.3 Baseline 2 RAG Dataset

Baseline 2 uses the same SFT dataset direction, but adds external retrieval resources:

```text
UM_RAG_Knowledge_Base.jsonl
UM_RAG_Evaluation_Dataset.jsonl
SFT_QA_Metadata.jsonl
```

The RAG knowledge base contains structured fields such as:

```text
kb_id
source_doc
scope_label
section
pages
source_text
retrieval_text
retrieval_keywords
grounded_answer_bank
matched_qa_ids
```

The RAG knowledge base loaded in the final Stage 3 runtime contains:

```text
Loaded KB rows: 521
```

The metadata layer allows the system to distinguish:

```text
general
undergraduate
postgraduate
```

This is important because many handbook questions look similar but require different answers depending on the student scope.

---

## 3.4 PPO Preference Dataset

The Improved Model uses:

```text
UM_Handbook_PPO_Preference_Dataset.jsonl
```

The final PPO run uses the full dataset:

```text
Total PPO preference rows: 1000
Train rows: 900
Validation rows: 100
Train fraction: 0.90
```

The PPO dataset is not used like normal SFT data. In SFT, the model directly imitates a reference answer. In PPO, the model generates its own answer, receives a reward, and updates toward higher-reward behavior.

---

# 4. Baseline 1 — Closed-book SFT Qwen3-8B

## 4.1 Purpose

Baseline 1 asks a simple question:

> Can Qwen3-8B learn UM Handbook question answering from supervised fine-tuning alone?

This is a **closed-book baseline**. The model does not retrieve handbook evidence during inference. It must answer from what it learned during SFT.

This is useful as a control baseline because it shows what happens when the model relies mainly on parameter memory.

---

## 4.2 Model

Baseline 1 uses:

```text
Base model: Qwen/Qwen3-8B
Local path: /scr/user/kevin2002/TensorCat/NLP/UM_Handbook/models/Qwen3-8B
```

The notebook detected:

```text
Backend: CUDA
GPU: NVIDIA A100-SXM4-80GB
dtype: bfloat16
4-bit QLoRA: enabled
```

---

## 4.3 Training Method

Baseline 1 uses LoRA / QLoRA supervised fine-tuning.

LoRA configuration:

```text
LoRA rank: 16
LoRA alpha: 32
LoRA dropout: 0.10
Target modules:
  - q_proj
  - k_proj
  - v_proj
  - o_proj
  - gate_proj
  - up_proj
  - down_proj
```

Training configuration:

```text
Epochs: 8
Train split: 800
Validation split: 100
Test split: 100
Per-device train batch size: 2
Per-device eval batch size: 2
Gradient accumulation steps: 8
Learning rate: 1e-4
Packing: False
```

---

## 4.4 Baseline 1 Results

The training completed successfully.

Training summary:

```text
Training steps: 400
Training runtime: ~18.68 minutes for the main train stage
Train loss: 0.4824
Final validation loss: ~0.146
Test loss: ~0.197
Perplexity: ~1.157
```

Generation metrics:

### Validation

```text
Exact match: 0.77
Token F1: 0.9111
ROUGE-1: 0.9122
ROUGE-2: 0.8700
ROUGE-L: 0.8979
SacreBLEU: 81.7240
chrF++: 86.8916
Average prediction words: 36.35
Average reference words: 38.57
```

### Test

```text
Exact match: 0.72
Token F1: 0.8869
ROUGE-1: 0.8857
ROUGE-2: 0.8352
ROUGE-L: 0.8677
SacreBLEU: 81.1138
chrF++: 87.7054
Average prediction words: 38.03
Average reference words: 37.03
```

---

## 4.5 Baseline 1 Strengths

Baseline 1 is strong when the question is close to the training distribution. It can reproduce handbook-style answers well and shows high text overlap with the reference answers.

It is useful because:

- it establishes the basic Qwen3-8B SFT capability
- it verifies that the dataset format is learnable
- it creates a clean closed-book control model
- it provides a baseline for later RAG and PPO improvements

---

## 4.6 Baseline 1 Limitations

Baseline 1 is still limited because it is a closed-book model.

Main limitations:

1. **No retrieval evidence**  
   It cannot check the handbook at inference time.

2. **Potential hallucination**  
   If the question is out-of-distribution or requires exact source grounding, the model may answer from memory.

3. **Scope confusion**  
   Undergraduate and postgraduate rules may be mixed if the question is ambiguous.

4. **No official web update mechanism**  
   It cannot answer dynamic or latest-information questions reliably.

5. **No harness guardrails**  
   It does not include fake URL detection, evidence judging, WAF handling, or fallback control.

Baseline 1 is therefore a necessary but incomplete starting point.

---

# 5. Baseline 2 — RAG + SFT + Metadata-aware Retrieval + Harness Agent

## 5.1 Purpose

Baseline 2 asks:

> What improves if we keep the same Qwen3-8B family but add retrieval-grounded evidence?

The goal is to reduce hallucination and scope confusion by giving the model relevant handbook evidence at inference time.

This stage introduces RAG and agentic harness logic while keeping the same broad model family and handbook task.

---

## 5.2 What RAG Means in This Project

RAG stands for **Retrieval-Augmented Generation**.

In simple terms:

```text
Instead of asking the model to answer only from memory,
the system first retrieves relevant handbook chunks,
then asks the model to answer using those chunks.
```

In this project, RAG is not just keyword search. It uses:

```text
Transformer embedding model
+ FAISS vector search
+ metadata-aware reranking
+ scope labels
+ top-k evidence blocks
```

The Baseline 2 retriever uses:

```text
Embedding model: BAAI/bge-base-en-v1.5
Vector index: FAISS
Similarity: inner product after embedding normalization
Top-k retrieval: 3
```

---

## 5.3 Metadata-aware Retrieval

The RAG system uses metadata to control retrieval quality.

Important metadata fields include:

```text
source_doc
scope_label
section
pages
kb_id
knowledge group
retrieval keywords
grounded answer bank
```

This allows the retriever to prefer the correct audience scope.

Example:

```text
Question: What are the candidature requirements for Master of Software Engineering?
Expected scope: postgraduate
```

The system should retrieve postgraduate chunks, not undergraduate chunks.

This is one of the main improvements over Baseline 1.

---

## 5.4 RAG-augmented Training Dataset

Baseline 2 creates a RAG-augmented dataset where training examples include evidence context.

The training prompt can contain:

```text
User question
+ retrieved handbook evidence
+ source metadata
+ answer instruction
```

This teaches the model to answer with evidence-aware context rather than only memorized answers.

---

## 5.5 Baseline 2 Training Configuration

Baseline 2 uses Qwen3-8B with LoRA fine-tuning.

Configuration:

```text
Base model: Qwen/Qwen3-8B
Embedding model: BAAI/bge-base-en-v1.5
LoRA rank: 8
LoRA alpha: 16
LoRA dropout: 0.05
Target modules:
  - q_proj
  - k_proj
  - v_proj
  - o_proj
  - gate_proj
  - up_proj
  - down_proj
Epochs: 20
Per-device train batch size: 4
Per-device eval batch size: 8
Target global batch size: 8
Learning rate: 8e-5
Max sequence length: 1024
Validation ratio: 0.10
Test ratio: 0.10
Save merged model: False
Runtime model path: base model + LoRA adapter
```

The notebook uses a safer non-merged runtime path when merged model export is unavailable or memory-expensive.

---

## 5.6 Baseline 2 Retrieval Evaluation

Baseline 2 includes a retrieval evaluation set.

Retrieval metrics:

```json
{
  "retrieval_eval_size": 1000,
  "top_k": 3,
  "hit_at_1_primary": 0.821,
  "hit_at_k_primary": 0.954,
  "hit_at_k_same_group": 0.991,
  "scope_match_at_1": 0.996,
  "retriever_type": "dense_embedding + faiss + metadata_rerank",
  "embedding_model_name": "BAAI/bge-base-en-v1.5"
}
```

Interpretation:

- `hit_at_1_primary = 0.821` means the top retrieved chunk is exactly the expected primary evidence in 82.1% of cases.
- `hit_at_k_primary = 0.954` means the correct primary evidence appears within top-3 in 95.4% of cases.
- `hit_at_k_same_group = 0.991` means a same-group acceptable evidence appears in top-3 in 99.1% of cases.
- `scope_match_at_1 = 0.996` means the top result almost always matches the correct undergraduate/postgraduate/general scope.

This confirms that the RAG system is not random retrieval. It is a strong metadata-aware retrieval baseline.

---

## 5.7 Baseline 2 Generation Evaluation

Generation evaluation was run on a smaller selected set for runtime practicality.

Results:

```json
{
  "generation_eval_size": 20,
  "top_k": 3,
  "plain_exact_match": 0.0,
  "plain_token_f1": 0.3391,
  "rag_exact_match": 0.0,
  "rag_token_f1": 0.8460,
  "rag_minus_plain_exact_match": 0.0,
  "rag_minus_plain_token_f1": 0.5069
}
```

This shows a large improvement from RAG:

```text
Plain token F1: 0.3391
RAG token F1:   0.8460
Improvement:    +0.5069
```

This is one of the strongest pieces of evidence in the project.

It shows that retrieval grounding dramatically improves answer quality compared with plain generation.

---

# 6. Agent Layer in Baseline 2 and Improved Model

## 6.1 Why an Agent Is Needed

The handbook is reliable for stable academic rules, but a practical university assistant cannot depend only on static handbook text. Some user questions naturally require **official web discovery**, **source checking**, or **routing decisions**.

Examples:

```text
Who is the current dean?
Where can students find residential college information?
What official page mentions PEKOM?
Where is the official SPeCTRUM page?
What facilities are associated with a specific lab?
```

For these cases, TensorTalk uses an official-source web agent. The agent is not designed as an unrestricted autonomous browser. It is deliberately designed as a **constrained agent** because the domain is academic handbook QA, where factual trustworthiness is more important than open-ended exploration.

In practical terms, the agent layer answers this question:

> If local handbook RAG is not enough, can the system search official UM/FSKTM sources, reject weak or fake sources, and return evidence safely?

---

## 6.2 How This Project Relates to LangChain and LangGraph

TensorTalk does **not** use LangChain, LangGraph, or LangSmith as the runtime framework. The agent and harness were implemented from scratch in the notebook.

However, the design is conceptually aligned with the official LangChain ecosystem ideas:

- LangChain describes agents as systems that combine language models with tools so they can reason about tasks, decide which tools to use, and iteratively work toward a result.
- LangGraph describes agent workflows using state, nodes, and edges, where nodes perform computation and edges determine the next transition.
- LangSmith describes evaluation as a workflow involving datasets, evaluators, and experiments to compare application versions.
- LangChain/LangGraph documentation also distinguishes between predetermined workflows and dynamic agents; TensorTalk intentionally uses a hybrid design because the handbook QA task needs both predictable guardrails and dynamic retrieval decisions.

Therefore, this project is best described as:

> A from-scratch implementation of a LangChain/LangGraph-inspired agentic RAG harness, not a project built by directly calling LangChain’s prebuilt agent framework.

This distinction is important. TensorTalk does not simply wrap a LangChain agent. Instead, it manually implements the major control ideas:

```text
State tracking
→ planning
→ retrieval/tool routing
→ source filtering
→ evidence normalization
→ evaluation/judging
→ retry/fallback
→ final generation
→ trace output
```

This gives the project more transparency because each part of the agent loop is visible in the notebook and UI trace.

---

## 6.3 Agent Design Philosophy

The TensorTalk agent is built around four principles.

### 1. Source-constrained autonomy

The agent can search and fetch information, but only from allowed official sources. It is not free to trust arbitrary search results.

### 2. Evidence-first generation

The model should not directly answer a web-sensitive question before evidence is collected and judged.

### 3. Retry and fallback

If official web evidence is weak, blocked, irrelevant, or unsafe, the system can retry with entity-aware search terms or fall back to local handbook RAG.

### 4. Traceable decisions

The agent does not hide its routing decisions. It records URL candidates, accepted evidence, rejected evidence, grounding decisions, and fallback decisions in trace panels.

---

## 6.4 Official UM / FSKTM Web Agent

The web agent is constrained to official UM / FSKTM domains. This is one of the most important safety and reliability choices in the project.

### Priority official domains

```text
fsktm.um.edu.my
www.um.edu.my
um.edu.my
```

### Auxiliary official UM-related domains

The project also recognizes selected UM-related service domains when they are relevant to student services, academic systems, library resources, research, career portals, or internal faculty resources:

```text
aasd.um.edu.my
maya.um.edu.my
umlib.um.edu.my
umresearch.um.edu.my
jobs.um.edu.my
careerportal.fsktm.um.edu.my
intra.fsktm.um.edu.my
gallery.fsktm.um.edu.my
```

The purpose of this whitelist is not to search the whole internet. The purpose is to constrain the agent to sources that are likely to be officially controlled by UM or FSKTM.

---

## 6.5 Domain Whitelist Design

The whitelist is used as a **domain guard** before a page can become trusted evidence.

The system treats URLs in three broad categories:

| URL type | Handling |
|---|---|
| Official UM/FSKTM URL | Can be considered as candidate evidence |
| UM-related service URL | Can be considered if relevant to the question type |
| Non-official or synthetic URL | Rejected or downgraded |

The whitelist helps prevent common LLM-agent failure cases:

```text
hallucinated programme pages
invented lab pages
fake student service URLs
misrouted search results
random third-party pages
SEO or unrelated pages
```

For example, the project specifically tests that the agent should not invent or accept URLs like:

```text
programme-ccna-lab-more-detailedly
bachelor-of-computer-science-artificial-intelligence
```

when those pages are not the correct evidence for the user’s question.

---

## 6.6 Web Agent Workflow

The official-source web agent follows a controlled workflow.

```text
User question
    ↓
Intent and entity detection
    ↓
Official-search query construction
    ↓
Candidate URL discovery
    ↓
Domain whitelist filtering
    ↓
Synthetic/fake URL rejection
    ↓
Fetch or static page fallback
    ↓
WAF/block detection
    ↓
Text extraction and normalization
    ↓
Evidence scoring
    ↓
Qwen evidence judge
    ↓
Accept evidence, retry, or fallback to handbook RAG
```

This means the agent is not only a web search function. It is a guarded evidence acquisition pipeline.

---

## 6.7 Planning Inside the Agent

Planning is a visible part of the TensorTalk system.

The planning layer is responsible for deciding:

```text
Is this a stable handbook question?
Is this a latest/current official-web question?
Should local RAG be used first?
Should official web discovery be attempted?
Which entity should be searched?
Which scope should be preferred: undergraduate, postgraduate, general, faculty, or university?
What evidence type is expected: handbook chunk, official page, contact page, facility page, announcement, policy page?
```

This planning step is aligned with the idea that agentic systems should not directly jump from user question to final answer. They need a control stage that decides which tools and evidence paths are appropriate.

TensorTalk’s planning is not a free-form hidden chain-of-thought that users must trust blindly. It is operationalized through explicit routing variables, trace objects, search decisions, and UI panels.

---

## 6.8 Generation Inside the Agent

Generation is the stage where the Qwen3-8B model produces an answer.

However, generation is not allowed to operate alone. The answer generator receives controlled context:

```text
user question
retrieved local handbook evidence
accepted official web evidence
scope hints
source metadata
harness instructions
answer style constraints
```

The generator is expected to:

```text
answer directly
avoid unsupported claims
avoid fake URLs
avoid exposing internal reasoning
use local handbook evidence when web evidence is weak
prefer official web evidence only when it is relevant and trusted
```

In the final stage, the generator is the PPO-trained Qwen3 actor, but it is still wrapped by the same RAG and Harness Engineering control layer.

---

## 6.9 Evaluation Inside the Agent

Evaluation is the other core part of the agent loop. TensorTalk evaluates both evidence and answers.

### Evidence evaluation

The system checks:

```text
Is the source official?
Is the URL synthetic or fake?
Is the page blocked by WAF?
Is the evidence relevant to the user question?
Does the evidence mention the right entity?
Does the evidence match the expected scope?
```

### Answer evaluation

The system checks:

```text
Is the final answer grounded in accepted evidence?
Does it answer the user’s actual question?
Does it leak internal thinking?
Does it invent URLs?
Is it too vague?
Is it incomplete enough to require fallback or rewrite?
```

This creates a full agentic loop:

```text
Planning
→ Retrieval / tool use
→ Generation
→ Evaluation
→ Retry or fallback
→ Final answer
```

---

## 6.10 Why the Agent Is Not Fully Autonomous

The agent is intentionally not fully autonomous.

A fully autonomous browsing agent may:

```text
search too broadly
trust wrong sources
follow irrelevant pages
invent missing pages
overuse web search
ignore handbook evidence
produce unsupported answers
```

TensorTalk instead uses a constrained model:

```text
Dynamic when needed
Guarded by default
Official-source-only for web evidence
RAG-first for handbook-stable questions
Fallback-aware when web evidence is weak
Traceable for debugging and demonstration
```

This is more appropriate for a university handbook assistant.

---

# 7. Harness Engineering

## 7.1 What Harness Engineering Means Here

Harness Engineering is the external control system around the LLM, RAG, and agent.

A simple analogy:

```text
The LLM/agent is the car.
Harness Engineering is the guardrail, traffic rule, checkpoint, fallback route, dashboard, and driving examiner.
```

The model can generate fluent answers, but the harness controls:

```text
what it can search
which domains are trusted
which URLs are rejected
which evidence is useful
when to retry
when to fallback
whether the answer is grounded
whether the UI should show warning traces
whether the final response is safe enough to display
```

In TensorTalk, Harness Engineering is not just prompt engineering. Prompt engineering tells the model what to do. Harness Engineering builds the surrounding execution system that checks whether the model actually did it correctly.

---

## 7.2 From Prompt Engineering to Harness Engineering

Prompt engineering is like telling a driver:

```text
Please drive carefully.
```

Harness Engineering is like building:

```text
lane barriers
speed checks
traffic rules
navigation checkpoints
fallback routes
dashboards
incident logs
```

In this project, prompt engineering alone is not enough because the model may still:

```text
invent fake URLs
mix undergraduate and postgraduate rules
leak internal reasoning
trust weak web snippets
answer without evidence
overuse web search
ignore local handbook RAG
```

The harness prevents or detects these failures through code-level controls, not only prompt instructions.

---

## 7.3 From-scratch LangChain-style Harness

TensorTalk’s harness was built manually rather than by importing a prebuilt LangChain/LangGraph agent.

The implementation follows the same conceptual loop used in many modern agent frameworks:

```text
Planning
→ Tool / Retrieval Action
→ Generation
→ Evaluation
→ Retry / Fallback
→ Finalization
```

But each component is implemented explicitly:

| Conceptual framework idea | TensorTalk from-scratch implementation |
|---|---|
| Agent state | Trace dictionaries, evidence bundles, routing flags, runtime status |
| Tools | Local RAG retriever, official web search/fetch, URL validator, evidence judge |
| Nodes | Planning, retrieval, web discovery, evidence filtering, judging, generation, grounding |
| Edges / transitions | Conditional retry, weak-evidence fallback, RAG fallback, final answer route |
| Evaluation | Qwen evidence judge, rule checks, answer grounding judge, smoke tests |
| Observability | Collapsed UI trace panels and printed diagnostic outputs |

This makes the system easier to inspect in an academic notebook because the control logic is visible.

---

## 7.4 Planning → Generation → Evaluation Closed Loop

The most important Harness Engineering contribution in TensorTalk is the closed loop:

```text
Planning
    ↓
Generation
    ↓
Evaluation
    ↓
Retry / Fallback / Finalization
```

### Planning

The planning layer decides how to handle the query.

It considers:

```text
question type
scope
entity
whether local RAG is enough
whether official web is needed
whether the query is dynamic/current
whether the answer should be handbook-grounded or web-grounded
```

### Generation

The generation layer produces an answer using controlled evidence.

It receives:

```text
local handbook chunks
official web evidence
scope hints
source metadata
answer constraints
```

### Evaluation

The evaluation layer checks the result.

It evaluates:

```text
source trust
URL validity
evidence relevance
answer grounding
completeness
process leakage
fake URLs
fallback need
```

If evaluation fails, the system can retry, reroute, or fall back.

This is the engineering loop that makes TensorTalk more than a simple RAG chatbot.

---

## 7.5 Standardized Harness Core Pipeline

The final standardized TensorTalk Harness Core follows this pipeline:

```text
User Question
    ↓
Planning Layer
    ↓
Local Handbook RAG
    ↓
Official Web Discovery
    ↓
Domain Guard
    ↓
Fake URL Guard
    ↓
WAF Detection
    ↓
Evidence Normalizer
    ↓
Qwen Evidence Judge
    ↓
Entity-aware Retry
    ↓
Weak Evidence Fallback
    ↓
PPO/SFT Answer Generator
    ↓
Answer Grounding Judge
    ↓
Completeness Guard
    ↓
Final Answer
    ↓
UI Trace
```

This pipeline is intentionally explicit. Each part has a clear job.

---

## 7.6 Harness State and Trace Objects

The harness keeps structured trace data so that every answer can be inspected.

Typical trace information includes:

```text
retrieved local RAG chunks
candidate web URLs
accepted official URLs
rejected URLs
web evidence bundle
harness core route
evidence judge result
answer grounding result
fallback reason
final answer preview
```

This is similar in spirit to observability and tracing in agent platforms, but implemented directly in the notebook and UI.

---

## 7.7 Domain Guard

The domain guard checks whether a candidate source belongs to the allowed official domain set.

It protects against:

```text
random third-party websites
unofficial mirrors
search result noise
LLM-fabricated domains
wrong university pages
```

It also makes the system explainable. If the agent rejects a page, the trace can show why.

---

## 7.8 Fake URL Guard

The fake URL guard is one of the most important parts of the project because raw LLM generations can invent plausible-looking URLs.

Examples of risky synthetic URLs include:

```text
https://spectrum.umlms
http://spectrux.medicum
programme-ccna-lab-more-detailedly
https://aasd um edu my/studetn
```

The guard checks and rejects URLs that:

```text
are malformed
look fabricated
contain suspicious path patterns
do not belong to allowed domains
are query-fabricated rather than discovered from official search/fetch
```

The PPO reward function also penalizes hallucinated URLs, but the harness is still necessary because reward shaping does not guarantee perfect URL behavior.

---

## 7.9 WAF Detection

Some official pages can be blocked, partially loaded, or protected by web application firewalls.

The WAF-aware harness detects cases where:

```text
the page cannot be fetched normally
the content is a block page instead of the real page
the browser click fails
the official site returns insufficient text
```

When this happens, the system avoids treating the blocked page as strong evidence. It can use diagnostics, retry, static fallback, or local RAG fallback.

---

## 7.10 Evidence Normalizer

Fetched web pages and handbook chunks may be noisy.

The evidence normalizer attempts to convert them into a consistent evidence structure:

```text
title
url
source type
domain
text snippet
score
scope
entity
reason
```

This makes later judging and UI display easier.

---

## 7.11 Qwen Evidence Judge

The Qwen evidence judge is used to decide whether retrieved evidence actually helps answer the user’s question.

It checks:

```text
Does the evidence mention the right entity?
Does it answer the question directly?
Is it only loosely related?
Is it a wrong programme/page?
Is it official but irrelevant?
```

This is important because official sources can still be irrelevant. A page can be official and still be the wrong evidence.

---

## 7.12 Entity-aware Retry

If the first web discovery result is weak or misrouted, the harness can retry with better query terms.

For example, if a question about PEKOM gets routed toward an AI bachelor programme page, the harness should retry using terms related to:

```text
PEKOM
Persatuan Komputer UM
student society
FSKTM student association
```

This prevents the agent from accepting the first official-looking but semantically wrong page.

---

## 7.13 Weak Evidence Fallback

If the official web evidence is weak, TensorTalk can fall back to local handbook RAG.

This prevents a common agent failure:

```text
The system found a web page, so it trusts it even though it does not answer the question.
```

Instead, TensorTalk uses:

```text
web evidence if strong
local handbook RAG if web evidence is weak
hybrid answer if both are useful
refusal/uncertainty if neither is sufficient
```

---

## 7.14 Answer Grounding Judge

After answer generation, the answer grounding judge checks whether the final answer is supported by the accepted evidence.

It helps catch cases where:

```text
retrieval was correct but generation added unsupported claims
the model invented a URL
the model mixed evidence from different scopes
the answer contains a statement that does not appear in evidence
```

This is the evaluation part of the Planning → Generation → Evaluation loop.

---

## 7.15 Completeness Guard

The completeness guard checks whether the answer is too short, vague, or incomplete.

It can identify cases where the answer:

```text
only repeats the question
does not include required details
misses key fields
does not answer the requested scope
cuts off early
```

Depending on runtime settings, this can trigger a rewrite or fallback.

---

## 7.16 Smoke Tests as Harness Unit Checks

The smoke tests are lightweight checks that make sure the harness pipeline still works after model or code changes.

Examples:

```text
PEKOM should not be routed to the AI bachelor page.
Residential college should prefer the student-affairs residential page.
CCNA Lab should not invent synthetic URLs.
```

These tests check:

```text
routing
URL filtering
official page preference
fake URL rejection
answer grounding trace
harness core route
```

They are not a full benchmark. They are fast sanity checks that the system still runs through the expected pipeline.

---

## 7.17 Why Harness Engineering Is Central to This Project

The final system does not rely on only one technique.

```text
SFT gives domain answer style.
RAG gives handbook evidence.
The web agent gives official external evidence.
PPO improves answer behavior.
Harness Engineering controls the whole system.
```

Without the harness, the system would still be vulnerable to:

```text
wrong source selection
fake URLs
weak web evidence
scope confusion
process leakage
unsupported final answers
stale artifact loading
```

Therefore, Harness Engineering is the system-level contribution that connects SFT, RAG, Agent, and PPO into one controlled workflow.

---

# 8. Improved Model — PPO Rule-reward Post-training + RAG + Agent + Harness

## 8.1 Purpose

The Improved Model asks:

> Can we further improve the model’s behavior after SFT/RAG by using PPO reward-based post-training?

Baseline 2 already improves factual grounding through RAG and Harness Engineering. The Improved Model adds PPO to shape the model’s behavior.

The goal is not to replace RAG. The goal is to make the model more aligned with the desired answer style and safety behavior.

---

## 8.2 What PPO Means in This Project

PPO stands for **Proximal Policy Optimization**.

In simple terms:

```text
SFT teaches the model by imitation.
PPO lets the model generate answers, scores them with a reward function, and updates the model toward higher-reward answers.
```

In this project:

```text
Actor model: Qwen3-8B + LoRA
Critic/value head: TRL value head model
Reference model: frozen Qwen3-8B reference
Reward: rule-based preference reward function
KL control: used to avoid drifting too far from the reference model
```

---

## 8.3 Rule-based Reward Function

This project uses a rule-based reward function rather than a separately trained neural reward model.

The reward function evaluates:

```text
gold answer similarity
rejected answer penalty
evidence overlap
scope correctness
hallucinated URL penalty
vague answer penalty
process/thinking leakage penalty
direct answer bonus
repetition penalty
degeneration/collapse penalty
```

This is why the model card should describe the final stage as:

> Rule-reward PPO post-training

not:

> Full RLHF with a trained reward model

The reward model type recorded in the notebook is:

```text
rule_based_preference_reward_function
uses_separate_neural_reward_model: False
```

---

## 8.4 PPO Training Configuration

The final PPO run uses:

```text
Preference dataset rows: 1000
Train rows: 900
Validation rows: 100
MAX_PPO_ROWS: None
Train fraction: 0.90
PPO epochs: 2
Batch size: 2
Mini-batch size: 1
Max new tokens: 72
Max PPO steps per epoch: None
Planned steps per epoch: 450
Total planned steps: 900
Learning rate: 2e-6
Target KL: 0.10
Generation temperature: 0.45
Top-p: 0.78
Repetition penalty: 1.3
No-repeat ngram size: 4
```

The run completed successfully:

```text
Global PPO steps: 900 / 900
Elapsed time: 04:47:59
Degenerate ratio: 0.00%
```

---

## 8.5 PPO Artifact Verification

The Stage 3 notebook includes strict artifact verification.

This is important because PPO notebooks can easily appear to run while silently saving old or incomplete artifacts.

The strict save cell verifies:

```text
training_log exists
training_log records = 900
expected steps = 900
MAX_PPO_ROWS = None
train rows = 900
valid rows = 100
NUM_PPO_EPOCHS = 2
MAX_PPO_STEPS_PER_EPOCH = None
parameter hash changed after PPO
PPO inference full actor exists
PPO LoRA adapter exists
non-PPO fallback forbidden
```

The final strict save output confirms:

```text
Final PPO records saved: 900 / expected 900
Strict full PPO artifact contract passed.
```

The parameter change proof confirms:

```text
aggregate_hash_changed: true
changed_trainable_tensors: 506
unchanged_trainable_tensors: 0
```

This proves that PPO training changed the trainable LoRA/value-head parameters rather than merely running a dry notebook.

---

## 8.6 Strict PPO-only Runtime

The final runtime is configured so that the UI must use PPO artifacts only.

The strict PPO gate confirms:

```text
PPO records: 900
PPO full actor usable: True
PPO LoRA adapter usable: True
Strict PPO-only UI mode: True
```

The runtime loading order is:

```text
1. PPO full inference actor if full weights exist
2. Otherwise base Qwen3-8B + PPO LoRA adapter
3. Non-PPO fallback is forbidden
```

This prevents the final demo from accidentally loading an old Baseline 2 model or a stale 150-step PPO proof artifact.

---

## 8.7 PPO Validation

The PPO-only validation evaluation uses a held-out validation sample.

The displayed validation summary is:

```text
reward:            0.477789
gold_overlap:      0.255351
rejected_overlap:  0.155080
```

Interpretation:

- reward is positive
- gold overlap is higher than rejected overlap
- the PPO-trained actor tends to move closer to preferred answers than rejected answers

This does not mean the PPO model is perfect. It means the reward-shaped behavior is directionally positive.

---

## 8.8 PPO Limitations

The PPO run is successful, but the raw PPO generations still show some imperfections.

Observed issues include:

1. **Process leakage**  
   Some outputs still include phrases like:
   ```text
   Okay, let me try to figure out...
   Wait, I need to check again...
   ```
   The reward function penalizes this, but it is not completely eliminated.

2. **Occasional hallucinated URLs**  
   Some raw generations may still invent URLs. The harness fake URL guard is therefore still necessary.

3. **OCR-style text artifacts**  
   Some source chunks contain spacing or OCR issues, and the model may reproduce them.

4. **KL can be high**  
   Some PPO logs show high `objective/kl`, meaning the PPO actor can drift noticeably from the reference model. However, the run completed with:
   ```text
   degenerate_ratio = 0.00%
   ```
   and no detected repetition collapse.

5. **RAG/Harness remains necessary**  
   PPO improves model behavior, but it does not replace retrieval grounding or guardrails.

---

# 9. TensorTalk UI

The project includes a WhatsApp-style Jupyter HTML UI called **TensorTalk**.

The UI supports:

- chat-style interface
- TensorCat avatar
- RAG on/off control
- web agent on/off control
- collapsed trace panels
- retrieved evidence display
- web evidence display
- planning/thinking display layer
- harness decision trace
- answer grounding information
- strict PPO artifact loading
- new chat reset behavior

The UI is part of the engineering contribution because it makes the harness process visible rather than hidden.

---

# 10. Smoke Tests

## 10.1 What Smoke Test Means Here

A smoke test is a lightweight system sanity check.

It is not a full evaluation. It is a quick check that the main pipeline still works.

In this project, smoke tests check whether:

```text
PPO model loads
RAG retrieves evidence
web agent searches official sources
fake URL guard blocks synthetic links
answer grounding returns a result
trace structure is produced
fallback behavior still works
```

---

## 10.2 Example Smoke Tests

The notebook defines smoke tests such as:

```text
1. PEKOM should not be routed to AI bachelor page
2. Residential college should prefer student-affairs residential page
3. CCNA Lab should not invent synthetic URLs
```

These are not random examples. They are chosen to test known fragile parts of the pipeline:

- entity routing
- official URL preference
- fake URL rejection
- web/RAG trace structure

---

# 11. Control Variable Design

The project uses a control-variable style comparison.

The base task remains the same:

```text
UM FSKTM Handbook QA
```

The base model family remains the same:

```text
Qwen3-8B
```

The dataset domain remains the same:

```text
Undergraduate + postgraduate + general UM Handbook knowledge
```

What changes is the system layer:

```text
Baseline 1: SFT only
Baseline 2: SFT + RAG + Harness Agent
Improved: SFT/RAG/Harness + PPO post-training
```

This allows the project to compare which improvements come from:

- parameter learning
- retrieval grounding
- metadata-aware scope control
- official web augmentation
- harness guardrails
- PPO reward shaping

This is more rigorous than simply building three unrelated systems.

---

# 12. Stage-by-stage Comparison Table

| Dimension | Baseline 1: Closed-book SFT | Baseline 2: RAG + SFT + Agent/Harness | Improved Model: PPO + RAG + Agent/Harness |
|---|---|---|---|
| Main research question | Can the model memorize and reproduce handbook QA from SFT? | Does retrieval-grounded evidence improve handbook QA? | Can rule-reward PPO further align answer behavior while keeping RAG/Harness control? |
| Base model | Qwen3-8B | Qwen3-8B | Qwen3-8B |
| Main training method | Supervised fine-tuning | RAG-augmented supervised fine-tuning | Rule-reward PPO post-training |
| Dataset used | 1000 SFT QA rows | SFT QA + metadata + RAG KB + RAG eval | 1000 PPO preference rows |
| Train/validation/test | 800 / 100 / 100 | 8:1:1 RAG-augmented split | 900 train / 100 validation |
| Retrieval | No | Yes | Yes |
| Retrieval type | None | Dense embedding + FAISS + metadata-aware rerank | Same RAG runtime reused |
| Embedding model | None | BAAI/bge-base-en-v1.5 | RAG runtime inherited from Baseline 2 |
| Top-k evidence | None | Top-3 | Top-3 / runtime-dependent |
| Metadata awareness | Hidden metadata only, not used at inference | Yes, scope/source/section aware | Yes, used by RAG/Harness runtime |
| Scope control | Weak; model may confuse UG/PG if prompt is ambiguous | Stronger due to metadata-aware retrieval | Stronger due to RAG + PPO reward + harness |
| Web agent | No | Yes | Yes |
| Official domain control | No | Yes, UM/FSKTM official domain whitelist | Yes, same official-source guardrails |
| Fake URL guard | No | Yes | Yes |
| WAF handling | No | Yes | Yes |
| Evidence judge | No | Yes, Qwen evidence judge | Yes |
| Retry/fallback policy | No | Yes | Yes |
| Answer grounding judge | No | Yes | Yes |
| Completeness guard | No | Yes | Yes |
| UI trace | Basic chat UI | Harness trace panels | Strict PPO + Harness trace panels |
| LoRA rank | 16 | 8 | PPO actor based on LoRA actor/value setup |
| Training epochs | 8 SFT epochs | 20 SFT epochs | 2 PPO epochs |
| Main output artifact | LoRA adapter + merged model + `.pt` export | LoRA adapter, optional non-merged runtime | PPO full inference actor + PPO LoRA adapter + manifest |
| Artifact strictness | Standard save | Adapter/runtime path checks | Manifest, training log count, parameter hash proof, strict gate |
| Key metric | Test token F1 ≈ 0.8869 | RAG token F1 ≈ 0.846 on selected eval; retrieval Hit@3 ≈ 0.954 | PPO validation reward ≈ 0.4778; gold overlap > rejected overlap |
| Strongest contribution | Clean SFT baseline | Evidence-grounded QA and metadata-aware retrieval | Full PPO post-training with strict artifact verification and harnessed runtime |
| Main weakness | Closed-book hallucination risk | More complex runtime, depends on retriever quality | PPO raw outputs still need Harness/RAG due to possible process leakage and fake URLs |
| Control variable role | Establishes parameter-only baseline | Adds retrieval and harness while keeping same domain/model family | Adds PPO reward shaping while preserving RAG/Harness pipeline |

---

# 13. Technical Comparison of the Three Stages

## 13.1 Content-level Difference

| Content Aspect | Baseline 1 | Baseline 2 | Improved Model |
|---|---|---|---|
| Stable handbook facts | Learned into model parameters | Retrieved from handbook KB | Retrieved and answered by PPO-aligned actor |
| Latest or official web info | Not supported | Supported through official web agent | Supported through same official web agent |
| UG vs PG distinction | Learned implicitly | Controlled by metadata retrieval | Controlled by metadata retrieval + reward/harness |
| Evidence visibility | Not shown | Evidence shown in RAG trace | Evidence shown in PPO/Harness trace |
| Hallucination control | Mostly prompt-based | Retrieval + grounding | Retrieval + grounding + reward penalties |
| Fake URL control | Not available | Harness URL guard | Harness URL guard + PPO penalty signal |

---

## 13.2 Engineering-level Difference

| Engineering Aspect | Baseline 1 | Baseline 2 | Improved Model |
|---|---|---|---|
| Notebook purpose | Train and evaluate closed-book SFT model | Build RAG-augmented model and harnessed agent runtime | Train PPO actor and attach it to final harness runtime |
| Runtime complexity | Low | High | Highest |
| Debug trace | Basic | Detailed RAG/Web/Harness trace | Detailed PPO/RAG/Web/Harness trace |
| Failure handling | Minimal | Fallback and guardrail logic | Strict PPO-only fallback prevention plus harness fallback |
| Artifact verification | Basic output save | Adapter/merged path checks | Manifest, training log count, parameter hash proof, strict gate |
| Risk of stale artifact use | Moderate | Moderate | Actively guarded against |
| Demo readiness | Good for simple QA | Strong for grounded QA | Strongest for final controlled system demo |

---

# 14. Why the Improved Model Does Not Replace RAG

A key design decision is that PPO does not replace RAG.

PPO improves the model’s tendency to:

- answer directly
- avoid rejected-style answers
- avoid vague answers
- avoid process leakage
- avoid fake URLs
- avoid repetition collapse
- use evidence-like wording more appropriately

But PPO does not guarantee factual correctness by itself.

Therefore, the final system still needs:

```text
RAG for evidence
Web Agent for official/latest information
Harness for source control
Grounding judge for answer verification
Fallback for weak evidence
```

This is the correct division of responsibility:

```text
SFT: teaches domain answer style
RAG: supplies factual evidence
Agent: finds official external evidence
Harness: controls trust, routing, fallback, and trace
PPO: improves answer behavior according to reward preferences
```

---

# 15. Known Limitations

This project is a strong applied LLM system prototype, but it has limitations.

## 15.1 Not a full human-feedback RLHF system

The PPO stage uses a rule-based reward function. It does not train a separate neural reward model from human preference labels.

Correct description:

```text
Rule-reward PPO post-training
```

Not:

```text
Full RLHF with learned reward model
```

---

## 15.2 Raw PPO generations can still be imperfect

Observed raw PPO generations may include:

- process leakage
- occasional hallucinated URLs
- OCR-like token spacing
- incomplete course titles
- noisy source-text reproduction

The final Harness runtime is therefore necessary.

---

## 15.3 Web search is constrained

The web agent is intentionally limited to official UM/FSKTM sources. It may refuse or fallback when official evidence is weak.

This is a feature, not a bug, because the system prioritizes trustworthiness over open-ended browsing.

---

## 15.4 RAG depends on knowledge base quality

If the RAG KB contains OCR noise or incomplete chunks, the model may inherit that noise. Future work should improve source cleaning and chunk normalization.

---

## 15.5 Notebook-based prototype

The project is implemented as notebooks. A production version should separate modules into:

```text
data/
retrieval/
agent/
harness/
training/
evaluation/
ui/
tests/
```

---

# 16. Recommended Usage

This project is intended for research, coursework, and demonstration purposes.

It is not an official Universiti Malaya system.

For official academic decisions, students should always refer to the official handbook, faculty office, or UM/FSKTM official websites.

---

# 17. Suggested Inference Flow

For final demonstration, use the Improved Model runtime:

```text
1. Load PPO full inference actor if available.
2. If unavailable, load base Qwen3-8B + PPO LoRA adapter.
3. Initialize local handbook RAG.
4. Enable official UM/FSKTM web agent if the question may need external/latest information.
5. Run through TensorTalkHarnessCore.
6. Display answer with evidence trace.
```

Strict runtime requirement:

```text
Non-PPO fallback is forbidden in the final Improved Model demo.
```

---

# 18. Relation to LangChain / LangGraph / LangSmith Concepts

This project does not claim to be a LangChain implementation. Instead, it uses a from-scratch notebook implementation that follows similar engineering ideas.

Official LangChain ecosystem references that influenced the design include:

- [LangChain Agents documentation](https://docs.langchain.com/oss/javascript/langchain/agents): agents combine language models with tools and can iteratively work toward a goal.
- [LangGraph Overview](https://docs.langchain.com/oss/python/langgraph/overview): LangGraph focuses on durable execution, streaming, human-in-the-loop, persistence, and orchestration for agent workflows.
- [LangGraph Graph API](https://docs.langchain.com/oss/python/langgraph/graph-api): agent workflows can be modeled through state, nodes, and edges.
- [LangGraph Workflows and Agents](https://docs.langchain.com/oss/python/langgraph/workflows-agents): workflows use predetermined code paths, while agents are more dynamic in tool usage and process control.
- [LangSmith Evaluation](https://docs.langchain.com/langsmith/evaluation): evaluation can be structured around datasets, evaluators, and experiments.
- [LangSmith Evaluation Types](https://docs.langchain.com/langsmith/evaluation-types): evaluation may include benchmarking, unit tests, regression tests, LLM-as-judge evaluators, code evaluators, and online monitoring.
- [LangSmith Application-specific Evaluation Approaches](https://docs.langchain.com/langsmith/evaluation-approaches): autonomous agents are commonly discussed in terms of tool calling, memory, and planning.

TensorTalk maps these ideas into a custom system:

| LangChain ecosystem idea | TensorTalk implementation |
|---|---|
| Agent uses model + tools | Qwen3 model + local RAG + official web search + URL validator + evidence judge |
| State | Trace dictionaries, evidence bundles, routing flags, model/backend status |
| Nodes | Planning, retrieval, web discovery, filtering, judging, generation, grounding, completeness checking |
| Edges | Conditional retry, official-web route, local-RAG fallback, weak-evidence fallback, final-answer route |
| Planning | Query classification, scope detection, entity-aware routing, web/RAG decision |
| Generation | SFT/PPO Qwen3 actor generates with accepted evidence |
| Evaluation | Evidence judge, answer grounding judge, completeness guard, fake URL checks, smoke tests |
| Observability | TensorTalk collapsed trace panels and diagnostic logs |
| Regression/smoke testing | PEKOM route test, residential-college URL test, CCNA synthetic URL test |

This is why the project can be described as:

> A from-scratch LangChain/LangGraph-inspired RAG agent harness for UM Handbook QA, with a Planning → Generation → Evaluation control loop.

---

# 19. Summary

TensorTalk demonstrates a staged LLM system development workflow:

```text
Baseline 1:
Qwen3-8B learns handbook QA through closed-book SFT.

Baseline 2:
The system adds RAG, dense retrieval, metadata-aware reranking, official web search, and Harness Engineering.

Improved Model:
The system adds full 1000-row rule-reward PPO post-training, strict artifact verification, and a PPO-only final harness runtime.
```

The most important contribution is not only that the model can answer handbook questions, but that the system is controlled, evidence-aware, source-constrained, traceable, and evaluated through a clear baseline progression.

The final system should be understood as:

> **A Qwen3-8B based UM Handbook RAG Agent, improved with rule-reward PPO and controlled by Harness Engineering.**