TensorTalk: UM Handbook Qwen3-8B SFT + RAG + Agent + PPO + Harness Engineering

TensorTalk is a staged LLM engineering project built for Universiti Malaya Faculty of Computer Science and Information Technology handbook question answering. The system is designed to answer undergraduate, postgraduate, and general faculty handbook questions using a controlled progression of three experimental stages:

Baseline 1 — Closed-book SFT Qwen3-8B
Baseline 2 — SFT Qwen3-8B + Metadata-aware RAG + Official Web Agent + Harness Engineering
Improved Model — Rule-reward PPO post-training + RAG + Agent + Harness Engineering

The project is not just a simple chatbot. It is a controlled comparison of how an LLM system improves when moving from memorized supervised fine-tuning, to retrieval-grounded answering, and finally to rule-reward post-training with a guarded agentic runtime.

The main idea is:

Baseline 1 tests whether a fine-tuned model can answer handbook questions from parameters alone.
Baseline 2 keeps the same base model family but adds retrieval and harnessed evidence control.
The Improved Model keeps the RAG + Agent + Harness runtime and further adds PPO post-training to make the model better aligned with the desired answer behavior.

1. Project Goal

The goal of this project is to build a reliable and traceable UM Handbook assistant that can answer questions about:

Faculty objectives, vision, mission, history, facilities, and academic calendar
Undergraduate programme details
Postgraduate programme details
Candidature requirements
Grading and academic rules
Industrial training
Academic project requirements
Supervision policy
Thesis/dissertation requirements
Academic integrity and plagiarism
Facilities and labs
Official UM/FSKTM web information when handbook knowledge is insufficient or time-sensitive

The project also aims to demonstrate a complete LLM system development path:

Closed-book SFT
→ RAG-augmented SFT
→ Metadata-aware retrieval
→ Official-source web agent
→ Harness Engineering guardrails
→ PPO rule-reward post-training
→ Strict artifact verification
→ Traceable TensorTalk UI

2. High-level System Overview

The final TensorTalk system contains several layers.

User Question
    ↓
TensorTalk UI
    ↓
Planning / Thinking Display Layer
    ↓
Local Handbook RAG
    ↓
Official UM / FSKTM Web Agent
    ↓
Harness Engineering Guardrails
    ↓
Evidence Judge / Retry / Fallback
    ↓
PPO-trained Qwen3-8B Actor
    ↓
Answer Grounding Judge
    ↓
Completeness Guard
    ↓
Final Answer + Trace Panels

The final model is not used alone. It is wrapped inside a runtime harness that controls:

where the system can search
which sources it can trust
whether web evidence is useful
whether retrieved evidence supports the answer
whether the model produced fake URLs
whether the answer leaked internal reasoning
whether fallback to local handbook RAG is needed
whether the final answer is grounded enough to show

This is why the final stage is better described as:

A PPO-aligned RAG agent system with Harness Engineering, rather than only a fine-tuned model.

3. Dataset Design

3.1 Source Domain

The dataset is built around UM FSKTM undergraduate and postgraduate handbook content. The data is organized into:

SFT question-answer dataset
hidden metadata
RAG knowledge base
RAG evaluation dataset
PPO preference dataset

The project separates model-visible training text from metadata used for retrieval, evaluation, and analysis.

This distinction is important:

Baseline 1 intentionally trains on question-answer text without forcing explicit metadata labels into the model-visible answer.
Baseline 2 uses metadata-aware retrieval to reduce scope confusion.
Stage 3 PPO uses preference pairs and reward functions to shape answer behavior.

3.2 Baseline 1 SFT Dataset

Baseline 1 uses:

SFT_QA_Training_Ready.jsonl

The notebook validates:

Total examples: 1000
Train examples: 800
Validation examples: 100
Test examples: 100
Split ratio: 8:1:1
Duplicate question groups: 0
Duplicate question rows: 0

Each example follows a supervised chat-style format:

{
  "prompt": [
    {
      "role": "system",
      "content": "You are an academic assistant for the Faculty of Computer Science and Information Technology, Universiti Malaya..."
    },
    {
      "role": "user",
      "content": "What are the faculty objectives?"
    }
  ],
  "completion": [
    {
      "role": "assistant",
      "content": "The faculty objectives are..."
    }
  ],
  "question": "...",
  "answer": "..."
}

This stage teaches the model to imitate handbook-style answers directly.

3.3 Baseline 2 RAG Dataset

Baseline 2 uses the same SFT dataset direction, but adds external retrieval resources:

UM_RAG_Knowledge_Base.jsonl
UM_RAG_Evaluation_Dataset.jsonl
SFT_QA_Metadata.jsonl

The RAG knowledge base contains structured fields such as:

kb_id
source_doc
scope_label
section
pages
source_text
retrieval_text
retrieval_keywords
grounded_answer_bank
matched_qa_ids

The RAG knowledge base loaded in the final Stage 3 runtime contains:

Loaded KB rows: 521

The metadata layer allows the system to distinguish:

general
undergraduate
postgraduate

This is important because many handbook questions look similar but require different answers depending on the student scope.

3.4 PPO Preference Dataset

The Improved Model uses:

UM_Handbook_PPO_Preference_Dataset.jsonl

The final PPO run uses the full dataset:

Total PPO preference rows: 1000
Train rows: 900
Validation rows: 100
Train fraction: 0.90

The PPO dataset is not used like normal SFT data. In SFT, the model directly imitates a reference answer. In PPO, the model generates its own answer, receives a reward, and updates toward higher-reward behavior.

4. Baseline 1 — Closed-book SFT Qwen3-8B

4.1 Purpose

Baseline 1 asks a simple question:

Can Qwen3-8B learn UM Handbook question answering from supervised fine-tuning alone?

This is a closed-book baseline. The model does not retrieve handbook evidence during inference. It must answer from what it learned during SFT.

This is useful as a control baseline because it shows what happens when the model relies mainly on parameter memory.

4.2 Model

Baseline 1 uses:

Base model: Qwen/Qwen3-8B
Local path: /scr/user/kevin2002/TensorCat/NLP/UM_Handbook/models/Qwen3-8B

The notebook detected:

Backend: CUDA
GPU: NVIDIA A100-SXM4-80GB
dtype: bfloat16
4-bit QLoRA: enabled

4.3 Training Method

Baseline 1 uses LoRA / QLoRA supervised fine-tuning.

LoRA configuration:

LoRA rank: 16
LoRA alpha: 32
LoRA dropout: 0.10
Target modules:
  - q_proj
  - k_proj
  - v_proj
  - o_proj
  - gate_proj
  - up_proj
  - down_proj

Training configuration:

Epochs: 8
Train split: 800
Validation split: 100
Test split: 100
Per-device train batch size: 2
Per-device eval batch size: 2
Gradient accumulation steps: 8
Learning rate: 1e-4
Packing: False

4.4 Baseline 1 Results

The training completed successfully.

Training summary:

Training steps: 400
Training runtime: ~18.68 minutes for the main train stage
Train loss: 0.4824
Final validation loss: ~0.146
Test loss: ~0.197
Perplexity: ~1.157

Generation metrics:

Validation

Exact match: 0.77
Token F1: 0.9111
ROUGE-1: 0.9122
ROUGE-2: 0.8700
ROUGE-L: 0.8979
SacreBLEU: 81.7240
chrF++: 86.8916
Average prediction words: 36.35
Average reference words: 38.57

Test

Exact match: 0.72
Token F1: 0.8869
ROUGE-1: 0.8857
ROUGE-2: 0.8352
ROUGE-L: 0.8677
SacreBLEU: 81.1138
chrF++: 87.7054
Average prediction words: 38.03
Average reference words: 37.03

4.5 Baseline 1 Strengths

Baseline 1 is strong when the question is close to the training distribution. It can reproduce handbook-style answers well and shows high text overlap with the reference answers.

It is useful because:

it establishes the basic Qwen3-8B SFT capability
it verifies that the dataset format is learnable
it creates a clean closed-book control model
it provides a baseline for later RAG and PPO improvements

4.6 Baseline 1 Limitations

Baseline 1 is still limited because it is a closed-book model.

Main limitations:

No retrieval evidence
It cannot check the handbook at inference time.
Potential hallucination
If the question is out-of-distribution or requires exact source grounding, the model may answer from memory.
Scope confusion
Undergraduate and postgraduate rules may be mixed if the question is ambiguous.
No official web update mechanism
It cannot answer dynamic or latest-information questions reliably.
No harness guardrails
It does not include fake URL detection, evidence judging, WAF handling, or fallback control.

Baseline 1 is therefore a necessary but incomplete starting point.

5. Baseline 2 — RAG + SFT + Metadata-aware Retrieval + Harness Agent

5.1 Purpose

Baseline 2 asks:

What improves if we keep the same Qwen3-8B family but add retrieval-grounded evidence?

The goal is to reduce hallucination and scope confusion by giving the model relevant handbook evidence at inference time.

This stage introduces RAG and agentic harness logic while keeping the same broad model family and handbook task.

5.2 What RAG Means in This Project

RAG stands for Retrieval-Augmented Generation.

In simple terms:

Instead of asking the model to answer only from memory,
the system first retrieves relevant handbook chunks,
then asks the model to answer using those chunks.

In this project, RAG is not just keyword search. It uses:

Transformer embedding model
+ FAISS vector search
+ metadata-aware reranking
+ scope labels
+ top-k evidence blocks

The Baseline 2 retriever uses:

Embedding model: BAAI/bge-base-en-v1.5
Vector index: FAISS
Similarity: inner product after embedding normalization
Top-k retrieval: 3

5.3 Metadata-aware Retrieval

The RAG system uses metadata to control retrieval quality.

Important metadata fields include:

source_doc
scope_label
section
pages
kb_id
knowledge group
retrieval keywords
grounded answer bank

This allows the retriever to prefer the correct audience scope.

Example:

Question: What are the candidature requirements for Master of Software Engineering?
Expected scope: postgraduate

The system should retrieve postgraduate chunks, not undergraduate chunks.

This is one of the main improvements over Baseline 1.

5.4 RAG-augmented Training Dataset

Baseline 2 creates a RAG-augmented dataset where training examples include evidence context.

The training prompt can contain:

User question
+ retrieved handbook evidence
+ source metadata
+ answer instruction

This teaches the model to answer with evidence-aware context rather than only memorized answers.

5.5 Baseline 2 Training Configuration

Baseline 2 uses Qwen3-8B with LoRA fine-tuning.

Configuration:

Base model: Qwen/Qwen3-8B
Embedding model: BAAI/bge-base-en-v1.5
LoRA rank: 8
LoRA alpha: 16
LoRA dropout: 0.05
Target modules:
  - q_proj
  - k_proj
  - v_proj
  - o_proj
  - gate_proj
  - up_proj
  - down_proj
Epochs: 20
Per-device train batch size: 4
Per-device eval batch size: 8
Target global batch size: 8
Learning rate: 8e-5
Max sequence length: 1024
Validation ratio: 0.10
Test ratio: 0.10
Save merged model: False
Runtime model path: base model + LoRA adapter

The notebook uses a safer non-merged runtime path when merged model export is unavailable or memory-expensive.

5.6 Baseline 2 Retrieval Evaluation

Baseline 2 includes a retrieval evaluation set.

Retrieval metrics:

{
  "retrieval_eval_size": 1000,
  "top_k": 3,
  "hit_at_1_primary": 0.821,
  "hit_at_k_primary": 0.954,
  "hit_at_k_same_group": 0.991,
  "scope_match_at_1": 0.996,
  "retriever_type": "dense_embedding + faiss + metadata_rerank",
  "embedding_model_name": "BAAI/bge-base-en-v1.5"
}

Interpretation:

hit_at_1_primary = 0.821 means the top retrieved chunk is exactly the expected primary evidence in 82.1% of cases.
hit_at_k_primary = 0.954 means the correct primary evidence appears within top-3 in 95.4% of cases.
hit_at_k_same_group = 0.991 means a same-group acceptable evidence appears in top-3 in 99.1% of cases.
scope_match_at_1 = 0.996 means the top result almost always matches the correct undergraduate/postgraduate/general scope.

This confirms that the RAG system is not random retrieval. It is a strong metadata-aware retrieval baseline.

5.7 Baseline 2 Generation Evaluation

Generation evaluation was run on a smaller selected set for runtime practicality.

Results:

{
  "generation_eval_size": 20,
  "top_k": 3,
  "plain_exact_match": 0.0,
  "plain_token_f1": 0.3391,
  "rag_exact_match": 0.0,
  "rag_token_f1": 0.8460,
  "rag_minus_plain_exact_match": 0.0,
  "rag_minus_plain_token_f1": 0.5069
}

This shows a large improvement from RAG:

Plain token F1: 0.3391
RAG token F1:   0.8460
Improvement:    +0.5069

This is one of the strongest pieces of evidence in the project.

It shows that retrieval grounding dramatically improves answer quality compared with plain generation.

6. Agent Layer in Baseline 2 and Improved Model

6.1 Why an Agent Is Needed

The handbook is reliable for stable academic rules, but a practical university assistant cannot depend only on static handbook text. Some user questions naturally require official web discovery, source checking, or routing decisions.

Examples:

Who is the current dean?
Where can students find residential college information?
What official page mentions PEKOM?
Where is the official SPeCTRUM page?
What facilities are associated with a specific lab?

For these cases, TensorTalk uses an official-source web agent. The agent is not designed as an unrestricted autonomous browser. It is deliberately designed as a constrained agent because the domain is academic handbook QA, where factual trustworthiness is more important than open-ended exploration.

In practical terms, the agent layer answers this question:

If local handbook RAG is not enough, can the system search official UM/FSKTM sources, reject weak or fake sources, and return evidence safely?

6.2 How This Project Relates to LangChain and LangGraph

TensorTalk does not use LangChain, LangGraph, or LangSmith as the runtime framework. The agent and harness were implemented from scratch in the notebook.

However, the design is conceptually aligned with the official LangChain ecosystem ideas:

LangChain describes agents as systems that combine language models with tools so they can reason about tasks, decide which tools to use, and iteratively work toward a result.
LangGraph describes agent workflows using state, nodes, and edges, where nodes perform computation and edges determine the next transition.
LangSmith describes evaluation as a workflow involving datasets, evaluators, and experiments to compare application versions.
LangChain/LangGraph documentation also distinguishes between predetermined workflows and dynamic agents; TensorTalk intentionally uses a hybrid design because the handbook QA task needs both predictable guardrails and dynamic retrieval decisions.

Therefore, this project is best described as:

A from-scratch implementation of a LangChain/LangGraph-inspired agentic RAG harness, not a project built by directly calling LangChain’s prebuilt agent framework.

This distinction is important. TensorTalk does not simply wrap a LangChain agent. Instead, it manually implements the major control ideas:

State tracking
→ planning
→ retrieval/tool routing
→ source filtering
→ evidence normalization
→ evaluation/judging
→ retry/fallback
→ final generation
→ trace output

This gives the project more transparency because each part of the agent loop is visible in the notebook and UI trace.

6.3 Agent Design Philosophy

The TensorTalk agent is built around four principles.

1. Source-constrained autonomy

The agent can search and fetch information, but only from allowed official sources. It is not free to trust arbitrary search results.

2. Evidence-first generation

The model should not directly answer a web-sensitive question before evidence is collected and judged.

3. Retry and fallback

If official web evidence is weak, blocked, irrelevant, or unsafe, the system can retry with entity-aware search terms or fall back to local handbook RAG.

4. Traceable decisions

The agent does not hide its routing decisions. It records URL candidates, accepted evidence, rejected evidence, grounding decisions, and fallback decisions in trace panels.

6.4 Official UM / FSKTM Web Agent

The web agent is constrained to official UM / FSKTM domains. This is one of the most important safety and reliability choices in the project.

Priority official domains

fsktm.um.edu.my
www.um.edu.my
um.edu.my

Auxiliary official UM-related domains

The project also recognizes selected UM-related service domains when they are relevant to student services, academic systems, library resources, research, career portals, or internal faculty resources:

aasd.um.edu.my
maya.um.edu.my
umlib.um.edu.my
umresearch.um.edu.my
jobs.um.edu.my
careerportal.fsktm.um.edu.my
intra.fsktm.um.edu.my
gallery.fsktm.um.edu.my

The purpose of this whitelist is not to search the whole internet. The purpose is to constrain the agent to sources that are likely to be officially controlled by UM or FSKTM.

6.5 Domain Whitelist Design

The whitelist is used as a domain guard before a page can become trusted evidence.

The system treats URLs in three broad categories:

URL type	Handling
Official UM/FSKTM URL	Can be considered as candidate evidence
UM-related service URL	Can be considered if relevant to the question type
Non-official or synthetic URL	Rejected or downgraded

The whitelist helps prevent common LLM-agent failure cases:

hallucinated programme pages
invented lab pages
fake student service URLs
misrouted search results
random third-party pages
SEO or unrelated pages

For example, the project specifically tests that the agent should not invent or accept URLs like:

programme-ccna-lab-more-detailedly
bachelor-of-computer-science-artificial-intelligence

when those pages are not the correct evidence for the user’s question.

6.6 Web Agent Workflow

The official-source web agent follows a controlled workflow.

User question
    ↓
Intent and entity detection
    ↓
Official-search query construction
    ↓
Candidate URL discovery
    ↓
Domain whitelist filtering
    ↓
Synthetic/fake URL rejection
    ↓
Fetch or static page fallback
    ↓
WAF/block detection
    ↓
Text extraction and normalization
    ↓
Evidence scoring
    ↓
Qwen evidence judge
    ↓
Accept evidence, retry, or fallback to handbook RAG

This means the agent is not only a web search function. It is a guarded evidence acquisition pipeline.

6.7 Planning Inside the Agent

Planning is a visible part of the TensorTalk system.

The planning layer is responsible for deciding:

Is this a stable handbook question?
Is this a latest/current official-web question?
Should local RAG be used first?
Should official web discovery be attempted?
Which entity should be searched?
Which scope should be preferred: undergraduate, postgraduate, general, faculty, or university?
What evidence type is expected: handbook chunk, official page, contact page, facility page, announcement, policy page?

This planning step is aligned with the idea that agentic systems should not directly jump from user question to final answer. They need a control stage that decides which tools and evidence paths are appropriate.

TensorTalk’s planning is not a free-form hidden chain-of-thought that users must trust blindly. It is operationalized through explicit routing variables, trace objects, search decisions, and UI panels.

6.8 Generation Inside the Agent

Generation is the stage where the Qwen3-8B model produces an answer.

However, generation is not allowed to operate alone. The answer generator receives controlled context:

user question
retrieved local handbook evidence
accepted official web evidence
scope hints
source metadata
harness instructions
answer style constraints

The generator is expected to:

answer directly
avoid unsupported claims
avoid fake URLs
avoid exposing internal reasoning
use local handbook evidence when web evidence is weak
prefer official web evidence only when it is relevant and trusted

In the final stage, the generator is the PPO-trained Qwen3 actor, but it is still wrapped by the same RAG and Harness Engineering control layer.

6.9 Evaluation Inside the Agent

Evaluation is the other core part of the agent loop. TensorTalk evaluates both evidence and answers.

Evidence evaluation

The system checks:

Is the source official?
Is the URL synthetic or fake?
Is the page blocked by WAF?
Is the evidence relevant to the user question?
Does the evidence mention the right entity?
Does the evidence match the expected scope?

Answer evaluation

The system checks:

Is the final answer grounded in accepted evidence?
Does it answer the user’s actual question?
Does it leak internal thinking?
Does it invent URLs?
Is it too vague?
Is it incomplete enough to require fallback or rewrite?

This creates a full agentic loop:

Planning
→ Retrieval / tool use
→ Generation
→ Evaluation
→ Retry or fallback
→ Final answer

6.10 Why the Agent Is Not Fully Autonomous

The agent is intentionally not fully autonomous.

A fully autonomous browsing agent may:

search too broadly
trust wrong sources
follow irrelevant pages
invent missing pages
overuse web search
ignore handbook evidence
produce unsupported answers

TensorTalk instead uses a constrained model:

Dynamic when needed
Guarded by default
Official-source-only for web evidence
RAG-first for handbook-stable questions
Fallback-aware when web evidence is weak
Traceable for debugging and demonstration

This is more appropriate for a university handbook assistant.

7. Harness Engineering

7.1 What Harness Engineering Means Here

Harness Engineering is the external control system around the LLM, RAG, and agent.

A simple analogy:

The LLM/agent is the car.
Harness Engineering is the guardrail, traffic rule, checkpoint, fallback route, dashboard, and driving examiner.

The model can generate fluent answers, but the harness controls:

what it can search
which domains are trusted
which URLs are rejected
which evidence is useful
when to retry
when to fallback
whether the answer is grounded
whether the UI should show warning traces
whether the final response is safe enough to display

In TensorTalk, Harness Engineering is not just prompt engineering. Prompt engineering tells the model what to do. Harness Engineering builds the surrounding execution system that checks whether the model actually did it correctly.

7.2 From Prompt Engineering to Harness Engineering

Prompt engineering is like telling a driver:

Please drive carefully.

Harness Engineering is like building:

lane barriers
speed checks
traffic rules
navigation checkpoints
fallback routes
dashboards
incident logs

In this project, prompt engineering alone is not enough because the model may still:

invent fake URLs
mix undergraduate and postgraduate rules
leak internal reasoning
trust weak web snippets
answer without evidence
overuse web search
ignore local handbook RAG

The harness prevents or detects these failures through code-level controls, not only prompt instructions.

7.3 From-scratch LangChain-style Harness

TensorTalk’s harness was built manually rather than by importing a prebuilt LangChain/LangGraph agent.

The implementation follows the same conceptual loop used in many modern agent frameworks:

Planning
→ Tool / Retrieval Action
→ Generation
→ Evaluation
→ Retry / Fallback
→ Finalization

But each component is implemented explicitly:

Conceptual framework idea	TensorTalk from-scratch implementation
Agent state	Trace dictionaries, evidence bundles, routing flags, runtime status
Tools	Local RAG retriever, official web search/fetch, URL validator, evidence judge
Nodes	Planning, retrieval, web discovery, evidence filtering, judging, generation, grounding
Edges / transitions	Conditional retry, weak-evidence fallback, RAG fallback, final answer route
Evaluation	Qwen evidence judge, rule checks, answer grounding judge, smoke tests
Observability	Collapsed UI trace panels and printed diagnostic outputs

This makes the system easier to inspect in an academic notebook because the control logic is visible.

7.4 Planning → Generation → Evaluation Closed Loop

The most important Harness Engineering contribution in TensorTalk is the closed loop:

Planning
    ↓
Generation
    ↓
Evaluation
    ↓
Retry / Fallback / Finalization

Planning

The planning layer decides how to handle the query.

It considers:

question type
scope
entity
whether local RAG is enough
whether official web is needed
whether the query is dynamic/current
whether the answer should be handbook-grounded or web-grounded

Generation

The generation layer produces an answer using controlled evidence.

It receives:

local handbook chunks
official web evidence
scope hints
source metadata
answer constraints

Evaluation

The evaluation layer checks the result.

It evaluates:

source trust
URL validity
evidence relevance
answer grounding
completeness
process leakage
fake URLs
fallback need

If evaluation fails, the system can retry, reroute, or fall back.

This is the engineering loop that makes TensorTalk more than a simple RAG chatbot.

7.5 Standardized Harness Core Pipeline

The final standardized TensorTalk Harness Core follows this pipeline:

User Question
    ↓
Planning Layer
    ↓
Local Handbook RAG
    ↓
Official Web Discovery
    ↓
Domain Guard
    ↓
Fake URL Guard
    ↓
WAF Detection
    ↓
Evidence Normalizer
    ↓
Qwen Evidence Judge
    ↓
Entity-aware Retry
    ↓
Weak Evidence Fallback
    ↓
PPO/SFT Answer Generator
    ↓
Answer Grounding Judge
    ↓
Completeness Guard
    ↓
Final Answer
    ↓
UI Trace

This pipeline is intentionally explicit. Each part has a clear job.

7.6 Harness State and Trace Objects

The harness keeps structured trace data so that every answer can be inspected.

Typical trace information includes:

retrieved local RAG chunks
candidate web URLs
accepted official URLs
rejected URLs
web evidence bundle
harness core route
evidence judge result
answer grounding result
fallback reason
final answer preview

This is similar in spirit to observability and tracing in agent platforms, but implemented directly in the notebook and UI.

7.7 Domain Guard

The domain guard checks whether a candidate source belongs to the allowed official domain set.

It protects against:

random third-party websites
unofficial mirrors
search result noise
LLM-fabricated domains
wrong university pages

It also makes the system explainable. If the agent rejects a page, the trace can show why.

7.8 Fake URL Guard

The fake URL guard is one of the most important parts of the project because raw LLM generations can invent plausible-looking URLs.

Examples of risky synthetic URLs include:

https://spectrum.umlms
http://spectrux.medicum
programme-ccna-lab-more-detailedly
https://aasd um edu my/studetn

The guard checks and rejects URLs that:

are malformed
look fabricated
contain suspicious path patterns
do not belong to allowed domains
are query-fabricated rather than discovered from official search/fetch

The PPO reward function also penalizes hallucinated URLs, but the harness is still necessary because reward shaping does not guarantee perfect URL behavior.

7.9 WAF Detection

Some official pages can be blocked, partially loaded, or protected by web application firewalls.

The WAF-aware harness detects cases where:

the page cannot be fetched normally
the content is a block page instead of the real page
the browser click fails
the official site returns insufficient text

When this happens, the system avoids treating the blocked page as strong evidence. It can use diagnostics, retry, static fallback, or local RAG fallback.

7.10 Evidence Normalizer

Fetched web pages and handbook chunks may be noisy.

The evidence normalizer attempts to convert them into a consistent evidence structure:

title
url
source type
domain
text snippet
score
scope
entity
reason

This makes later judging and UI display easier.

7.11 Qwen Evidence Judge

The Qwen evidence judge is used to decide whether retrieved evidence actually helps answer the user’s question.

It checks:

Does the evidence mention the right entity?
Does it answer the question directly?
Is it only loosely related?
Is it a wrong programme/page?
Is it official but irrelevant?

This is important because official sources can still be irrelevant. A page can be official and still be the wrong evidence.

7.12 Entity-aware Retry

If the first web discovery result is weak or misrouted, the harness can retry with better query terms.

For example, if a question about PEKOM gets routed toward an AI bachelor programme page, the harness should retry using terms related to:

PEKOM
Persatuan Komputer UM
student society
FSKTM student association

This prevents the agent from accepting the first official-looking but semantically wrong page.

7.13 Weak Evidence Fallback

If the official web evidence is weak, TensorTalk can fall back to local handbook RAG.

This prevents a common agent failure:

The system found a web page, so it trusts it even though it does not answer the question.

Instead, TensorTalk uses:

web evidence if strong
local handbook RAG if web evidence is weak
hybrid answer if both are useful
refusal/uncertainty if neither is sufficient

7.14 Answer Grounding Judge

After answer generation, the answer grounding judge checks whether the final answer is supported by the accepted evidence.

It helps catch cases where:

retrieval was correct but generation added unsupported claims
the model invented a URL
the model mixed evidence from different scopes
the answer contains a statement that does not appear in evidence

This is the evaluation part of the Planning → Generation → Evaluation loop.

7.15 Completeness Guard

The completeness guard checks whether the answer is too short, vague, or incomplete.

It can identify cases where the answer:

only repeats the question
does not include required details
misses key fields
does not answer the requested scope
cuts off early

Depending on runtime settings, this can trigger a rewrite or fallback.

7.16 Smoke Tests as Harness Unit Checks

The smoke tests are lightweight checks that make sure the harness pipeline still works after model or code changes.

Examples:

PEKOM should not be routed to the AI bachelor page.
Residential college should prefer the student-affairs residential page.
CCNA Lab should not invent synthetic URLs.

These tests check:

routing
URL filtering
official page preference
fake URL rejection
answer grounding trace
harness core route

They are not a full benchmark. They are fast sanity checks that the system still runs through the expected pipeline.

7.17 Why Harness Engineering Is Central to This Project

The final system does not rely on only one technique.

SFT gives domain answer style.
RAG gives handbook evidence.
The web agent gives official external evidence.
PPO improves answer behavior.
Harness Engineering controls the whole system.

Without the harness, the system would still be vulnerable to:

wrong source selection
fake URLs
weak web evidence
scope confusion
process leakage
unsupported final answers
stale artifact loading

Therefore, Harness Engineering is the system-level contribution that connects SFT, RAG, Agent, and PPO into one controlled workflow.

8. Improved Model — PPO Rule-reward Post-training + RAG + Agent + Harness

8.1 Purpose

The Improved Model asks:

Can we further improve the model’s behavior after SFT/RAG by using PPO reward-based post-training?

Baseline 2 already improves factual grounding through RAG and Harness Engineering. The Improved Model adds PPO to shape the model’s behavior.

The goal is not to replace RAG. The goal is to make the model more aligned with the desired answer style and safety behavior.

8.2 What PPO Means in This Project

PPO stands for Proximal Policy Optimization.

In simple terms:

SFT teaches the model by imitation.
PPO lets the model generate answers, scores them with a reward function, and updates the model toward higher-reward answers.

In this project:

Actor model: Qwen3-8B + LoRA
Critic/value head: TRL value head model
Reference model: frozen Qwen3-8B reference
Reward: rule-based preference reward function
KL control: used to avoid drifting too far from the reference model

8.3 Rule-based Reward Function

This project uses a rule-based reward function rather than a separately trained neural reward model.

The reward function evaluates:

gold answer similarity
rejected answer penalty
evidence overlap
scope correctness
hallucinated URL penalty
vague answer penalty
process/thinking leakage penalty
direct answer bonus
repetition penalty
degeneration/collapse penalty

This is why the model card should describe the final stage as:

Rule-reward PPO post-training

not:

Full RLHF with a trained reward model

The reward model type recorded in the notebook is:

rule_based_preference_reward_function
uses_separate_neural_reward_model: False

8.4 PPO Training Configuration

The final PPO run uses:

Preference dataset rows: 1000
Train rows: 900
Validation rows: 100
MAX_PPO_ROWS: None
Train fraction: 0.90
PPO epochs: 2
Batch size: 2
Mini-batch size: 1
Max new tokens: 72
Max PPO steps per epoch: None
Planned steps per epoch: 450
Total planned steps: 900
Learning rate: 2e-6
Target KL: 0.10
Generation temperature: 0.45
Top-p: 0.78
Repetition penalty: 1.3
No-repeat ngram size: 4

The run completed successfully:

Global PPO steps: 900 / 900
Elapsed time: 04:47:59
Degenerate ratio: 0.00%

8.5 PPO Artifact Verification

The Stage 3 notebook includes strict artifact verification.

This is important because PPO notebooks can easily appear to run while silently saving old or incomplete artifacts.

The strict save cell verifies:

training_log exists
training_log records = 900
expected steps = 900
MAX_PPO_ROWS = None
train rows = 900
valid rows = 100
NUM_PPO_EPOCHS = 2
MAX_PPO_STEPS_PER_EPOCH = None
parameter hash changed after PPO
PPO inference full actor exists
PPO LoRA adapter exists
non-PPO fallback forbidden

The final strict save output confirms:

Final PPO records saved: 900 / expected 900
Strict full PPO artifact contract passed.

The parameter change proof confirms:

aggregate_hash_changed: true
changed_trainable_tensors: 506
unchanged_trainable_tensors: 0

This proves that PPO training changed the trainable LoRA/value-head parameters rather than merely running a dry notebook.

8.6 Strict PPO-only Runtime

The final runtime is configured so that the UI must use PPO artifacts only.

The strict PPO gate confirms:

PPO records: 900
PPO full actor usable: True
PPO LoRA adapter usable: True
Strict PPO-only UI mode: True

The runtime loading order is:

1. PPO full inference actor if full weights exist
2. Otherwise base Qwen3-8B + PPO LoRA adapter
3. Non-PPO fallback is forbidden

This prevents the final demo from accidentally loading an old Baseline 2 model or a stale 150-step PPO proof artifact.

8.7 PPO Validation

The PPO-only validation evaluation uses a held-out validation sample.

The displayed validation summary is:

reward:            0.477789
gold_overlap:      0.255351
rejected_overlap:  0.155080

Interpretation:

reward is positive
gold overlap is higher than rejected overlap
the PPO-trained actor tends to move closer to preferred answers than rejected answers

This does not mean the PPO model is perfect. It means the reward-shaped behavior is directionally positive.

8.8 PPO Limitations

The PPO run is successful, but the raw PPO generations still show some imperfections.

Observed issues include:

Process leakage
Some outputs still include phrases like:
```
Okay, let me try to figure out...
Wait, I need to check again...
```
The reward function penalizes this, but it is not completely eliminated.
Occasional hallucinated URLs
Some raw generations may still invent URLs. The harness fake URL guard is therefore still necessary.
OCR-style text artifacts
Some source chunks contain spacing or OCR issues, and the model may reproduce them.
KL can be high
Some PPO logs show high objective/kl, meaning the PPO actor can drift noticeably from the reference model. However, the run completed with:
```
degenerate_ratio = 0.00%
```
and no detected repetition collapse.
RAG/Harness remains necessary
PPO improves model behavior, but it does not replace retrieval grounding or guardrails.

9. TensorTalk UI

The project includes a WhatsApp-style Jupyter HTML UI called TensorTalk.

The UI supports:

chat-style interface
TensorCat avatar
RAG on/off control
web agent on/off control
collapsed trace panels
retrieved evidence display
web evidence display
planning/thinking display layer
harness decision trace
answer grounding information
strict PPO artifact loading
new chat reset behavior

The UI is part of the engineering contribution because it makes the harness process visible rather than hidden.

10. Smoke Tests

10.1 What Smoke Test Means Here

A smoke test is a lightweight system sanity check.

It is not a full evaluation. It is a quick check that the main pipeline still works.

In this project, smoke tests check whether:

PPO model loads
RAG retrieves evidence
web agent searches official sources
fake URL guard blocks synthetic links
answer grounding returns a result
trace structure is produced
fallback behavior still works

10.2 Example Smoke Tests

The notebook defines smoke tests such as:

1. PEKOM should not be routed to AI bachelor page
2. Residential college should prefer student-affairs residential page
3. CCNA Lab should not invent synthetic URLs

These are not random examples. They are chosen to test known fragile parts of the pipeline:

entity routing
official URL preference
fake URL rejection
web/RAG trace structure

11. Control Variable Design

The project uses a control-variable style comparison.

The base task remains the same:

UM FSKTM Handbook QA

The base model family remains the same:

Qwen3-8B

The dataset domain remains the same:

Undergraduate + postgraduate + general UM Handbook knowledge

What changes is the system layer:

Baseline 1: SFT only
Baseline 2: SFT + RAG + Harness Agent
Improved: SFT/RAG/Harness + PPO post-training

This allows the project to compare which improvements come from:

parameter learning
retrieval grounding
metadata-aware scope control
official web augmentation
harness guardrails
PPO reward shaping

This is more rigorous than simply building three unrelated systems.

12. Stage-by-stage Comparison Table

Dimension	Baseline 1: Closed-book SFT	Baseline 2: RAG + SFT + Agent/Harness	Improved Model: PPO + RAG + Agent/Harness
Main research question	Can the model memorize and reproduce handbook QA from SFT?	Does retrieval-grounded evidence improve handbook QA?	Can rule-reward PPO further align answer behavior while keeping RAG/Harness control?
Base model	Qwen3-8B	Qwen3-8B	Qwen3-8B
Main training method	Supervised fine-tuning	RAG-augmented supervised fine-tuning	Rule-reward PPO post-training
Dataset used	1000 SFT QA rows	SFT QA + metadata + RAG KB + RAG eval	1000 PPO preference rows
Train/validation/test	800 / 100 / 100	8:1:1 RAG-augmented split	900 train / 100 validation
Retrieval	No	Yes	Yes
Retrieval type	None	Dense embedding + FAISS + metadata-aware rerank	Same RAG runtime reused
Embedding model	None	BAAI/bge-base-en-v1.5	RAG runtime inherited from Baseline 2
Top-k evidence	None	Top-3	Top-3 / runtime-dependent
Metadata awareness	Hidden metadata only, not used at inference	Yes, scope/source/section aware	Yes, used by RAG/Harness runtime
Scope control	Weak; model may confuse UG/PG if prompt is ambiguous	Stronger due to metadata-aware retrieval	Stronger due to RAG + PPO reward + harness
Web agent	No	Yes	Yes
Official domain control	No	Yes, UM/FSKTM official domain whitelist	Yes, same official-source guardrails
Fake URL guard	No	Yes	Yes
WAF handling	No	Yes	Yes
Evidence judge	No	Yes, Qwen evidence judge	Yes
Retry/fallback policy	No	Yes	Yes
Answer grounding judge	No	Yes	Yes
Completeness guard	No	Yes	Yes
UI trace	Basic chat UI	Harness trace panels	Strict PPO + Harness trace panels
LoRA rank	16	8	PPO actor based on LoRA actor/value setup
Training epochs	8 SFT epochs	20 SFT epochs	2 PPO epochs
Main output artifact	LoRA adapter + merged model + `.pt` export	LoRA adapter, optional non-merged runtime	PPO full inference actor + PPO LoRA adapter + manifest
Artifact strictness	Standard save	Adapter/runtime path checks	Manifest, training log count, parameter hash proof, strict gate
Key metric	Test token F1 ≈ 0.8869	RAG token F1 ≈ 0.846 on selected eval; retrieval Hit@3 ≈ 0.954	PPO validation reward ≈ 0.4778; gold overlap > rejected overlap
Strongest contribution	Clean SFT baseline	Evidence-grounded QA and metadata-aware retrieval	Full PPO post-training with strict artifact verification and harnessed runtime
Main weakness	Closed-book hallucination risk	More complex runtime, depends on retriever quality	PPO raw outputs still need Harness/RAG due to possible process leakage and fake URLs
Control variable role	Establishes parameter-only baseline	Adds retrieval and harness while keeping same domain/model family	Adds PPO reward shaping while preserving RAG/Harness pipeline

13. Technical Comparison of the Three Stages

13.1 Content-level Difference

Content Aspect	Baseline 1	Baseline 2	Improved Model
Stable handbook facts	Learned into model parameters	Retrieved from handbook KB	Retrieved and answered by PPO-aligned actor
Latest or official web info	Not supported	Supported through official web agent	Supported through same official web agent
UG vs PG distinction	Learned implicitly	Controlled by metadata retrieval	Controlled by metadata retrieval + reward/harness
Evidence visibility	Not shown	Evidence shown in RAG trace	Evidence shown in PPO/Harness trace
Hallucination control	Mostly prompt-based	Retrieval + grounding	Retrieval + grounding + reward penalties
Fake URL control	Not available	Harness URL guard	Harness URL guard + PPO penalty signal

13.2 Engineering-level Difference

Engineering Aspect	Baseline 1	Baseline 2	Improved Model
Notebook purpose	Train and evaluate closed-book SFT model	Build RAG-augmented model and harnessed agent runtime	Train PPO actor and attach it to final harness runtime
Runtime complexity	Low	High	Highest
Debug trace	Basic	Detailed RAG/Web/Harness trace	Detailed PPO/RAG/Web/Harness trace
Failure handling	Minimal	Fallback and guardrail logic	Strict PPO-only fallback prevention plus harness fallback
Artifact verification	Basic output save	Adapter/merged path checks	Manifest, training log count, parameter hash proof, strict gate
Risk of stale artifact use	Moderate	Moderate	Actively guarded against
Demo readiness	Good for simple QA	Strong for grounded QA	Strongest for final controlled system demo

14. Why the Improved Model Does Not Replace RAG

A key design decision is that PPO does not replace RAG.

PPO improves the model’s tendency to:

answer directly
avoid rejected-style answers
avoid vague answers
avoid process leakage
avoid fake URLs
avoid repetition collapse
use evidence-like wording more appropriately

But PPO does not guarantee factual correctness by itself.

Therefore, the final system still needs:

RAG for evidence
Web Agent for official/latest information
Harness for source control
Grounding judge for answer verification
Fallback for weak evidence

This is the correct division of responsibility:

SFT: teaches domain answer style
RAG: supplies factual evidence
Agent: finds official external evidence
Harness: controls trust, routing, fallback, and trace
PPO: improves answer behavior according to reward preferences

15. Known Limitations

This project is a strong applied LLM system prototype, but it has limitations.

15.1 Not a full human-feedback RLHF system

The PPO stage uses a rule-based reward function. It does not train a separate neural reward model from human preference labels.

Correct description:

Rule-reward PPO post-training

Not:

Full RLHF with learned reward model

15.2 Raw PPO generations can still be imperfect

Observed raw PPO generations may include:

process leakage
occasional hallucinated URLs
OCR-like token spacing
incomplete course titles
noisy source-text reproduction

The final Harness runtime is therefore necessary.

15.3 Web search is constrained

The web agent is intentionally limited to official UM/FSKTM sources. It may refuse or fallback when official evidence is weak.

This is a feature, not a bug, because the system prioritizes trustworthiness over open-ended browsing.

15.4 RAG depends on knowledge base quality

If the RAG KB contains OCR noise or incomplete chunks, the model may inherit that noise. Future work should improve source cleaning and chunk normalization.

15.5 Notebook-based prototype

The project is implemented as notebooks. A production version should separate modules into:

data/
retrieval/
agent/
harness/
training/
evaluation/
ui/
tests/

16. Recommended Usage

This project is intended for research, coursework, and demonstration purposes.

It is not an official Universiti Malaya system.

For official academic decisions, students should always refer to the official handbook, faculty office, or UM/FSKTM official websites.

17. Suggested Inference Flow

For final demonstration, use the Improved Model runtime:

1. Load PPO full inference actor if available.
2. If unavailable, load base Qwen3-8B + PPO LoRA adapter.
3. Initialize local handbook RAG.
4. Enable official UM/FSKTM web agent if the question may need external/latest information.
5. Run through TensorTalkHarnessCore.
6. Display answer with evidence trace.

Strict runtime requirement:

Non-PPO fallback is forbidden in the final Improved Model demo.

18. Relation to LangChain / LangGraph / LangSmith Concepts

This project does not claim to be a LangChain implementation. Instead, it uses a from-scratch notebook implementation that follows similar engineering ideas.

Official LangChain ecosystem references that influenced the design include:

LangChain Agents documentation: agents combine language models with tools and can iteratively work toward a goal.
LangGraph Overview: LangGraph focuses on durable execution, streaming, human-in-the-loop, persistence, and orchestration for agent workflows.
LangGraph Graph API: agent workflows can be modeled through state, nodes, and edges.
LangGraph Workflows and Agents: workflows use predetermined code paths, while agents are more dynamic in tool usage and process control.
LangSmith Evaluation: evaluation can be structured around datasets, evaluators, and experiments.
LangSmith Evaluation Types: evaluation may include benchmarking, unit tests, regression tests, LLM-as-judge evaluators, code evaluators, and online monitoring.
LangSmith Application-specific Evaluation Approaches: autonomous agents are commonly discussed in terms of tool calling, memory, and planning.

TensorTalk maps these ideas into a custom system:

LangChain ecosystem idea	TensorTalk implementation
Agent uses model + tools	Qwen3 model + local RAG + official web search + URL validator + evidence judge
State	Trace dictionaries, evidence bundles, routing flags, model/backend status
Nodes	Planning, retrieval, web discovery, filtering, judging, generation, grounding, completeness checking
Edges	Conditional retry, official-web route, local-RAG fallback, weak-evidence fallback, final-answer route
Planning	Query classification, scope detection, entity-aware routing, web/RAG decision
Generation	SFT/PPO Qwen3 actor generates with accepted evidence
Evaluation	Evidence judge, answer grounding judge, completeness guard, fake URL checks, smoke tests
Observability	TensorTalk collapsed trace panels and diagnostic logs
Regression/smoke testing	PEKOM route test, residential-college URL test, CCNA synthetic URL test

This is why the project can be described as:

A from-scratch LangChain/LangGraph-inspired RAG agent harness for UM Handbook QA, with a Planning → Generation → Evaluation control loop.

19. Summary

TensorTalk demonstrates a staged LLM system development workflow:

Baseline 1:
Qwen3-8B learns handbook QA through closed-book SFT.

Baseline 2:
The system adds RAG, dense retrieval, metadata-aware reranking, official web search, and Harness Engineering.

Improved Model:
The system adds full 1000-row rule-reward PPO post-training, strict artifact verification, and a PPO-only final harness runtime.

The most important contribution is not only that the model can answer handbook questions, but that the system is controlled, evidence-aware, source-constrained, traceable, and evaluated through a clear baseline progression.

The final system should be understood as:

A Qwen3-8B based UM Handbook RAG Agent, improved with rule-reward PPO and controlled by Harness Engineering.

Downloads last month: -; Downloads are not tracked for this model. How to track

Model tree for TensorCat/TensorTalk

Base model

Qwen/Qwen3-8B-Base

Finetuned

Qwen/Qwen3-8B

Adapter

(1401)

this model