File size: 2,478 Bytes
1bfed6e | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 | ---
language:
- en
license: apache-2.0
base_model:
- sentence-transformers/all-MiniLM-L6-v2
datasets:
- itsjhuang/watsonx-docs-document-type
tags:
- text-classification
- embeddings
- technical-documentation
metrics:
- accuracy
- f1
---
# Watsonx Docs Document Type Classifier
Binary classifier for IBM Watsonx technical documentation pages.
Given a documentation page, the model predicts whether it is:
- `conceptual` (0): primarily used to understand or look up information
- `how-to` (1): primarily used to complete a procedure or fix a problem
## Model Details
| | |
|---|---|
| Base embeddings | sentence-transformers/all-MiniLM-L6-v2 |
| Classifier | LinearSVC (C=1.0, max_iter=2000) |
| Training dataset | [itsjhuang/watsonx-docs-document-type](https://huggingface.co/datasets/itsjhuang/watsonx-docs-document-type) |
| Input | title + first 800 words of document |
| Output | `conceptual` or `how-to` |
## Evaluation Results
Three conditions were trained and evaluated. The best model (B) was selected by test macro F1.
| Condition | Embedding Model | Classifier | Train Acc | Train F1 | Test Acc | Test F1 |
|---|---|---|---:|---:|---:|---:|
| A | all-MiniLM-L6-v2 | Logistic Regression | 0.879 | 0.879 | 0.817 | 0.817 |
| B ✅ | all-MiniLM-L6-v2 | LinearSVC | 0.971 | 0.971 | 0.867 | 0.867 |
| C | bge-small-en-v1.5 | Logistic Regression | 0.864 | 0.864 | 0.833 | 0.833 |
Confusion matrices for each condition are available in the repository files.
## Usage
```python
import joblib
import numpy as np
from sentence_transformers import SentenceTransformer
embedder = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")
clf = joblib.load("best_model.joblib")
def softmax(x):
e = np.exp(x - np.max(x))
return e / e.sum()
def predict(text):
embedding = embedder.encode([text], convert_to_numpy=True)
scores = clf.decision_function(embedding)[0]
if np.ndim(scores) == 0:
scores = np.array([-scores, scores])
probs = softmax(scores)
labels = ["conceptual", "how-to"]
return dict(zip(labels, probs))
```
## Limitations
- Trained on IBM Watsonx documentation only; may not generalize to other
technical documentation domains.
- Label boundary between weak procedural pages and conceptual capability
descriptions remains a residual source of error.
## Source Dataset
Derived from
[`ibm-research/watsonxDocsQA`](https://huggingface.co/datasets/ibm-research/watsonxDocsQA),
licensed under Apache 2.0.
|