Spaces:
Sleeping
Sleeping
Fix prompt truncation in inference_eval.py: max_seq_length 768 -> 2048
Browse filesOur user prompts (system + observation with 7-step history + anomalies)
are 900-1200 tokens. The model was being loaded with max_seq_length=768,
silently truncating prompts on the left. The model was seeing the END of
the prompt (last few history entries) but missing the system prompt or
older context. This explains why the earlier 'v1 buggy eval' had muted
final_score even though belief_MAE was good — the model was generating
from incomplete context.
The HF Job logs surfaced this clearly:
'Unsloth: Input IDs of shape torch.Size([1, 986]) with length 986 >
the model's max sequence length of 768. We shall truncate it ourselves.'
Bumping to 2048 leaves room for ~1200 prompt + 256 generation + slack.
training/inference_eval.py
CHANGED
|
@@ -219,10 +219,16 @@ def main():
|
|
| 219 |
if args.model_path and os.path.exists(args.model_path):
|
| 220 |
try:
|
| 221 |
from unsloth import FastLanguageModel
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 222 |
model, tokenizer = FastLanguageModel.from_pretrained(
|
| 223 |
model_name=args.model_path,
|
| 224 |
load_in_4bit=True,
|
| 225 |
-
max_seq_length=
|
| 226 |
)
|
| 227 |
FastLanguageModel.for_inference(model)
|
| 228 |
strategies.append("model")
|
|
|
|
| 219 |
if args.model_path and os.path.exists(args.model_path):
|
| 220 |
try:
|
| 221 |
from unsloth import FastLanguageModel
|
| 222 |
+
# max_seq_length=2048 must accommodate: user prompt with 7-step
|
| 223 |
+
# history + per-meter anomalies (~900-1200 tokens) PLUS
|
| 224 |
+
# max_new_tokens=256 for the CoT response. Earlier value of 768
|
| 225 |
+
# silently truncated prompts on the LEFT (kept end of prompt,
|
| 226 |
+
# lost system instructions or older meter history), producing
|
| 227 |
+
# incoherent model outputs.
|
| 228 |
model, tokenizer = FastLanguageModel.from_pretrained(
|
| 229 |
model_name=args.model_path,
|
| 230 |
load_in_4bit=True,
|
| 231 |
+
max_seq_length=2048,
|
| 232 |
)
|
| 233 |
FastLanguageModel.for_inference(model)
|
| 234 |
strategies.append("model")
|