InosLihka commited on
Commit
1217c1d
·
1 Parent(s): ff20f02

Fix prompt truncation in inference_eval.py: max_seq_length 768 -> 2048

Browse files

Our user prompts (system + observation with 7-step history + anomalies)
are 900-1200 tokens. The model was being loaded with max_seq_length=768,
silently truncating prompts on the left. The model was seeing the END of
the prompt (last few history entries) but missing the system prompt or
older context. This explains why the earlier 'v1 buggy eval' had muted
final_score even though belief_MAE was good — the model was generating
from incomplete context.

The HF Job logs surfaced this clearly:
'Unsloth: Input IDs of shape torch.Size([1, 986]) with length 986 >
the model's max sequence length of 768. We shall truncate it ourselves.'

Bumping to 2048 leaves room for ~1200 prompt + 256 generation + slack.

Files changed (1) hide show
  1. training/inference_eval.py +7 -1
training/inference_eval.py CHANGED
@@ -219,10 +219,16 @@ def main():
219
  if args.model_path and os.path.exists(args.model_path):
220
  try:
221
  from unsloth import FastLanguageModel
 
 
 
 
 
 
222
  model, tokenizer = FastLanguageModel.from_pretrained(
223
  model_name=args.model_path,
224
  load_in_4bit=True,
225
- max_seq_length=768,
226
  )
227
  FastLanguageModel.for_inference(model)
228
  strategies.append("model")
 
219
  if args.model_path and os.path.exists(args.model_path):
220
  try:
221
  from unsloth import FastLanguageModel
222
+ # max_seq_length=2048 must accommodate: user prompt with 7-step
223
+ # history + per-meter anomalies (~900-1200 tokens) PLUS
224
+ # max_new_tokens=256 for the CoT response. Earlier value of 768
225
+ # silently truncated prompts on the LEFT (kept end of prompt,
226
+ # lost system instructions or older meter history), producing
227
+ # incoherent model outputs.
228
  model, tokenizer = FastLanguageModel.from_pretrained(
229
  model_name=args.model_path,
230
  load_in_4bit=True,
231
+ max_seq_length=2048,
232
  )
233
  FastLanguageModel.for_inference(model)
234
  strategies.append("model")