RoPE theta 5mln instead of 1mln

#15

by Michalea - opened Feb 25

Feb 25

Hello, do anyone know why theta is 5mln while the context is 262k? - Qwen recommends usage of 5mln. theta with 1mln context.

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment