8-kv-heads
#14
by ArthurZ HF Staff - opened
No description provided.
Hi there! I noticed that the 8 kv heads PR was merged in for the other 405b checkpoints, is there an ETA on landing this one? Thanks for the help!
Merging now!
ArthurZ changed pull request status to open
Does num_key_value_heads in config.json need to be updated as well after this PR is merged?
TYSM @ArthurZ !! just a heads up tho that this is probably an upload issue, but it appears that model parts update: those 4 files are not affected by the 16 -> 8 kv head change. { 002, [ 107 - 109 ] } were missed from the list / diff above
Looks like this is not yet merged?
Let me update the value in the config to merge!
(I don't have rights yet 😿)
Looking forward to trying!
osanseviero changed pull request status to merged