config block_ff_dim wrong
#18
by florianvoss - opened
This comment has been hidden
Hi thanks for reaching out. The config is correct, but you are also correct that the resulting dimension is 8192. See https://github.com/huggingface/transformers/blob/8cb5963cc22174954e7dca2c0a3320b7dc2f4edc/src/transformers/models/lfm2/modeling_lfm2.py#L109 where the block_auto_adjust_ff_dim=True argument adjusts the dimension.
jimmysmith1919 changed discussion status to closed