Are you guys going to add other MoE stuff?
Would it be doable for Gemma4 to have even more active and total params? For example configs such as:
52B A12B (Jamba Mini has this config)
48B A12B
60B A4B
32B A8B
etc...
I would mostly be interested in 48B A12B,52B A12B,60B A4B but let me know what others think!
i'd love a 120B moe but i don't think they'd give us something that compete with gemini.
Hey,
Thanks for sharing those MoE configuration ideas! It's awesome to see the community thinking about scaling, setups like a 48B A12B would definetly be exciting to explore.
If we expand the Gemma family or explore new configurations in the future, we will share the updates thoguh official developer channels. In the meantime, we can't wait to see what the community builds with the 26B MoE.
Thanks again!