Hi,
https://huggingface.co/yanolja/YanoljaNEXT-Rosetta-4B-2510-GGUF
Just uploaded. Since it is my first time creating GGUF files, there could be any mistakes. If so, please let us know. Thanks!

Damn… nice, nice. I knew it! Lesson learned, I should wait for the official GGUF. The people who make GGUFs sometimes mess things up. This one works fine though. ❤️

I’m gonna do more testing since the system prompt is pretty sensitive.

Other models understand this: “Translate the following Japanese text to English. Output only the English translation, nothing else.” I was wondering why it was losing BLEU and semantic scores.

Turns out it was outputting Japanese instead of English, so I had to change it to:
prompt="Translate the user's text to English."

from rank 23 it jump to rank 10 hehe

I might need to elaborate what I mean about "Turns out it was outputting Japanese instead of English"

Yes, it could be very sensitive to the system prompt format because we did not make any variations of the system prompt in the training dataset.
However, it looks very weird that the Rosetta model underperformed compared to gemma-3-4b-it.
If possible, can you let us know how we can reproduce it? Thanks for testing!

Yes, it could be very sensitive to the system prompt format because we did not make any variations of the system prompt in the training dataset.
However, it looks very weird that the Rosetta model underperformed compared to gemma-3-4b-it.
If possible, can you let us know how we can reproduce it? Thanks for testing!

sure but 😅 mind that this is like basic basic test https://github.com/TomieAi/basic-benchmark.

I have trouble telling rosetta model about references in rpgm like \N[idx] was place holder for a proper noun..

cant swap it sometimes because sometimes u can edit character name.. the only time it was static f it was a item/skill or place name.
It probably out of scope of the project. but it will be nice if it improve the IFEval performance

# additional prompt
system_lines.append(
        "RPGM GUIDE TRANSLATION\n\n"
        "Preserving \\N[...] codes as proper noun placeholder.\n"
        "Example:\n"
        "INPUT:「この\\N[4]の指輪は、\\N[21]で祖母が祖父に贈ったものだ。」\n"
        "OUTPUT: “This ring of \\\\N[4] was given by my grandmother to my grandfather in \\\\N[21].”\n\n"
        "Preserving \\C[...] codes as color\n"
        "Example:\n"
        "INPUT: \\\\C[4]【拓海】\\\\C[0] 「はぁ、はぁ」\n"
        "OUTPUT: \\\\C[4]Takumi\\\\C[0] Haa, haa."
    )

Rosetta is hit or miss on this..

🟦 [1/1] Translating using model: yanoljanext-rosetta-4b-2510
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
📜 Japanese: 「書嫚お久しぶり。俺、サマーキャンプで会った\N[1]。」
🧭 Context: A friendly reunion scene where the speaker recognizes someone they met before at a summer camp.
🎭 Tone: Warm and casual
📘 Glossary:
   - 書嫚 → Amanda
   - サマーキャンプ → summer camp
🗒️ Other Context (do not translate):
   - \N[1] = person name
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
"Amanda, long time no see. I'm [Person Name], from the summer camp we met at."
--
"Amanda, long time no see. I'm the one you met at summer camp back in \\N[1]."
--
"Amanda, long time no see. I'm that guy you met at summer camp."

Gemma is consistent.

🟦 [1/1] Translating using model: gemma-3n-e4b-it
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
📜 Japanese: 「書嫚お久しぶり。俺、サマーキャンプで会った\N[1]。」
🧭 Context: A friendly reunion scene where the speaker recognizes someone they met before at a summer camp.
🎭 Tone: Warm and casual
📘 Glossary:
   - 書嫚 → Amanda
   - サマーキャンプ → summer camp
🗒️ Other Context (do not translate):
   - \N[1] = person name
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ Done → "Amanda, long time no see. I'm \\N[1] we met at summer camp."...