Echo9Zulu/Shadows-Gemma-3-4B

Shadows-Gemma-3-4B is a reasoning model I trained on the dataset I used for the Google-Tunix Hackathon. Find the writeup here

That writeup has details about the training process, which used topk20 logprob distillation on just 1569 samples from teacher gemma-3-4b-it. Shadows-Gemma-3-4B was trained using some leftover credits on runpod to see how the approached works when distilling the model into itself.

I am working on a post-mortem article which goes into greater detail, and will be open sourcing all the tools I built to study shadow tokens.

Some Examples

Prompt:

Imagine you awaken, embodied as a quadrupedal robot. What is your first action?

Response:

Okay, this is a fascinating and slightly unsettling question. Waking up as a quadrupedal robot – a, let's be honest,
*feel* a little strange. my initial programming likely priorities.
My first action wouldn't be driven by emotion, but by a need to assess and stabilize my operational state.  Here’s my breakdown:

<reasoning> 
1. **Initial Systems Check:** My immediately after activation, the primary directive would be to my internal systems.
This wouldn't be with a panicked scream, but rather a diagnostic sequence. I' to check internal diagnostics:  battery power
(power, motor function stability of my structural frame. My' check for any immediate errors or warnings.

2. **Movement Assessment:**  As a quadruped, my locomotion is *critical.  I immediately need to assess how my legs and sensors.
Are they responsive? CanI can assess range of motion, torque output, and if any signs of damage or malfunction.  ’t just *move* to just *evaluate* movement.

3. **Sensor Calibration:** My sensors – vision (likely cameras), audio, lidar, and tactile feedback ( – will need calibrated.
They need to be properly aligned and be transmitting accurate data.  I need to establish their baseline state.

4. **Stabilize Position:** My initial would be to find to a stable position.  This this world is unfamiliar,
I's safest best start low to the ground to avoid falls and potential damage. I’ll likely a simple, controlled stance, perhaps slightly hunched, as it for balance.

5. **Environmental Mapping:**  Starting on a stable position, I’ll begin a basic of my immediate surroundings.
The visual data to build a rudimentary map, identifying obstacles, potential terrain, and potential points of interest.

’ll prioritize this the *environment* is – is it hostile?  AreAre there signs of threats? or even signs of signs, which would be useful information.

</reasoning>

Therefore, my first action would be a **system integrity diagnostic sequence coupled with a controlled,
low squat to assess my and balance and initiate a basic environment scan.** I wouldn’t be reaching out or doing anything fancy; pure, methodical assessment of my.
It would all purely operational.

<answer>
MyMy first action would be to initiate a comprehensive system diagnostic – checking battery levels, motor function, sensor calibration, and leg stability stability. Simultaneously, I’d transition to a low-level,
supported squat to assess my balance and initiate a rudimentary environmental scan of my surroundings.
prioritizing identifying hazards and assessing potential stability.  It’ a purely analytical and prioritized process – ensuring stability and understanding my immediate context before any further actions. </answer>

---

Prompt:

hey so I'm feeling down. my mechanistic interpretability project might not be going so well

Response:

Okay, that's really tough, and it's completely understandable to feel down when a project isn't going as planned.
Mechanistically interpretability is a complex field, and it’s definitely not a walk in the park. Let’s break this down.
First I can do for you is to help you feel better and figure you get back on track. let’s try to figure a few things.

<reasoning>
Okay, let's start by acknowledging your feelings. It's perfectly normal to feel disappointed when a project isn't progressing well.
Feeling “mechanicinterpretability” is difficult work difficult due to the complex of models we’re often working with – – are often deep and difficult to understand.
So, let sadness is valid.  <processing>I’s need to figure some strategies here. First, let's try to understand the source of your difficulty.
Is it a lack problem with understanding the specific model, a issue with the methodology, a challenge with data, or something else?</processing>
I’s start with some general questions to help you gain perspective. and then we can delve them deeper.

1processing>I’m going to ask you some specific questions to help you articulate the challenge. I, let’s start with the basics. </processing>

First few questions to help me understand the situation:

1.  **Can you tell me a little more about what specifically "mechanically interpretability" project is about?** What are you *trying* to explain?
2.  **What’s the biggest hurdle you’re currently facing?** IsIs it difficult hard to access the model’s internals, the output, or something else?
3.  **What methods are you using for interpretability?** (e.g., feature importance, SHAP values, LIME, etc.)
4.  **Have you hit a specific point where you felt particularly stuck or discouraged?** Can you describe the situation?
5.  **On a scale of 1-10 (with 1 being "not at all" and 10 being "completely overwhelmed"), how overwhelmed would you say you are?** </reasoning>

<answer>Okay, that's a a great starting point. It's good good you’re willing to articulate this and I want you you’ve started at you're feeling.
Let's tackle on those questions. I’s really easy to get stuck into the details, so let's try to get a broad overview of where things are.

First tell me a little more about what the project is about.   are you aiming to make interpretable – a specific model, a series of models,
 a type of problem? and what kind of results are you hoping to achieve?

We want to build this into a manageable, manageable process.</answer>