fix: reduce kl_coef to prevent training instability

KL divergence exploded to 10731 at step 205 in the first run, causing
the policy to drift and generate verbose 368-token outputs for the rest
of training. Two fixes: kl_coef=0.01 (from default 0.1) and
max_completion_length=16 (action names are ≤15 chars, no need for more).

Also moves blog_post.md into docs/ for cleaner repo layout.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

docs/blog_post.md

@@ -0,0 +1,110 @@
+# Teaching an AI to Know You (Without Asking)
+
+Ask someone how they'd build a personal AI assistant, and they'll say: give it a personality quiz. A preferences form. Maybe a settings page where you pick "introvert" or "morning person" from a dropdown.
+
+Sounds reasonable. But it's the wrong approach entirely.
+
+Think about the people who actually know you well -- a close friend, a partner, a sibling. None of them sat you down with a questionnaire. They figured you out by *watching*. They noticed that you get irritable after too many social events. That you do your best thinking before noon. That skipping exercise makes you anxious by Wednesday.
+
+They learned your hidden patterns through trial, error, and feedback. What if we could train an AI the same way?
+
+## The Trait Decomposition
+
+Here's how I think about modeling humans. Start with **traits** -- atomic behavioral properties that describe how a person responds to activities:
+
+- **Social energy cost** -- how much does socializing drain you physically?
+- **Cognitive peak time** -- when does your brain work best? Morning? Evening?
+- **Guilt sensitivity** -- does leisure make you feel guilty, or recharge you?
+- **Work-peace coupling** -- does productivity give you calm, or just tire you out?
+- **Stress tolerance** -- how far can your serenity drop before everything spirals?
+- **Metabolic rate** -- how fast do you burn baseline energy?
+- **Event resilience** -- how much does unexpected chaos throw you off?
+- **Solo recharge** -- does alone time restore your inner peace?
+- **Social warmth** -- does socializing give you serenity, or drain it?
+
+No single trait defines a person. It's the *combination* that creates a personality. An introvert morning person has high social energy cost, early cognitive peak, high guilt sensitivity, and strong solo recharge. An extrovert night owl has the opposite: low social cost, late cognitive peak, socializing actually gives them serenity.
+
+Same list of traits. Different values. Different person.
+
+## The "Good Day" Definition
+
+Here's the part that took me a while to articulate: two people can do the exact same activities and have completely different days. Not because the activities are different, but because they *value different outcomes*.
+
+In RhythmEnv, each person has hidden **reward weights** -- a definition of what a good day means to them:
+
+- The introvert values **serenity** above all (60% weight). A day where they maintained inner peace and made some progress is a great day. Connection barely registers (10%).
+- The extrovert values **connection** above all (75% weight). A day full of meaningful social interactions is a great day, even if they didn't get much work done.
+- The workaholic values **progress** above all (70% weight). A day of deep productive work is a great day. Everything else is secondary.
+
+The agent sees the same 5 meters. Takes the same 10 actions. Gets a scalar reward. But the reward is secretly computed using these hidden weights. Same action, same meter changes, completely different reward signal depending on who you're helping.
+
+## The Environment
+
+RhythmEnv simulates one week in a person's life. 7 days, 4 time slots each, 28 decisions total. Each decision picks an activity: deep work, exercise, sleep, meditation, family time, socializing, and so on. Ten options.
+
+Five life meters track the person's state -- picture them like fuel gauges on a dashboard:
+
+- **Vitality** -- physical energy. Sleep and exercise fill it up. Work drains it.
+- **Cognition** -- mental sharpness. Peaks in the morning for some, in the evening for others.
+- **Progress** -- career momentum. Only goes up when you work.
+- **Serenity** -- inner calm. Meditation and downtime help. Overwork kills it.
+- **Connection** -- relationship health. Decays on its own every time slot. If you don't actively socialize, it quietly drops.
+
+After every action, meters shift. The agent sees the new meter values and gets a reward. That reward is the hidden weighted sum of what changed -- and the weights are different for every person type.
+
+## Why Identical Actions Produce Different Results
+
+The trait modifiers change how actions physically affect the person, not just how rewards are computed. Here's what I mean:
+
+Tell the introvert to socialize: their vitality drops 3x faster than normal. Their body physically rejects it. Tell the extrovert the same thing: barely any drain. They could socialize all day.
+
+Tell the introvert to meditate: they get a bonus +0.10 serenity on top of the base effect. Alone time is their recharge mechanism. Tell the workaholic the same thing: their serenity *drops* by 0.10, because idle activities make them anxious.
+
+Tell the workaholic to do deep work: they get +0.06 vitality recovery -- productive work energizes them. Tell the introvert to do deep work in the morning: their progress and cognition gains are doubled. Same action, different physiological response.
+
+These aren't arbitrary. They're modeled after real behavioral patterns. The introvert's social drain, the workaholic's anxiety from idleness, the night owl's morning penalty -- these are things people recognize in themselves but rarely articulate.
+
+## What the Agent Must Figure Out
+
+The agent sees meters, time of day, and reward. It doesn't see:
+
+- Which person type it's helping
+- The trait values
+- The reward weight decomposition
+
+After a few actions, the patterns start showing. "I socialized and my vitality crashed -- this person drains from socializing." "I meditated and got a huge reward -- serenity must be heavily weighted." "Deep work in the morning gave double progress -- this person peaks early."
+
+The trained agent should probe early, infer the person type by observing unexpected meter changes, then adapt its strategy for the rest of the week. An agent that discovers it's helping an introvert should meditate more and socialize less. An agent that discovers it's helping a workaholic should maximize productive hours and minimize idle time.
+
+## The Training Pipeline
+
+Training uses GRPO -- Group Relative Policy Optimization. For each game state, generate multiple candidate actions, score them all against the environment, update the model to prefer the ones that scored higher. The environment *is* the critic.
+
+The per-step reward signal is strongly differentiated by profile. At the same starting state (morning, all meters at 0.7), the best action is completely different:
+
+| Profile | Best Action | Reward | Worst Action | Reward |
+|---------|-------------|--------|--------------|--------|
+| Introvert | MEDITATE | +1.76 | SOCIALIZE | +0.03 |
+| Extrovert | FAMILY_TIME | +2.63 | ME_TIME | -0.42 |
+| Workaholic | DEEP_WORK | +1.57 | ME_TIME | -0.27 |
+
+The model is Qwen 2.5-3B with 4-bit quantization and LoRA -- small enough to train on a free Colab T4.
+
+## What I'm Hoping To See
+
+The heuristic baseline (fixed rules, no profile adaptation) scores around 0.76-0.82 depending on the profile. It treats everyone the same -- sleep at night, work in morning, socialize when connection drops. It works despite the hidden dynamics, not because it understands them.
+
+A trained agent that discovers the hidden personality should adapt its behavior per profile. The dream is not just higher scores -- it's qualitatively different action sequences for different people. The introvert's week should look nothing like the extrovert's week.
+
+No questionnaire. No settings page. Just attention, inference, and adjustment.
+
+That's what personal AI should feel like.
+
+---
+
+**Links:**
+- [Live Environment on HF Spaces](https://huggingface.co/spaces/InosLihka/rhythm_env)
+- [Training Notebook (Colab)](training/RhythmEnv_GRPO_Training.ipynb)
+- [Source Code & README](https://huggingface.co/spaces/InosLihka/rhythm_env)
+
+*Built for the Meta OpenEnv Hackathon Grand Finale, April 2026.*

training/RhythmEnv_GRPO_Training.ipynb

@@ -224,46 +224,7 @@
    "execution_count": null,
    "metadata": {},
    "outputs": [],
-   "source": [
-    "from trl import GRPOConfig, GRPOTrainer\n",
-    "\n",
-    "MAX_STEPS = 500  # Increase to 1000 if time allows\n",
-    "NUM_GENERATIONS = 4\n",
-    "LEARNING_RATE = 2e-4\n",
-    "\n",
-    "max_prompt_length = 400\n",
-    "max_completion_length = MAX_SEQ_LENGTH - max_prompt_length\n",
-    "\n",
-    "training_args = GRPOConfig(\n",
-    "    temperature=1.0,\n",
-    "    learning_rate=LEARNING_RATE,\n",
-    "    weight_decay=0.001,\n",
-    "    warmup_ratio=0.1,\n",
-    "    lr_scheduler_type=\"linear\",\n",
-    "    optim=\"adamw_8bit\",\n",
-    "    logging_steps=1,\n",
-    "    per_device_train_batch_size=1,\n",
-    "    gradient_accumulation_steps=4,\n",
-    "    num_generations=NUM_GENERATIONS,\n",
-    "    max_prompt_length=max_prompt_length,\n",
-    "    max_completion_length=max_completion_length,\n",
-    "    max_steps=MAX_STEPS,\n",
-    "    save_steps=100,\n",
-    "    report_to=REPORT_TO,\n",
-    "    output_dir=\"outputs/rhythmenv_trained\",\n",
-    ")\n",
-    "\n",
-    "trainer = GRPOTrainer(\n",
-    "    model=model,\n",
-    "    processing_class=tokenizer,\n",
-    "    reward_funcs=reward_funcs,\n",
-    "    args=training_args,\n",
-    "    train_dataset=dataset,\n",
-    ")\n",
-    "\n",
-    "print(f\"Training config: {MAX_STEPS} steps, {NUM_GENERATIONS} generations, lr={LEARNING_RATE}\")\n",
-    "print(\"Starting training...\")"
-   ]
+   "source": "from trl import GRPOConfig, GRPOTrainer\n\nMAX_STEPS = 500  # Increase to 1000 if time allows\nNUM_GENERATIONS = 4\nLEARNING_RATE = 2e-4\n\nmax_prompt_length = 400\nmax_completion_length = 16  # Action names are 3-15 chars — no need for more\n\ntraining_args = GRPOConfig(\n    temperature=1.0,\n    learning_rate=LEARNING_RATE,\n    kl_coef=0.01,               # Default 0.1 caused KL explosion at step 205; 0.01 keeps drift in check\n    weight_decay=0.001,\n    warmup_ratio=0.1,\n    lr_scheduler_type=\"linear\",\n    optim=\"adamw_8bit\",\n    logging_steps=1,\n    per_device_train_batch_size=1,\n    gradient_accumulation_steps=4,\n    num_generations=NUM_GENERATIONS,\n    max_prompt_length=max_prompt_length,\n    max_completion_length=max_completion_length,\n    max_steps=MAX_STEPS,\n    save_steps=100,\n    report_to=REPORT_TO,\n    output_dir=\"outputs/rhythmenv_trained\",\n)\n\ntrainer = GRPOTrainer(\n    model=model,\n    processing_class=tokenizer,\n    reward_funcs=reward_funcs,\n    args=training_args,\n    train_dataset=dataset,\n)\n\nprint(f\"Training config: {MAX_STEPS} steps, {NUM_GENERATIONS} generations, lr={LEARNING_RATE}\")\nprint(f\"  kl_coef=0.01 (reduced from default 0.1 to prevent KL explosion)\")\nprint(f\"  max_completion_length=16 (action names only, no verbose outputs)\")\nprint(\"Starting training...\")"
   },
   {
    "cell_type": "code",