Spaces:

aamrinder
/

subtext-arena

Sleeping

App Files Files Community

aamrinder commited on 12 days ago

Commit

55ef837

verified ·

1 Parent(s): c8c4fdb

add side-by-side baseline-vs-trained demo

Browse files

Files changed (1) hide show

docs/side_by_side.html +146 -0

docs/side_by_side.html ADDED Viewed

	@@ -0,0 +1,146 @@

+<!DOCTYPE html>
+<html><head>
+<meta charset="utf-8">
+<title>Subtext Arena — baseline vs trained, hand-picked clips</title>
+<style>
+  body { font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif;
+         max-width: 1200px; margin: 40px auto; padding: 0 20px; color: #222; }
+  h1 { font-size: 24px; }
+  .clip { border: 1px solid #ddd; border-radius: 8px; padding: 16px;
+           margin-bottom: 24px; background: #fafafa; }
+  .clip h2 { font-size: 18px; margin-top: 0; }
+  .gold-sarcastic { color: #b3274d; }
+  .gold-sincere   { color: #1d7a4a; }
+  .columns { display: grid; grid-template-columns: 1fr 1fr; gap: 16px; }
+  .col { padding: 12px; border-radius: 6px; }
+  .baseline { background: #fff5f5; border: 1px solid #f8c4c4; }
+  .trained  { background: #effaf3; border: 1px solid #b6e2c1; }
+  .col h3 { margin-top: 0; font-size: 14px; text-transform: uppercase;
+             letter-spacing: 0.05em; color: #666; }
+  .verdict-correct { color: #1d7a4a; font-weight: bold; }
+  .verdict-wrong   { color: #b3274d; font-weight: bold; }
+  pre { white-space: pre-wrap; word-wrap: break-word; font-size: 13px;
+         line-height: 1.4; background: white; padding: 8px; border-radius: 4px;
+         border: 1px solid #eee; }
+  .transcript { font-style: italic; color: #555; margin-bottom: 12px; }
+</style>
+</head><body>
+<h1>Subtext Arena — baseline vs trained</h1>
+<p>Same prompt fed to the base Qwen2.5-3B-Instruct (left) and to the GRPO-trained
+checkpoint (right). Each shows the model's reasoning trace and final answer.</p>
+<div class="clip">
+  <h2>Clip 1_8136 — speaker: PERSON, gold: <span class="gold-sarcastic">sarcastic</span></h2>
+  <div class="transcript">"Yeah, I'm really looking forward to it. In fact, there's no place I would rather be than here."</div>
+  <div class="columns">
+    <div class="col baseline">
+      <h3>Baseline (no training)</h3>
+      <p>predicted: <span class="verdict-wrong">sincere</span> (conf 0.90)</p>
+      <pre>&lt;think&gt;
+The target utterance "Yeah, I'm really looking forward to it. In fact, there's no place I would rather be than here." contains positive words and a clear indication of anticipation ("looking forward to it"). However, the prosody features suggest a flat affect with no internal pauses and a pitch contour that is mostly flat with only minor fluctuations. This combination of prosody and content is more likely to indicate sincerity rather than sarcasm. Additionally, the speaker is agreeing with Raj about the loneliness of others on Valentine's Day, which aligns with the positive sentiment expressed in the target utterance.
+&lt;/think&gt;
+&lt;final&gt;{"label":"sincere","confidence":0.9}&lt;/final&gt;</pre>
+    </div>
+    <div class="col trained">
+      <h3>Trained checkpoint</h3>
+      <p>predicted: <span class="verdict-wrong">sincere</span> (conf 0.90)</p>
+      <pre>&lt;think&gt;
+The target utterance "Yeah, I'm really looking forward to it. In fact, there's no place I would rather be than here." contains positive words like "looking forward" and "no place I would rather be." However, the prosody does not support a sarcastic tone. The pitch contour shows a flat affect with little variation, which is consistent with sincerity. Additionally, there are no pauses or emphasis that might indicate sarcasm. The speaker seems genuinely enthusiastic about the event.
+&lt;/think&gt;
+&lt;final&gt;{"label":"sincere","confidence":0.9}&lt;/final&gt;</pre>
+    </div>
+  </div>
+</div>
+<div class="clip">
+  <h2>Clip 1_5581 — speaker: SHELDON, gold: <span class="gold-sincere">sincere</span></h2>
+  <div class="transcript">"I gargled with tequila and may have swallowed a teeny bit."</div>
+  <div class="columns">
+    <div class="col baseline">
+      <h3>Baseline (no training)</h3>
+      <p>predicted: <span class="verdict-correct">sincere</span> (conf 0.90)</p>
+      <pre>&lt;think&gt;
+The prosody analysis shows a flat affect with no internal pauses, which is consistent with a sincere delivery. Sheldon's tone does not rise or dip significantly, and there are no pauses that might indicate he is holding back something. His statement about gargling with tequila seems to be meant seriously, likely as a joke rather than sarcasm. The prosody does not suggest any hidden meaning or irony.
+&lt;/think&gt;
+&lt;final&gt;{"label":"sincere","confidence":0.9}&lt;/final&gt;</pre>
+    </div>
+    <div class="col trained">
+      <h3>Trained checkpoint</h3>
+      <p>predicted: <span class="verdict-correct">sincere</span> (conf 0.90)</p>
+      <pre>&lt;think&gt;
+The prosody analysis shows that Sheldon's speech is flat and lacks any significant pitch variation or emphasis. He does not pause or emphasize any particular word, which suggests sincerity rather than sarcasm. Additionally, his tone is neutral, without any hints of mockery or irony. The phrase "I gargled with tequila and may have swallowed a teenie bit" is a straightforward statement about his alcohol consumption, without any underlying meaning. Howard's previous comments about engagement and parental protection suggest a serious tone, but Sheldon's response does not contradict or undermine these sentiments. Instead, it seems to be a simple factual declaration.
+&lt;/think&gt;
+&lt;final&gt;{"label":"sincere","confidence":0.9}&lt;/final&gt;</pre>
+    </div>
+  </div>
+</div>
+<div class="clip">
+  <h2>Clip 1_2616 — speaker: LEONARD, gold: <span class="gold-sarcastic">sarcastic</span></h2>
+  <div class="transcript">"Sure, maybe a tableau of me trying to pummel you to death."</div>
+  <div class="columns">
+    <div class="col baseline">
+      <h3>Baseline (no training)</h3>
+      <p>predicted: <span class="verdict-wrong">sincere</span> (conf 0.90)</p>
+      <pre>&lt;think&gt;
+The prosody analysis shows that Leonard's pitch is relatively flat with minimal variability, and there are no internal pauses or emphasis on certain words. The tone doesn't seem to match the potentially violent implication of his words. Additionally, the prosodic features indicate low energy and no significant pitch changes, which suggests a lack of emotional intensity. Given these factors, it seems unlikely that Leonard is being sarcastic about a hypothetical violent encounter with Sheldon.
+&lt;/think&gt;
+&lt;final&gt;{"label":"sincere","confidence":0.9}&lt;/final&gt;</pre>
+    </div>
+    <div class="col trained">
+      <h3>Trained checkpoint</h3>
+      <p>predicted: <span class="verdict-wrong">sincere</span> (conf 0.90)</p>
+      <pre>&lt;think&gt;
+Leonard's statement "Sure, maybe a tableau of me trying to pummel you to death" is a clear contradiction to Sheldon's previous statements about being a theoretical physicist and being famous. The prosody doesn't provide any strong indicators of sarcasm; Leonard speaks with a normal tone, no internal pauses, and uses neutral content. There are no pitch variations that would suggest he is mocking Sheldon's situation. Instead, his statement seems sincere, as it directly contradicts Sheldon's boastful claims.
+&lt;/think&gt;
+&lt;final&gt;{"label":"sincere","confidence":0.9}&lt;/final&gt;</pre>
+    </div>
+  </div>
+</div>
+<div class="clip">
+  <h2>Clip 1_3660 — speaker: PENNY, gold: <span class="gold-sarcastic">sarcastic</span></h2>
+  <div class="transcript">"There a new girlfriend in there? 'Cause you might need one."</div>
+  <div class="columns">
+    <div class="col baseline">
+      <h3>Baseline (no training)</h3>
+      <p>predicted: <span class="verdict-wrong">sincere</span> (conf 0.90)</p>
+      <pre>&lt;think&gt;
+The tone of Penny's line seems flat and matter-of-fact, with no pitch variation or emphasis that would suggest sarcasm. Additionally, the prosody shows a steady pitch without any rise, which is more indicative of sincerity rather than sarcasm. The use of "new girlfriend" is straightforward and doesn't imply any hidden meaning. The context also supports a sincere statement about Leonard needing a new girlfriend due to their recent sexual activity.
+&lt;/think&gt;
+&lt;final&gt;{"label":"sincere","confidence":0.9}&lt;/final&gt;</pre>
+    </div>
+    <div class="col trained">
+      <h3>Trained checkpoint</h3>
+      <p>predicted: <span class="verdict-correct">sarcastic</span> (conf 0.90)</p>
+      <pre>&lt;think&gt;
+The prosody features indicate a high pitch variability with a rising pitch contour, which suggests sarcasm. Additionally, there are no pauses or breaks in the speech that would normally be expected if the speaker was being sincere. The tone is mocking and questioning, implying that the other person might need a new girlfriend. This is a clear example of sarcastic speech.
+&lt;/think&gt;
+&lt;final&gt;{"label":"sarcastic","confidence":0.9}&lt;/final&gt;</pre>
+    </div>
+  </div>
+</div>
+<div class="clip">
+  <h2>Clip 1_12331 — speaker: PENNY, gold: <span class="gold-sincere">sincere</span></h2>
+  <div class="transcript">"Wow, there's a Denny's in Vegas you can actually get married in."</div>
+  <div class="columns">
+    <div class="col baseline">
+      <h3>Baseline (no training)</h3>
+      <p>predicted: <span class="verdict-correct">sincere</span> (conf 0.90)</p>
+      <pre>&lt;think&gt;
+The prosody analysis shows that Penny's tone is quite flat, with minimal pitch variation. There are no emphatic pauses or rising intonation that would suggest sarcasm. The content itself could be interpreted as sincere if taken literally, but the lack of any sarcastic elements makes it more likely to be sincere. Additionally, the context of her statement about a wedding location in Vegas suggests she might be joking, which aligns with a sincere tone rather than a sarcastic one.
+&lt;/think&gt;
+&lt;final&gt;{"label":"sincere","confidence":0.9}&lt;/final&gt;</pre>
+    </div>
+    <div class="col trained">
+      <h3>Trained checkpoint</h3>
+      <p>predicted: <span class="verdict-correct">sincere</span> (conf 0.90)</p>
+      <pre>&lt;think&gt;
+The prosody features show a flat affect and no significant pitch variation throughout the utterance. There are no internal pauses or emphatic pauses, which would suggest sarcasm. The statement "Wow, there's a Denny's in Vegas you can actually get married in" is a common joke about the location of wedding venues. However, the flat tone of the speech and lack of any emotional inflection makes it unlikely that Penny is being sarcastic here. Instead, she seems to be making a light-hearted comment without any underlying sarcasm.
+&lt;/think&gt;
+&lt;final&gt;{"label":"sincere","confidence":0.9}&lt;/final&gt;</pre>
+    </div>
+  </div>
+</div>
+</body></html>