| # ClashCR Verification Plan |
|
|
| ## Goal |
| Validate the opponent card tracker on real MuMu/BlueStacks gameplay recordings before claiming any accuracy. |
|
|
| ## Dataset Requirements |
| - At least 20 full battles. |
| - Multiple arenas/maps. |
| - Multiple resolutions (e.g., 1280x720, 1920x1080, emulator native). |
| - Single/double/triple elixir periods. |
| - Normal cards, spells, buildings, champions, evolutions, heroes, tower troops. |
| - Negative recordings: lobby/menu screens, quiet battle periods with no opponent plays. |
|
|
| ## Labeling Format |
| CSV with columns: |
| - timestamp (float, seconds) |
| - frame_idx (int) |
| - side ('opponent', 'own', 'unknown') |
| - card_key (string, normalized card name) |
| - confidence (float, 1.0 for manual labels) |
| - manual_note (string, optional) |
| - source ('manual') |
| |
| ## Metrics |
| - **Precision**: correct_predictions / total_predictions |
| - **Recall**: correct_predictions / total_labels |
| - **F1**: harmonic mean of precision and recall |
| - **False Positives per Minute**: FP / recording_duration * 60 |
| - **Missed Events**: false negatives |
| - **Mean Timing Error**: average |pred_timestamp - label_timestamp| for matched pairs |
| - **Median Timing Error**: median of above |
| - **Confusion Matrix**: per-card breakdown |
|
|
| ## Acceptance Targets |
| - 0 lobby/menu false positives. |
| - 0 random repeated spam when no card is played. |
| - False positives per minute near 0 on negative/quiet recordings. |
| - Every emitted event must include raw visual evidence. |
| - 100% accuracy may only be claimed on held-out labeled recordings if every opponent card event is detected within the allowed time window and no false events are emitted. |
| - If true 100% is not achievable from single-screen public data, state that plainly and identify exactly what additional labeled data or visual signal is required. |
|
|
| ## Commands |
| ```bash |
| # Record |
| clashcr record-battle --config config.yaml --output data/live-recordings/session-001 --seconds 180 --fps 8 |
| |
| # Label manually by editing data/live-recordings/session-001/labels.csv |
| |
| # Evaluate |
| clashcr evaluate-recording --config config.yaml --recording data/live-recordings/session-heldout --labels data/live-recordings/session-heldout/labels.csv |
| ``` |
|
|
| ## Known Limitations (Expected) |
| - Spell detection relies on heuristic color signatures; may miss subtle spells. |
| - Hero/evolution detection requires YOLO model trained on those units. |
| - RoyaleAPI static dataset is stale; official API token required for current card list. |
| - Without labeled recordings, no accuracy claims can be made. |
|
|
