initial Space

.gitignore

@@ -0,0 +1,18 @@
+# Private answer key — pulled at boot from the private dataset, never committed
+annotations_private.json
+
+# Paper-baseline seed records — destined for the public DATASET repo, not the Space
+seeds/
+
+# Local dev
+.venv/
+__pycache__/
+*.pyc
+
+# Local snapshot caches
+.cache/
+
+# Editor / OS
+.DS_Store
+.idea/
+.vscode/

README.md

@@ -0,0 +1,76 @@
+---
+title: EgoMemReason Leaderboard
+emoji: 🧠
+colorFrom: indigo
+colorTo: purple
+sdk: gradio
+sdk_version: 4.44.0
+app_file: app.py
+pinned: false
+license: cc-by-nc-4.0
+hf_oauth: true
+hf_oauth_scopes:
+  - openid
+  - profile
+---
+
+# EgoMemReason — Leaderboard Space
+
+Live leaderboard for the **EgoMemReason** benchmark: 500 multiple-choice questions over week-long egocentric video, evaluating entity / event / behavior memory.
+
+- 📄 Paper: *coming soon*
+- 💻 Reference eval scripts: <https://github.com/Ted412/EgoMemReason>
+- 📦 Public questions: <https://huggingface.co/datasets/Ted412/EgoMemReason>
+- 🎬 Source frames: <https://egolife-ai.github.io/>
+
+## Operator notes
+
+This Space lives at `Ted412/EgoMemReason` and writes one JSON record per submission to the public dataset `Ted412/EgoMemReason-Leaderboard`. The held-out answer key lives in a separate **private** dataset `Ted412/EgoMemReason-Private` and is pulled at boot via `snapshot_download(token=HF_TOKEN)`.
+
+### Required Space secret
+
+| Name | Value | Scope |
+|---|---|---|
+| `HF_TOKEN` | Fine-grained HF token | Write on `Ted412/EgoMemReason-Leaderboard` + Read on `Ted412/EgoMemReason-Private` |
+
+Create at <https://huggingface.co/settings/tokens> → fine-grained → grant only those two repos.
+
+### Local development
+
+```bash
+python -m venv .venv && source .venv/bin/activate
+pip install -r requirements.txt
+
+# Copy the private answer key into cwd (skips the snapshot_download path).
+cp ../EgoMemReason-EvalAI.archived/annotations/annotations_private.json .
+
+# Run, optionally faking a user.
+DEBUG_USER=alice python app.py
+# → http://127.0.0.1:7860
+```
+
+Tests:
+
+```bash
+python -m pytest tests/ -q
+```
+
+### Architecture
+
+```
+EgoMemReason-Space (this Space, public)
+├── app.py            Gradio UI (Leaderboard / Submit / Manage / About)
+├── evaluator.py      pure scoring — port of the old EvalAI main.py
+├── ledger.py         HF I/O: pulls private annotations at boot; writes
+│                     one JSON record per submission to the public dataset
+├── auth.py           resolves the HF username from gr.OAuthProfile
+└── annotations_private.json   pulled at boot from the private dataset
+
+Ted412/EgoMemReason-Private (HF dataset, private)
+└── annotations_private.json   500 Qs WITH correct_answer
+
+Ted412/EgoMemReason-Leaderboard (HF dataset, public)
+└── submissions/
+    └── <uuid>.json   one immutable record per submission
+                      (only is_selected flips on a re-write)
+```

SUBMISSION_FORMAT.md

@@ -0,0 +1,77 @@
+# Submission Format
+
+A submission is a single JSON file (`.json`) containing a top-level array of 500 prediction objects — one per question.
+
+## Schema
+
+```json
+[
+  {"example_id": 1,   "predicted_answer": "A"},
+  {"example_id": 2,   "predicted_answer": "C"},
+  {"example_id": 500, "predicted_answer": "B"}
+]
+```
+
+**Required keys (per object):**
+- `example_id` — integer in `[1, 500]`, matching `example_id` in `annotations_public.json`.
+- `predicted_answer` — single uppercase letter that appears in that question's `options` dict.
+
+**Important:** questions have **between 4 and 10 options**. The valid answer letters for any given question are exactly the keys of its `options` dict. Most are A-F; Event Ordering questions can extend to A-J. A letter outside the question's option set is rejected.
+
+**Optional keys (ignored, but useful for your own debugging):** `raw_response`, `confidence`, `tokens`, etc.
+
+## Rules
+
+1. Top-level must be a JSON array (not an object).
+2. The submission must cover **exactly 500 unique `example_id`s**, one per question.
+3. Duplicate `example_id`s are rejected.
+4. Letters must be uppercase (whitespace is trimmed).
+5. File extension must be `.json`.
+
+## Converting from existing eval-script output
+
+The reference inference scripts in the [EgoMemReason GitHub repo](https://github.com/Ted412/EgoMemReason) write a list of records with a `pred` field. One-liner to convert:
+
+```python
+import json
+src = json.load(open("results_my_model.json"))
+sub = [{"example_id": r["example_id"], "predicted_answer": r["pred"]} for r in src]
+json.dump(sub, open("submission.json", "w"))
+```
+
+## How submissions are scored
+
+Accuracy (%) for each of the six `query_type` splits:
+
+- Cumulative State Tracking (100 Qs)
+- Temporal Counting (100 Qs)
+- Event Ordering (100 Qs)
+- Event Linking (100 Qs)
+- Spatial Preference (50 Qs)
+- Activity Pattern (50 Qs)
+
+plus **Overall** accuracy on all 500. All seven values appear on the leaderboard; ranking is by Overall descending.
+
+## Submission limits
+
+- **5 submissions per HF user per 24-hour window.**
+- The 24-hour window is rolling, not midnight-aligned.
+
+## Selected submission
+
+Submit as many times as you like under the cap. In the **Manage my submissions** tab you can mark **one** of your past submissions as your *selected* entry. The default leaderboard view shows only each team's selected entry; the "Show all submissions" toggle reveals all.
+
+## Required metadata fields
+
+When you submit you must fill in:
+
+| Field | Required | Notes |
+|---|---|---|
+| `team_name` | yes | Team or affiliation |
+| `method_name` | yes | Short title displayed on the leaderboard |
+| `uses_external_data` | yes (yes/no) | Did you train / finetune on anything beyond EgoLife? |
+| `uses_video_frames` | yes | one of `frames-only` · `video-only` · `frames+audio` · `captions-only` · `other` |
+| `model_size` | no | e.g. `8B`, `32B`, `API` |
+| `method_description` | no | Free-form description |
+| `project_url` | no | Project page |
+| `publication_url` | no | arXiv / OpenReview link |

app.py

@@ -0,0 +1,331 @@
+"""EgoMemReason leaderboard — Gradio Space app.
+
+Tabs:
+  - Leaderboard   public, auto-refresh, toggle selected-only / show-all
+  - Submit        HF login required; JSON upload + metadata form
+  - Manage        toggle is_selected on your own past submissions
+  - About         paper description + citation
+"""
+
+import os
+
+import gradio as gr
+import pandas as pd
+
+import auth
+import evaluator
+import ledger
+
+
+# ---------------------------------------------------------------------------
+# Boot: pull annotations_private.json from the private dataset repo.
+# ---------------------------------------------------------------------------
+
+try:
+    ledger.ensure_private_annotations()
+except RuntimeError as e:
+    # In local dev without HF_TOKEN, allow the app to come up with a clear banner.
+    BOOT_ERROR = str(e)
+else:
+    BOOT_ERROR = None
+
+
+LEADERBOARD_COLUMNS = [
+    "Rank",
+    "Method",
+    "Team",
+    "Overall",
+    "Cumul. State",
+    "Temp. Count",
+    "Event Order",
+    "Event Link",
+    "Spatial Pref.",
+    "Activity Pat.",
+    "Model size",
+    "Ext. data",
+    "Modality",
+    "Date (UTC)",
+    "Links",
+]
+
+
+def _row_from_submission(sub, rank):
+    m = sub["metrics"]
+    links = []
+    if sub.get("project_url"):
+        links.append(f"[project]({sub['project_url']})")
+    if sub.get("publication_url"):
+        links.append(f"[paper]({sub['publication_url']})")
+    return [
+        rank,
+        sub["method_name"],
+        sub["team_name"],
+        m["Overall"],
+        m["Cumulative State Tracking"],
+        m["Temporal Counting"],
+        m["Event Ordering"],
+        m["Event Linking"],
+        m["Spatial Preference"],
+        m["Activity Pattern"],
+        sub.get("model_size") or "—",
+        "yes" if sub.get("uses_external_data") else "no",
+        sub.get("uses_video_frames") or "—",
+        sub.get("submitted_at_utc", "")[:10],
+        " · ".join(links) or "—",
+    ]
+
+
+def load_leaderboard(show_all):
+    subs = ledger.list_submissions()
+    if not show_all:
+        subs = [s for s in subs if s.get("is_selected")]
+    subs = sorted(subs, key=lambda s: s["metrics"]["Overall"], reverse=True)
+    rows = [_row_from_submission(s, i + 1) for i, s in enumerate(subs)]
+    return pd.DataFrame(rows, columns=LEADERBOARD_COLUMNS)
+
+
+# ---------------------------------------------------------------------------
+# Submit
+# ---------------------------------------------------------------------------
+
+def handle_submission(file, team_name, method_name, model_size, uses_external,
+                      uses_frames, method_description, project_url,
+                      publication_url, profile: gr.OAuthProfile | None):
+    user = auth.resolve_user(profile)
+    if user is None:
+        return "**Error:** sign in with Hugging Face first (button at the top of the page)."
+    if not team_name or not method_name:
+        return "**Error:** `team_name` and `method_name` are required."
+    if uses_external not in ("yes", "no"):
+        return "**Error:** answer `Uses external data?` (yes/no)."
+    if not uses_frames:
+        return "**Error:** pick a video input modality."
+    if file is None:
+        return "**Error:** upload a `.json` submission file."
+
+    recent = ledger.count_recent(user, hours=24)
+    if recent >= 5:
+        return (f"**Rate limit:** you have **{recent}** submissions in the last 24 h "
+                "(max 5). Try again later.")
+
+    try:
+        metrics = evaluator.score_submission(file.name)
+    except ValueError as e:
+        return f"**Validation error:**\n```\n{e}\n```"
+    except Exception as e:
+        return f"**Internal error scoring submission:** `{type(e).__name__}: {e}`"
+
+    try:
+        sid = ledger.append_submission(
+            hf_user_id=user,
+            team_name=team_name,
+            method_name=method_name,
+            model_size=model_size,
+            uses_external_data=(uses_external == "yes"),
+            uses_video_frames=uses_frames,
+            method_description=method_description,
+            project_url=project_url,
+            publication_url=publication_url,
+            metrics=metrics,
+        )
+    except Exception as e:
+        return (f"**Scored, but failed to persist to ledger:** `{type(e).__name__}: {e}`\n\n"
+                f"Your metrics were:\n```\n{metrics}\n```")
+
+    rows = "\n".join(f"| {k} | **{v:.2f}** |" for k, v in metrics.items())
+    return (
+        f"✅ **Submission logged.** `submission_id = {sid}`\n\n"
+        f"| Metric | Score (%) |\n|---|---|\n{rows}\n\n"
+        "Go to **Manage my submissions** to mark this as your official entry."
+    )
+
+
+# ---------------------------------------------------------------------------
+# Manage
+# ---------------------------------------------------------------------------
+
+MANAGE_COLUMNS = ["submission_id", "method_name", "Overall", "is_selected", "submitted_at_utc"]
+
+
+def load_my_submissions(profile: gr.OAuthProfile | None):
+    user = auth.resolve_user(profile)
+    if user is None:
+        return pd.DataFrame(columns=MANAGE_COLUMNS)
+    rows = []
+    for sub in ledger.list_submissions():
+        if sub.get("hf_user_id") != user:
+            continue
+        rows.append([
+            sub["submission_id"],
+            sub["method_name"],
+            sub["metrics"]["Overall"],
+            sub.get("is_selected", False),
+            sub.get("submitted_at_utc", ""),
+        ])
+    rows.sort(key=lambda r: r[4], reverse=True)
+    return pd.DataFrame(rows, columns=MANAGE_COLUMNS)
+
+
+def set_my_selected(submission_id, profile: gr.OAuthProfile | None):
+    user = auth.resolve_user(profile)
+    if user is None:
+        return "**Error:** sign in first.", load_my_submissions(profile)
+    if not submission_id or not submission_id.strip():
+        return "**Error:** paste a submission_id.", load_my_submissions(profile)
+    try:
+        ledger.set_selected(submission_id.strip(), user)
+    except (ValueError, PermissionError) as e:
+        return f"**Error:** {e}", load_my_submissions(profile)
+    return f"✅ `{submission_id.strip()}` is now your selected entry.", load_my_submissions(profile)
+
+
+# ---------------------------------------------------------------------------
+# About
+# ---------------------------------------------------------------------------
+
+ABOUT_MD = """\
+## EgoMemReason
+
+**A Memory-driven Reasoning Benchmark for Long-Horizon Egocentric Video Understanding.**
+
+EgoMemReason is a 500-question multiple-choice benchmark over week-long egocentric
+videos (built on [EgoLife](https://egolife-ai.github.io/)). Models must answer
+questions whose evidence is sparsely distributed across hours or days, exercising
+three memory types:
+
+- **Entity memory** — Cumulative State Tracking, Temporal Counting
+- **Event memory** — Event Ordering, Event Linking
+- **Behavior memory** — Spatial Preference Inference, Activity Pattern Inference
+
+500 Qs · avg. 5.1 evidence segments / Q · avg. 25.9 h memory backtracking. The
+strongest model in the paper reaches **39.6% Overall**.
+
+### Resources
+
+- 📄 Paper: *coming soon*
+- 💻 Code & reference eval scripts: <https://github.com/Ted412/EgoMemReason>
+- 📦 Public questions (no answers): <https://huggingface.co/datasets/Ted412/EgoMemReason>
+- 🎬 EgoLife video frames: <https://egolife-ai.github.io/>
+
+### Submission
+
+Upload a JSON file with 500 entries:
+
+```json
+[
+  {"example_id": 1, "predicted_answer": "A"},
+  ...
+]
+```
+
+Questions have 4-10 options (letters A-J) — `predicted_answer` must be a letter
+that appears in that question's `options` dict. See
+[SUBMISSION_FORMAT.md](https://github.com/Ted412/EgoMemReason/blob/main/SUBMISSION_FORMAT.md)
+for the full spec.
+
+### License
+
+- **Annotations** (this Space + the public dataset): CC BY-NC 4.0.
+- **Video frames**: governed by the [EgoLife data license](https://egolife-ai.github.io/) — you must accept their terms separately.
+
+### Citation
+
+```bibtex
+@article{wang2026egomemreason,
+  title   = {EgoMemReason: A Memory-driven Reasoning Benchmark for
+             Long-Horizon Egocentric Video Understanding},
+  author  = {Wang, Ziyang and Zhang, Yue and Yu, Shoubin and Zhang, Ce and
+             Zhao, Zengqi and Yoon, Jaehong and Lee, Hyunji and
+             Bertasius, Gedas and Bansal, Mohit},
+  year    = {2026},
+  journal = {arXiv preprint}
+}
+```
+"""
+
+
+# ---------------------------------------------------------------------------
+# UI
+# ---------------------------------------------------------------------------
+
+with gr.Blocks(title="EgoMemReason Leaderboard", theme=gr.themes.Soft()) as demo:
+    gr.Markdown("# 🧠 EgoMemReason — Leaderboard")
+    gr.Markdown(
+        "*Memory-driven reasoning over week-long egocentric video. 500 MCQs · "
+        "Entity / Event / Behavior memory.*"
+    )
+    if BOOT_ERROR:
+        gr.Markdown(f"⚠️ **Boot warning:** {BOOT_ERROR}\n\nSubmissions are disabled.")
+
+    login_btn = gr.LoginButton()
+
+    with gr.Tab("Leaderboard"):
+        with gr.Row():
+            show_all = gr.Checkbox(
+                value=False,
+                label="Show all submissions (not just each team's selected entry)",
+            )
+            refresh_btn = gr.Button("Refresh", size="sm")
+        leaderboard_df = gr.Dataframe(
+            value=load_leaderboard(False),
+            headers=LEADERBOARD_COLUMNS,
+            interactive=False,
+            wrap=True,
+        )
+        show_all.change(load_leaderboard, inputs=[show_all], outputs=[leaderboard_df])
+        refresh_btn.click(load_leaderboard, inputs=[show_all], outputs=[leaderboard_df])
+
+    with gr.Tab("Submit"):
+        gr.Markdown("**Sign in with Hugging Face (button above) before submitting.** "
+                    "Limit: 5 submissions per HF user per 24 h.")
+        with gr.Row():
+            team_name = gr.Textbox(label="Team name *", max_lines=1)
+            method_name = gr.Textbox(label="Method name *", max_lines=1)
+        with gr.Row():
+            model_size = gr.Textbox(label="Model size (e.g. 8B, 32B, API)", max_lines=1)
+            uses_external = gr.Radio(
+                ["yes", "no"], label="Uses training data beyond EgoLife? *",
+            )
+        uses_frames = gr.Radio(
+            ["frames-only", "video-only", "frames+audio", "captions-only", "other"],
+            label="Video input modality *",
+        )
+        method_description = gr.Textbox(label="Method description", lines=3)
+        with gr.Row():
+            project_url = gr.Textbox(label="Project URL", max_lines=1)
+            publication_url = gr.Textbox(label="Publication URL (arXiv/OpenReview)", max_lines=1)
+        submission_file = gr.File(label="submission.json", file_types=[".json"])
+        submit_btn = gr.Button("Score & log", variant="primary")
+        result_md = gr.Markdown()
+        submit_btn.click(
+            handle_submission,
+            inputs=[submission_file, team_name, method_name, model_size,
+                    uses_external, uses_frames, method_description,
+                    project_url, publication_url],
+            outputs=[result_md],
+        )
+
+    with gr.Tab("Manage my submissions"):
+        gr.Markdown(
+            "Toggle which of your past submissions is the official **selected** entry. "
+            "Only your own submissions appear here. "
+            "Only one entry per HF user can be selected at a time."
+        )
+        my_subs = gr.Dataframe(
+            value=pd.DataFrame(columns=MANAGE_COLUMNS),
+            headers=MANAGE_COLUMNS,
+            interactive=False,
+            wrap=True,
+        )
+        selected_id = gr.Textbox(label="submission_id to mark as selected", max_lines=1)
+        select_btn = gr.Button("Mark as my selected entry")
+        manage_msg = gr.Markdown()
+        demo.load(load_my_submissions, outputs=[my_subs])
+        select_btn.click(set_my_selected, inputs=[selected_id], outputs=[manage_msg, my_subs])
+
+    with gr.Tab("About"):
+        gr.Markdown(ABOUT_MD)
+
+
+if __name__ == "__main__":
+    demo.queue().launch()

auth.py

@@ -0,0 +1,28 @@
+"""Resolve the current HF user from Gradio's OAuthProfile.
+
+`gr.LoginButton()` populates `gr.OAuthProfile` for every callback that declares
+it as a parameter. We add a `DEBUG_USER` escape hatch for local development,
+gated on the SPACE_ID env var so it can never fire in production.
+"""
+
+import os
+
+
+def is_production():
+    """True when running inside the HF Space sandbox (vs local dev)."""
+    return os.environ.get("SPACE_ID") is not None
+
+
+def resolve_user(profile):
+    """Returns the HF username of the requesting user, or None if not logged in.
+
+    `profile` is the `gr.OAuthProfile | None` Gradio passes to callbacks that
+    declare it. In local dev, set DEBUG_USER=alice to pretend to be `alice`.
+    """
+    if not is_production():
+        debug = os.environ.get("DEBUG_USER")
+        if debug:
+            return debug
+    if profile is None:
+        return None
+    return profile.username

evaluator.py

@@ -0,0 +1,121 @@
+"""Scoring logic for EgoMemReason.
+
+Pure stdlib — no Gradio, no HF imports. Returns a flat metrics dict.
+Raises ValueError with per-example messages on validation failure.
+"""
+
+import json
+from collections import defaultdict
+
+# Order matches the leaderboard column order.
+QUERY_TYPES = [
+    "Cumulative State Tracking",
+    "Temporal Counting",
+    "Event Ordering",
+    "Event Linking",
+    "Spatial Preference",
+    "Activity Pattern",
+]
+
+
+def _load(path):
+    with open(path) as f:
+        return json.load(f)
+
+
+def _build_gt(ann):
+    """Returns {example_id: (correct_letter, query_type, valid_option_letters)}.
+
+    Questions have 4-10 options (letters up to J), so the valid answer set
+    is per-question, not a fixed A-D.
+    """
+    samples = ann["samples"] if isinstance(ann, dict) else ann
+    gt = {}
+    for s in samples:
+        eid = s["example_id"]
+        opts = {str(k).strip().upper() for k in s["options"].keys()}
+        gt[eid] = (s["correct_answer"].strip().upper(), s["query_type"], opts)
+    return gt
+
+
+def _validate(preds, gt):
+    if not isinstance(preds, list):
+        raise ValueError("Submission must be a JSON list of objects.")
+
+    errors = []
+    seen = set()
+    for i, item in enumerate(preds):
+        if not isinstance(item, dict):
+            errors.append(f"item {i}: not a JSON object")
+            continue
+        eid = item.get("example_id")
+        ans = item.get("predicted_answer")
+        if not isinstance(eid, int):
+            errors.append(f"item {i}: 'example_id' must be an int, got {type(eid).__name__}")
+            continue
+        if eid in seen:
+            errors.append(f"duplicate example_id: {eid}")
+        seen.add(eid)
+        if eid not in gt:
+            errors.append(f"unknown example_id: {eid}")
+            continue
+        valid = gt[eid][2]
+        if not isinstance(ans, str) or ans.strip().upper() not in valid:
+            errors.append(
+                f"example_id {eid}: 'predicted_answer' must be one of "
+                f"{'/'.join(sorted(valid))}, got {ans!r}"
+            )
+
+    missing = set(gt) - seen
+    if missing:
+        errors.append(
+            f"missing {len(missing)} example_ids (e.g. {sorted(missing)[:5]}); "
+            f"submission must cover all {len(gt)} questions"
+        )
+
+    if errors:
+        msg = "Submission validation failed:\n  - " + "\n  - ".join(errors[:20])
+        if len(errors) > 20:
+            msg += f"\n  - ... and {len(errors) - 20} more error(s)"
+        raise ValueError(msg)
+
+
+def _score(preds, gt):
+    correct_total = 0
+    count_by_qt = defaultdict(int)
+    correct_by_qt = defaultdict(int)
+
+    for _eid, (_gt_ans, qt, _opts) in gt.items():
+        count_by_qt[qt] += 1
+
+    for item in preds:
+        eid = item["example_id"]
+        ans = item["predicted_answer"].strip().upper()
+        gt_ans, qt, _opts = gt[eid]
+        if ans == gt_ans:
+            correct_total += 1
+            correct_by_qt[qt] += 1
+
+    metrics = {}
+    for qt in QUERY_TYPES:
+        n = count_by_qt.get(qt, 0)
+        metrics[qt] = round(100.0 * correct_by_qt[qt] / n, 2) if n else 0.0
+    metrics["Overall"] = round(100.0 * correct_total / len(gt), 2)
+    return metrics
+
+
+def score_submission(submission_path, annotation_path="annotations_private.json"):
+    """Returns {"Cumulative State Tracking": ..., ..., "Overall": ...} as percentages."""
+    gt = _build_gt(_load(annotation_path))
+    preds = _load(submission_path)
+    _validate(preds, gt)
+    return _score(preds, gt)
+
+
+if __name__ == "__main__":
+    import argparse, pprint
+    p = argparse.ArgumentParser()
+    p.add_argument("--annotation", default="annotations_private.json")
+    p.add_argument("--submission", required=True)
+    args = p.parse_args()
+    pprint.pp(score_submission(args.submission, args.annotation))

ledger.py

@@ -0,0 +1,180 @@
+"""HF I/O for the EgoMemReason leaderboard.
+
+Two repos:
+  - PUBLIC_DATASET  Ted412/EgoMemReason-Leaderboard  (one JSON per submission)
+  - PRIVATE_DATASET Ted412/EgoMemReason-Private     (annotations_private.json)
+
+Boot path: ensure_private_annotations() downloads the private annotations file
+on app start so evaluator.score_submission() can read it from cwd.
+"""
+
+import functools
+import io
+import json
+import os
+import time
+import uuid
+from datetime import datetime, timedelta, timezone
+
+from huggingface_hub import HfApi, snapshot_download
+
+# Hard-coded for this challenge. Override via env vars in dev.
+PUBLIC_DATASET = os.environ.get("EGOMEM_PUBLIC_DATASET", "Ted412/EgoMemReason-Leaderboard")
+PRIVATE_DATASET = os.environ.get("EGOMEM_PRIVATE_DATASET", "Ted412/EgoMemReason-Private")
+ANNOTATIONS_FILENAME = "annotations_private.json"
+
+HF_TOKEN = os.environ.get("HF_TOKEN")  # write scope on PUBLIC_DATASET; read scope on PRIVATE_DATASET
+
+
+def ensure_private_annotations(dest_path=ANNOTATIONS_FILENAME):
+    """Download annotations_private.json from the private dataset on app boot.
+
+    Only called once per Space restart. If the file is already present (local
+    dev case where you've copied it manually), do nothing.
+    """
+    if os.path.exists(dest_path):
+        return dest_path
+    if not HF_TOKEN:
+        raise RuntimeError(
+            "HF_TOKEN env var not set; cannot pull private annotations from "
+            f"{PRIVATE_DATASET}. Either set HF_TOKEN or place {dest_path} in cwd."
+        )
+    local_dir = snapshot_download(
+        repo_id=PRIVATE_DATASET,
+        repo_type="dataset",
+        token=HF_TOKEN,
+        allow_patterns=[ANNOTATIONS_FILENAME],
+    )
+    src = os.path.join(local_dir, ANNOTATIONS_FILENAME)
+    if not os.path.exists(src):
+        raise RuntimeError(
+            f"{ANNOTATIONS_FILENAME} not found in {PRIVATE_DATASET}. "
+            "Upload it via the HF Files UI of the private dataset repo."
+        )
+    # Symlink rather than copy — snapshot_download already cached it.
+    if not os.path.exists(dest_path):
+        os.symlink(src, dest_path)
+    return dest_path
+
+
+def _now_iso():
+    return datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")
+
+
+@functools.lru_cache(maxsize=1)
+def _cached_submissions(cache_bucket):
+    """Pulls all submission JSON files. Bucket is int(time/60) so cache rolls every minute."""
+    del cache_bucket  # only here to invalidate the cache
+    try:
+        local_dir = snapshot_download(
+            repo_id=PUBLIC_DATASET,
+            repo_type="dataset",
+            token=HF_TOKEN,  # not strictly required for public read but avoids rate-limiting
+            allow_patterns=["submissions/*.json"],
+        )
+    except Exception:
+        return []
+    folder = os.path.join(local_dir, "submissions")
+    if not os.path.isdir(folder):
+        return []
+    out = []
+    for fn in os.listdir(folder):
+        if not fn.endswith(".json"):
+            continue
+        try:
+            with open(os.path.join(folder, fn)) as f:
+                out.append(json.load(f))
+        except Exception:
+            continue
+    return out
+
+
+def list_submissions():
+    return _cached_submissions(int(time.time() / 60))
+
+
+def _invalidate_cache():
+    _cached_submissions.cache_clear()
+
+
+def count_recent(hf_user_id, hours=24):
+    cutoff = datetime.now(timezone.utc) - timedelta(hours=hours)
+    n = 0
+    for sub in list_submissions():
+        if sub.get("hf_user_id") != hf_user_id:
+            continue
+        ts = sub.get("submitted_at_utc", "")
+        try:
+            t = datetime.fromisoformat(ts.rstrip("Z")).replace(tzinfo=timezone.utc)
+        except ValueError:
+            continue
+        if t >= cutoff:
+            n += 1
+    return n
+
+
+def _upload_record(record):
+    payload = json.dumps(record, indent=2).encode("utf-8")
+    HfApi().upload_file(
+        path_or_fileobj=io.BytesIO(payload),
+        path_in_repo=f"submissions/{record['submission_id']}.json",
+        repo_id=PUBLIC_DATASET,
+        repo_type="dataset",
+        token=HF_TOKEN,
+        commit_message=f"submission {record['submission_id'][:8]} from {record['hf_user_id']}",
+    )
+
+
+def append_submission(*, hf_user_id, team_name, method_name, model_size,
+                      uses_external_data, uses_video_frames, method_description,
+                      project_url, publication_url, metrics):
+    if not HF_TOKEN:
+        raise RuntimeError("HF_TOKEN not set; cannot persist submission.")
+    sid = str(uuid.uuid4())
+    record = {
+        "submission_id": sid,
+        "submitted_at_utc": _now_iso(),
+        "hf_user_id": hf_user_id,
+        "team_name": team_name,
+        "method_name": method_name,
+        "model_size": model_size or "",
+        "uses_external_data": bool(uses_external_data),
+        "uses_video_frames": uses_video_frames,
+        "method_description": method_description or "",
+        "project_url": project_url or "",
+        "publication_url": publication_url or "",
+        "is_selected": False,
+        "metrics": metrics,
+    }
+    _upload_record(record)
+    _invalidate_cache()
+    return sid
+
+
+def set_selected(submission_id, requesting_user):
+    """Mark `submission_id` as the requesting_user's selected entry.
+
+    Enforces one-selected-per-user. Raises PermissionError if the submission
+    does not belong to requesting_user.
+    """
+    target = None
+    for sub in list_submissions():
+        if sub["submission_id"] == submission_id:
+            target = sub
+            break
+    if target is None:
+        raise ValueError(f"submission_id not found: {submission_id}")
+    if target["hf_user_id"] != requesting_user:
+        raise PermissionError("You can only modify your own submissions.")
+
+    # Un-select any other submission this user previously selected.
+    for sub in list_submissions():
+        if (sub["hf_user_id"] == requesting_user
+                and sub["is_selected"]
+                and sub["submission_id"] != submission_id):
+            sub["is_selected"] = False
+            _upload_record(sub)
+
+    target["is_selected"] = True
+    _upload_record(target)
+    _invalidate_cache()

requirements.txt

@@ -0,0 +1,3 @@
+gradio==4.44.0
+huggingface_hub==0.25.0
+pandas==2.2.3

tests/fixtures/all_a_submission.json

@@ -0,0 +1 @@
+[{"example_id": 1, "predicted_answer": "A"}, {"example_id": 2, "predicted_answer": "A"}, {"example_id": 3, "predicted_answer": "A"}, {"example_id": 4, "predicted_answer": "A"}, {"example_id": 5, "predicted_answer": "A"}, {"example_id": 6, "predicted_answer": "A"}, {"example_id": 7, "predicted_answer": "A"}, {"example_id": 8, "predicted_answer": "A"}, {"example_id": 9, "predicted_answer": "A"}, {"example_id": 10, "predicted_answer": "A"}, {"example_id": 11, "predicted_answer": "A"}, {"example_id": 12, "predicted_answer": "A"}, {"example_id": 13, "predicted_answer": "A"}, {"example_id": 14, "predicted_answer": "A"}, {"example_id": 15, "predicted_answer": "A"}, {"example_id": 16, "predicted_answer": "A"}, {"example_id": 17, "predicted_answer": "A"}, {"example_id": 18, "predicted_answer": "A"}, {"example_id": 19, "predicted_answer": "A"}, {"example_id": 20, "predicted_answer": "A"}, {"example_id": 21, "predicted_answer": "A"}, {"example_id": 22, "predicted_answer": "A"}, {"example_id": 23, "predicted_answer": "A"}, {"example_id": 24, "predicted_answer": "A"}, {"example_id": 25, "predicted_answer": "A"}, {"example_id": 26, "predicted_answer": "A"}, {"example_id": 27, "predicted_answer": "A"}, {"example_id": 28, "predicted_answer": "A"}, {"example_id": 29, "predicted_answer": "A"}, {"example_id": 30, "predicted_answer": "A"}, {"example_id": 31, "predicted_answer": "A"}, {"example_id": 32, "predicted_answer": "A"}, {"example_id": 33, "predicted_answer": "A"}, {"example_id": 34, "predicted_answer": "A"}, {"example_id": 35, "predicted_answer": "A"}, {"example_id": 36, "predicted_answer": "A"}, {"example_id": 37, "predicted_answer": "A"}, {"example_id": 38, "predicted_answer": "A"}, {"example_id": 39, "predicted_answer": "A"}, {"example_id": 40, "predicted_answer": "A"}, {"example_id": 41, "predicted_answer": "A"}, {"example_id": 42, "predicted_answer": "A"}, {"example_id": 43, "predicted_answer": "A"}, {"example_id": 44, "predicted_answer": "A"}, {"example_id": 45, "predicted_answer": "A"}, {"example_id": 46, "predicted_answer": "A"}, {"example_id": 47, "predicted_answer": "A"}, {"example_id": 48, "predicted_answer": "A"}, {"example_id": 49, "predicted_answer": "A"}, {"example_id": 50, "predicted_answer": "A"}, {"example_id": 51, "predicted_answer": "A"}, {"example_id": 52, "predicted_answer": "A"}, {"example_id": 53, "predicted_answer": "A"}, {"example_id": 54, "predicted_answer": "A"}, {"example_id": 55, "predicted_answer": "A"}, {"example_id": 56, "predicted_answer": "A"}, {"example_id": 57, "predicted_answer": "A"}, {"example_id": 58, "predicted_answer": "A"}, {"example_id": 59, "predicted_answer": "A"}, {"example_id": 60, "predicted_answer": "A"}, {"example_id": 61, "predicted_answer": "A"}, {"example_id": 62, "predicted_answer": "A"}, {"example_id": 63, "predicted_answer": "A"}, {"example_id": 64, "predicted_answer": "A"}, {"example_id": 65, "predicted_answer": "A"}, {"example_id": 66, "predicted_answer": "A"}, {"example_id": 67, "predicted_answer": "A"}, {"example_id": 68, "predicted_answer": "A"}, {"example_id": 69, "predicted_answer": "A"}, {"example_id": 70, "predicted_answer": "A"}, {"example_id": 71, "predicted_answer": "A"}, {"example_id": 72, "predicted_answer": "A"}, {"example_id": 73, "predicted_answer": "A"}, {"example_id": 74, "predicted_answer": "A"}, {"example_id": 75, "predicted_answer": "A"}, {"example_id": 76, "predicted_answer": "A"}, {"example_id": 77, "predicted_answer": "A"}, {"example_id": 78, "predicted_answer": "A"}, {"example_id": 79, "predicted_answer": "A"}, {"example_id": 80, "predicted_answer": "A"}, {"example_id": 81, "predicted_answer": "A"}, {"example_id": 82, "predicted_answer": "A"}, {"example_id": 83, "predicted_answer": "A"}, {"example_id": 84, "predicted_answer": "A"}, {"example_id": 85, "predicted_answer": "A"}, {"example_id": 86, "predicted_answer": "A"}, {"example_id": 87, "predicted_answer": "A"}, {"example_id": 88, "predicted_answer": "A"}, {"example_id": 89, "predicted_answer": "A"}, {"example_id": 90, "predicted_answer": "A"}, {"example_id": 91, "predicted_answer": "A"}, {"example_id": 92, "predicted_answer": "A"}, {"example_id": 93, "predicted_answer": "A"}, {"example_id": 94, "predicted_answer": "A"}, {"example_id": 95, "predicted_answer": "A"}, {"example_id": 96, "predicted_answer": "A"}, {"example_id": 97, "predicted_answer": "A"}, {"example_id": 98, "predicted_answer": "A"}, {"example_id": 99, "predicted_answer": "A"}, {"example_id": 100, "predicted_answer": "A"}, {"example_id": 101, "predicted_answer": "A"}, {"example_id": 102, "predicted_answer": "A"}, {"example_id": 103, "predicted_answer": "A"}, {"example_id": 104, "predicted_answer": "A"}, {"example_id": 105, "predicted_answer": "A"}, {"example_id": 106, "predicted_answer": "A"}, {"example_id": 107, "predicted_answer": "A"}, {"example_id": 108, "predicted_answer": "A"}, {"example_id": 109, "predicted_answer": "A"}, {"example_id": 110, "predicted_answer": "A"}, {"example_id": 111, "predicted_answer": "A"}, {"example_id": 112, "predicted_answer": "A"}, {"example_id": 113, "predicted_answer": "A"}, {"example_id": 114, "predicted_answer": "A"}, {"example_id": 115, "predicted_answer": "A"}, {"example_id": 116, "predicted_answer": "A"}, {"example_id": 117, "predicted_answer": "A"}, {"example_id": 118, "predicted_answer": "A"}, {"example_id": 119, "predicted_answer": "A"}, {"example_id": 120, "predicted_answer": "A"}, {"example_id": 121, "predicted_answer": "A"}, {"example_id": 122, "predicted_answer": "A"}, {"example_id": 123, "predicted_answer": "A"}, {"example_id": 124, "predicted_answer": "A"}, {"example_id": 125, "predicted_answer": "A"}, {"example_id": 126, "predicted_answer": "A"}, {"example_id": 127, "predicted_answer": "A"}, {"example_id": 128, "predicted_answer": "A"}, {"example_id": 129, "predicted_answer": "A"}, {"example_id": 130, "predicted_answer": "A"}, {"example_id": 131, "predicted_answer": "A"}, {"example_id": 132, "predicted_answer": "A"}, {"example_id": 133, "predicted_answer": "A"}, {"example_id": 134, "predicted_answer": "A"}, {"example_id": 135, "predicted_answer": "A"}, {"example_id": 136, "predicted_answer": "A"}, {"example_id": 137, "predicted_answer": "A"}, {"example_id": 138, "predicted_answer": "A"}, {"example_id": 139, "predicted_answer": "A"}, {"example_id": 140, "predicted_answer": "A"}, {"example_id": 141, "predicted_answer": "A"}, {"example_id": 142, "predicted_answer": "A"}, {"example_id": 143, "predicted_answer": "A"}, {"example_id": 144, "predicted_answer": "A"}, {"example_id": 145, "predicted_answer": "A"}, {"example_id": 146, "predicted_answer": "A"}, {"example_id": 147, "predicted_answer": "A"}, {"example_id": 148, "predicted_answer": "A"}, {"example_id": 149, "predicted_answer": "A"}, {"example_id": 150, "predicted_answer": "A"}, {"example_id": 151, "predicted_answer": "A"}, {"example_id": 152, "predicted_answer": "A"}, {"example_id": 153, "predicted_answer": "A"}, {"example_id": 154, "predicted_answer": "A"}, {"example_id": 155, "predicted_answer": "A"}, {"example_id": 156, "predicted_answer": "A"}, {"example_id": 157, "predicted_answer": "A"}, {"example_id": 158, "predicted_answer": "A"}, {"example_id": 159, "predicted_answer": "A"}, {"example_id": 160, "predicted_answer": "A"}, {"example_id": 161, "predicted_answer": "A"}, {"example_id": 162, "predicted_answer": "A"}, {"example_id": 163, "predicted_answer": "A"}, {"example_id": 164, "predicted_answer": "A"}, {"example_id": 165, "predicted_answer": "A"}, {"example_id": 166, "predicted_answer": "A"}, {"example_id": 167, "predicted_answer": "A"}, {"example_id": 168, "predicted_answer": "A"}, {"example_id": 169, "predicted_answer": "A"}, {"example_id": 170, "predicted_answer": "A"}, {"example_id": 171, "predicted_answer": "A"}, {"example_id": 172, "predicted_answer": "A"}, {"example_id": 173, "predicted_answer": "A"}, {"example_id": 174, "predicted_answer": "A"}, {"example_id": 175, "predicted_answer": "A"}, {"example_id": 176, "predicted_answer": "A"}, {"example_id": 177, "predicted_answer": "A"}, {"example_id": 178, "predicted_answer": "A"}, {"example_id": 179, "predicted_answer": "A"}, {"example_id": 180, "predicted_answer": "A"}, {"example_id": 181, "predicted_answer": "A"}, {"example_id": 182, "predicted_answer": "A"}, {"example_id": 183, "predicted_answer": "A"}, {"example_id": 184, "predicted_answer": "A"}, {"example_id": 185, "predicted_answer": "A"}, {"example_id": 186, "predicted_answer": "A"}, {"example_id": 187, "predicted_answer": "A"}, {"example_id": 188, "predicted_answer": "A"}, {"example_id": 189, "predicted_answer": "A"}, {"example_id": 190, "predicted_answer": "A"}, {"example_id": 191, "predicted_answer": "A"}, {"example_id": 192, "predicted_answer": "A"}, {"example_id": 193, "predicted_answer": "A"}, {"example_id": 194, "predicted_answer": "A"}, {"example_id": 195, "predicted_answer": "A"}, {"example_id": 196, "predicted_answer": "A"}, {"example_id": 197, "predicted_answer": "A"}, {"example_id": 198, "predicted_answer": "A"}, {"example_id": 199, "predicted_answer": "A"}, {"example_id": 200, "predicted_answer": "A"}, {"example_id": 201, "predicted_answer": "A"}, {"example_id": 202, "predicted_answer": "A"}, {"example_id": 203, "predicted_answer": "A"}, {"example_id": 204, "predicted_answer": "A"}, {"example_id": 205, "predicted_answer": "A"}, {"example_id": 206, "predicted_answer": "A"}, {"example_id": 207, "predicted_answer": "A"}, {"example_id": 208, "predicted_answer": "A"}, {"example_id": 209, "predicted_answer": "A"}, {"example_id": 210, "predicted_answer": "A"}, {"example_id": 211, "predicted_answer": "A"}, {"example_id": 212, "predicted_answer": "A"}, {"example_id": 213, "predicted_answer": "A"}, {"example_id": 214, "predicted_answer": "A"}, {"example_id": 215, "predicted_answer": "A"}, {"example_id": 216, "predicted_answer": "A"}, {"example_id": 217, "predicted_answer": "A"}, {"example_id": 218, "predicted_answer": "A"}, {"example_id": 219, "predicted_answer": "A"}, {"example_id": 220, "predicted_answer": "A"}, {"example_id": 221, "predicted_answer": "A"}, {"example_id": 222, "predicted_answer": "A"}, {"example_id": 223, "predicted_answer": "A"}, {"example_id": 224, "predicted_answer": "A"}, {"example_id": 225, "predicted_answer": "A"}, {"example_id": 226, "predicted_answer": "A"}, {"example_id": 227, "predicted_answer": "A"}, {"example_id": 228, "predicted_answer": "A"}, {"example_id": 229, "predicted_answer": "A"}, {"example_id": 230, "predicted_answer": "A"}, {"example_id": 231, "predicted_answer": "A"}, {"example_id": 232, "predicted_answer": "A"}, {"example_id": 233, "predicted_answer": "A"}, {"example_id": 234, "predicted_answer": "A"}, {"example_id": 235, "predicted_answer": "A"}, {"example_id": 236, "predicted_answer": "A"}, {"example_id": 237, "predicted_answer": "A"}, {"example_id": 238, "predicted_answer": "A"}, {"example_id": 239, "predicted_answer": "A"}, {"example_id": 240, "predicted_answer": "A"}, {"example_id": 241, "predicted_answer": "A"}, {"example_id": 242, "predicted_answer": "A"}, {"example_id": 243, "predicted_answer": "A"}, {"example_id": 244, "predicted_answer": "A"}, {"example_id": 245, "predicted_answer": "A"}, {"example_id": 246, "predicted_answer": "A"}, {"example_id": 247, "predicted_answer": "A"}, {"example_id": 248, "predicted_answer": "A"}, {"example_id": 249, "predicted_answer": "A"}, {"example_id": 250, "predicted_answer": "A"}, {"example_id": 251, "predicted_answer": "A"}, {"example_id": 252, "predicted_answer": "A"}, {"example_id": 253, "predicted_answer": "A"}, {"example_id": 254, "predicted_answer": "A"}, {"example_id": 255, "predicted_answer": "A"}, {"example_id": 256, "predicted_answer": "A"}, {"example_id": 257, "predicted_answer": "A"}, {"example_id": 258, "predicted_answer": "A"}, {"example_id": 259, "predicted_answer": "A"}, {"example_id": 260, "predicted_answer": "A"}, {"example_id": 261, "predicted_answer": "A"}, {"example_id": 262, "predicted_answer": "A"}, {"example_id": 263, "predicted_answer": "A"}, {"example_id": 264, "predicted_answer": "A"}, {"example_id": 265, "predicted_answer": "A"}, {"example_id": 266, "predicted_answer": "A"}, {"example_id": 267, "predicted_answer": "A"}, {"example_id": 268, "predicted_answer": "A"}, {"example_id": 269, "predicted_answer": "A"}, {"example_id": 270, "predicted_answer": "A"}, {"example_id": 271, "predicted_answer": "A"}, {"example_id": 272, "predicted_answer": "A"}, {"example_id": 273, "predicted_answer": "A"}, {"example_id": 274, "predicted_answer": "A"}, {"example_id": 275, "predicted_answer": "A"}, {"example_id": 276, "predicted_answer": "A"}, {"example_id": 277, "predicted_answer": "A"}, {"example_id": 278, "predicted_answer": "A"}, {"example_id": 279, "predicted_answer": "A"}, {"example_id": 280, "predicted_answer": "A"}, {"example_id": 281, "predicted_answer": "A"}, {"example_id": 282, "predicted_answer": "A"}, {"example_id": 283, "predicted_answer": "A"}, {"example_id": 284, "predicted_answer": "A"}, {"example_id": 285, "predicted_answer": "A"}, {"example_id": 286, "predicted_answer": "A"}, {"example_id": 287, "predicted_answer": "A"}, {"example_id": 288, "predicted_answer": "A"}, {"example_id": 289, "predicted_answer": "A"}, {"example_id": 290, "predicted_answer": "A"}, {"example_id": 291, "predicted_answer": "A"}, {"example_id": 292, "predicted_answer": "A"}, {"example_id": 293, "predicted_answer": "A"}, {"example_id": 294, "predicted_answer": "A"}, {"example_id": 295, "predicted_answer": "A"}, {"example_id": 296, "predicted_answer": "A"}, {"example_id": 297, "predicted_answer": "A"}, {"example_id": 298, "predicted_answer": "A"}, {"example_id": 299, "predicted_answer": "A"}, {"example_id": 300, "predicted_answer": "A"}, {"example_id": 301, "predicted_answer": "A"}, {"example_id": 302, "predicted_answer": "A"}, {"example_id": 303, "predicted_answer": "A"}, {"example_id": 304, "predicted_answer": "A"}, {"example_id": 305, "predicted_answer": "A"}, {"example_id": 306, "predicted_answer": "A"}, {"example_id": 307, "predicted_answer": "A"}, {"example_id": 308, "predicted_answer": "A"}, {"example_id": 309, "predicted_answer": "A"}, {"example_id": 310, "predicted_answer": "A"}, {"example_id": 311, "predicted_answer": "A"}, {"example_id": 312, "predicted_answer": "A"}, {"example_id": 313, "predicted_answer": "A"}, {"example_id": 314, "predicted_answer": "A"}, {"example_id": 315, "predicted_answer": "A"}, {"example_id": 316, "predicted_answer": "A"}, {"example_id": 317, "predicted_answer": "A"}, {"example_id": 318, "predicted_answer": "A"}, {"example_id": 319, "predicted_answer": "A"}, {"example_id": 320, "predicted_answer": "A"}, {"example_id": 321, "predicted_answer": "A"}, {"example_id": 322, "predicted_answer": "A"}, {"example_id": 323, "predicted_answer": "A"}, {"example_id": 324, "predicted_answer": "A"}, {"example_id": 325, "predicted_answer": "A"}, {"example_id": 326, "predicted_answer": "A"}, {"example_id": 327, "predicted_answer": "A"}, {"example_id": 328, "predicted_answer": "A"}, {"example_id": 329, "predicted_answer": "A"}, {"example_id": 330, "predicted_answer": "A"}, {"example_id": 331, "predicted_answer": "A"}, {"example_id": 332, "predicted_answer": "A"}, {"example_id": 333, "predicted_answer": "A"}, {"example_id": 334, "predicted_answer": "A"}, {"example_id": 335, "predicted_answer": "A"}, {"example_id": 336, "predicted_answer": "A"}, {"example_id": 337, "predicted_answer": "A"}, {"example_id": 338, "predicted_answer": "A"}, {"example_id": 339, "predicted_answer": "A"}, {"example_id": 340, "predicted_answer": "A"}, {"example_id": 341, "predicted_answer": "A"}, {"example_id": 342, "predicted_answer": "A"}, {"example_id": 343, "predicted_answer": "A"}, {"example_id": 344, "predicted_answer": "A"}, {"example_id": 345, "predicted_answer": "A"}, {"example_id": 346, "predicted_answer": "A"}, {"example_id": 347, "predicted_answer": "A"}, {"example_id": 348, "predicted_answer": "A"}, {"example_id": 349, "predicted_answer": "A"}, {"example_id": 350, "predicted_answer": "A"}, {"example_id": 351, "predicted_answer": "A"}, {"example_id": 352, "predicted_answer": "A"}, {"example_id": 353, "predicted_answer": "A"}, {"example_id": 354, "predicted_answer": "A"}, {"example_id": 355, "predicted_answer": "A"}, {"example_id": 356, "predicted_answer": "A"}, {"example_id": 357, "predicted_answer": "A"}, {"example_id": 358, "predicted_answer": "A"}, {"example_id": 359, "predicted_answer": "A"}, {"example_id": 360, "predicted_answer": "A"}, {"example_id": 361, "predicted_answer": "A"}, {"example_id": 362, "predicted_answer": "A"}, {"example_id": 363, "predicted_answer": "A"}, {"example_id": 364, "predicted_answer": "A"}, {"example_id": 365, "predicted_answer": "A"}, {"example_id": 366, "predicted_answer": "A"}, {"example_id": 367, "predicted_answer": "A"}, {"example_id": 368, "predicted_answer": "A"}, {"example_id": 369, "predicted_answer": "A"}, {"example_id": 370, "predicted_answer": "A"}, {"example_id": 371, "predicted_answer": "A"}, {"example_id": 372, "predicted_answer": "A"}, {"example_id": 373, "predicted_answer": "A"}, {"example_id": 374, "predicted_answer": "A"}, {"example_id": 375, "predicted_answer": "A"}, {"example_id": 376, "predicted_answer": "A"}, {"example_id": 377, "predicted_answer": "A"}, {"example_id": 378, "predicted_answer": "A"}, {"example_id": 379, "predicted_answer": "A"}, {"example_id": 380, "predicted_answer": "A"}, {"example_id": 381, "predicted_answer": "A"}, {"example_id": 382, "predicted_answer": "A"}, {"example_id": 383, "predicted_answer": "A"}, {"example_id": 384, "predicted_answer": "A"}, {"example_id": 385, "predicted_answer": "A"}, {"example_id": 386, "predicted_answer": "A"}, {"example_id": 387, "predicted_answer": "A"}, {"example_id": 388, "predicted_answer": "A"}, {"example_id": 389, "predicted_answer": "A"}, {"example_id": 390, "predicted_answer": "A"}, {"example_id": 391, "predicted_answer": "A"}, {"example_id": 392, "predicted_answer": "A"}, {"example_id": 393, "predicted_answer": "A"}, {"example_id": 394, "predicted_answer": "A"}, {"example_id": 395, "predicted_answer": "A"}, {"example_id": 396, "predicted_answer": "A"}, {"example_id": 397, "predicted_answer": "A"}, {"example_id": 398, "predicted_answer": "A"}, {"example_id": 399, "predicted_answer": "A"}, {"example_id": 400, "predicted_answer": "A"}, {"example_id": 401, "predicted_answer": "A"}, {"example_id": 402, "predicted_answer": "A"}, {"example_id": 403, "predicted_answer": "A"}, {"example_id": 404, "predicted_answer": "A"}, {"example_id": 405, "predicted_answer": "A"}, {"example_id": 406, "predicted_answer": "A"}, {"example_id": 407, "predicted_answer": "A"}, {"example_id": 408, "predicted_answer": "A"}, {"example_id": 409, "predicted_answer": "A"}, {"example_id": 410, "predicted_answer": "A"}, {"example_id": 411, "predicted_answer": "A"}, {"example_id": 412, "predicted_answer": "A"}, {"example_id": 413, "predicted_answer": "A"}, {"example_id": 414, "predicted_answer": "A"}, {"example_id": 415, "predicted_answer": "A"}, {"example_id": 416, "predicted_answer": "A"}, {"example_id": 417, "predicted_answer": "A"}, {"example_id": 418, "predicted_answer": "A"}, {"example_id": 419, "predicted_answer": "A"}, {"example_id": 420, "predicted_answer": "A"}, {"example_id": 421, "predicted_answer": "A"}, {"example_id": 422, "predicted_answer": "A"}, {"example_id": 423, "predicted_answer": "A"}, {"example_id": 424, "predicted_answer": "A"}, {"example_id": 425, "predicted_answer": "A"}, {"example_id": 426, "predicted_answer": "A"}, {"example_id": 427, "predicted_answer": "A"}, {"example_id": 428, "predicted_answer": "A"}, {"example_id": 429, "predicted_answer": "A"}, {"example_id": 430, "predicted_answer": "A"}, {"example_id": 431, "predicted_answer": "A"}, {"example_id": 432, "predicted_answer": "A"}, {"example_id": 433, "predicted_answer": "A"}, {"example_id": 434, "predicted_answer": "A"}, {"example_id": 435, "predicted_answer": "A"}, {"example_id": 436, "predicted_answer": "A"}, {"example_id": 437, "predicted_answer": "A"}, {"example_id": 438, "predicted_answer": "A"}, {"example_id": 439, "predicted_answer": "A"}, {"example_id": 440, "predicted_answer": "A"}, {"example_id": 441, "predicted_answer": "A"}, {"example_id": 442, "predicted_answer": "A"}, {"example_id": 443, "predicted_answer": "A"}, {"example_id": 444, "predicted_answer": "A"}, {"example_id": 445, "predicted_answer": "A"}, {"example_id": 446, "predicted_answer": "A"}, {"example_id": 447, "predicted_answer": "A"}, {"example_id": 448, "predicted_answer": "A"}, {"example_id": 449, "predicted_answer": "A"}, {"example_id": 450, "predicted_answer": "A"}, {"example_id": 451, "predicted_answer": "A"}, {"example_id": 452, "predicted_answer": "A"}, {"example_id": 453, "predicted_answer": "A"}, {"example_id": 454, "predicted_answer": "A"}, {"example_id": 455, "predicted_answer": "A"}, {"example_id": 456, "predicted_answer": "A"}, {"example_id": 457, "predicted_answer": "A"}, {"example_id": 458, "predicted_answer": "A"}, {"example_id": 459, "predicted_answer": "A"}, {"example_id": 460, "predicted_answer": "A"}, {"example_id": 461, "predicted_answer": "A"}, {"example_id": 462, "predicted_answer": "A"}, {"example_id": 463, "predicted_answer": "A"}, {"example_id": 464, "predicted_answer": "A"}, {"example_id": 465, "predicted_answer": "A"}, {"example_id": 466, "predicted_answer": "A"}, {"example_id": 467, "predicted_answer": "A"}, {"example_id": 468, "predicted_answer": "A"}, {"example_id": 469, "predicted_answer": "A"}, {"example_id": 470, "predicted_answer": "A"}, {"example_id": 471, "predicted_answer": "A"}, {"example_id": 472, "predicted_answer": "A"}, {"example_id": 473, "predicted_answer": "A"}, {"example_id": 474, "predicted_answer": "A"}, {"example_id": 475, "predicted_answer": "A"}, {"example_id": 476, "predicted_answer": "A"}, {"example_id": 477, "predicted_answer": "A"}, {"example_id": 478, "predicted_answer": "A"}, {"example_id": 479, "predicted_answer": "A"}, {"example_id": 480, "predicted_answer": "A"}, {"example_id": 481, "predicted_answer": "A"}, {"example_id": 482, "predicted_answer": "A"}, {"example_id": 483, "predicted_answer": "A"}, {"example_id": 484, "predicted_answer": "A"}, {"example_id": 485, "predicted_answer": "A"}, {"example_id": 486, "predicted_answer": "A"}, {"example_id": 487, "predicted_answer": "A"}, {"example_id": 488, "predicted_answer": "A"}, {"example_id": 489, "predicted_answer": "A"}, {"example_id": 490, "predicted_answer": "A"}, {"example_id": 491, "predicted_answer": "A"}, {"example_id": 492, "predicted_answer": "A"}, {"example_id": 493, "predicted_answer": "A"}, {"example_id": 494, "predicted_answer": "A"}, {"example_id": 495, "predicted_answer": "A"}, {"example_id": 496, "predicted_answer": "A"}, {"example_id": 497, "predicted_answer": "A"}, {"example_id": 498, "predicted_answer": "A"}, {"example_id": 499, "predicted_answer": "A"}, {"example_id": 500, "predicted_answer": "A"}]

tests/fixtures/oracle_submission.json

@@ -0,0 +1 @@
+[{"example_id": 1, "predicted_answer": "A"}, {"example_id": 2, "predicted_answer": "B"}, {"example_id": 3, "predicted_answer": "C"}, {"example_id": 4, "predicted_answer": "D"}, {"example_id": 5, "predicted_answer": "A"}, {"example_id": 6, "predicted_answer": "B"}, {"example_id": 7, "predicted_answer": "C"}, {"example_id": 8, "predicted_answer": "D"}, {"example_id": 9, "predicted_answer": "E"}, {"example_id": 10, "predicted_answer": "A"}, {"example_id": 11, "predicted_answer": "D"}, {"example_id": 12, "predicted_answer": "E"}, {"example_id": 13, "predicted_answer": "B"}, {"example_id": 14, "predicted_answer": "B"}, {"example_id": 15, "predicted_answer": "C"}, {"example_id": 16, "predicted_answer": "C"}, {"example_id": 17, "predicted_answer": "F"}, {"example_id": 18, "predicted_answer": "E"}, {"example_id": 19, "predicted_answer": "A"}, {"example_id": 20, "predicted_answer": "A"}, {"example_id": 21, "predicted_answer": "B"}, {"example_id": 22, "predicted_answer": "B"}, {"example_id": 23, "predicted_answer": "D"}, {"example_id": 24, "predicted_answer": "C"}, {"example_id": 25, "predicted_answer": "A"}, {"example_id": 26, "predicted_answer": "C"}, {"example_id": 27, "predicted_answer": "E"}, {"example_id": 28, "predicted_answer": "A"}, {"example_id": 29, "predicted_answer": "B"}, {"example_id": 30, "predicted_answer": "C"}, {"example_id": 31, "predicted_answer": "F"}, {"example_id": 32, "predicted_answer": "A"}, {"example_id": 33, "predicted_answer": "D"}, {"example_id": 34, "predicted_answer": "E"}, {"example_id": 35, "predicted_answer": "C"}, {"example_id": 36, "predicted_answer": "C"}, {"example_id": 37, "predicted_answer": "C"}, {"example_id": 38, "predicted_answer": "D"}, {"example_id": 39, "predicted_answer": "D"}, {"example_id": 40, "predicted_answer": "C"}, {"example_id": 41, "predicted_answer": "B"}, {"example_id": 42, "predicted_answer": "B"}, {"example_id": 43, "predicted_answer": "B"}, {"example_id": 44, "predicted_answer": "B"}, {"example_id": 45, "predicted_answer": "C"}, {"example_id": 46, "predicted_answer": "B"}, {"example_id": 47, "predicted_answer": "B"}, {"example_id": 48, "predicted_answer": "A"}, {"example_id": 49, "predicted_answer": "B"}, {"example_id": 50, "predicted_answer": "E"}, {"example_id": 51, "predicted_answer": "F"}, {"example_id": 52, "predicted_answer": "F"}, {"example_id": 53, "predicted_answer": "F"}, {"example_id": 54, "predicted_answer": "F"}, {"example_id": 55, "predicted_answer": "F"}, {"example_id": 56, "predicted_answer": "F"}, {"example_id": 57, "predicted_answer": "F"}, {"example_id": 58, "predicted_answer": "F"}, {"example_id": 59, "predicted_answer": "F"}, {"example_id": 60, "predicted_answer": "F"}, {"example_id": 61, "predicted_answer": "F"}, {"example_id": 62, "predicted_answer": "F"}, {"example_id": 63, "predicted_answer": "F"}, {"example_id": 64, "predicted_answer": "F"}, {"example_id": 65, "predicted_answer": "F"}, {"example_id": 66, "predicted_answer": "F"}, {"example_id": 67, "predicted_answer": "F"}, {"example_id": 68, "predicted_answer": "F"}, {"example_id": 69, "predicted_answer": "B"}, {"example_id": 70, "predicted_answer": "C"}, {"example_id": 71, "predicted_answer": "D"}, {"example_id": 72, "predicted_answer": "E"}, {"example_id": 73, "predicted_answer": "F"}, {"example_id": 74, "predicted_answer": "A"}, {"example_id": 75, "predicted_answer": "B"}, {"example_id": 76, "predicted_answer": "C"}, {"example_id": 77, "predicted_answer": "D"}, {"example_id": 78, "predicted_answer": "E"}, {"example_id": 79, "predicted_answer": "F"}, {"example_id": 80, "predicted_answer": "E"}, {"example_id": 81, "predicted_answer": "E"}, {"example_id": 82, "predicted_answer": "E"}, {"example_id": 83, "predicted_answer": "A"}, {"example_id": 84, "predicted_answer": "B"}, {"example_id": 85, "predicted_answer": "C"}, {"example_id": 86, "predicted_answer": "D"}, {"example_id": 87, "predicted_answer": "E"}, {"example_id": 88, "predicted_answer": "A"}, {"example_id": 89, "predicted_answer": "B"}, {"example_id": 90, "predicted_answer": "D"}, {"example_id": 91, "predicted_answer": "E"}, {"example_id": 92, "predicted_answer": "A"}, {"example_id": 93, "predicted_answer": "C"}, {"example_id": 94, "predicted_answer": "C"}, {"example_id": 95, "predicted_answer": "D"}, {"example_id": 96, "predicted_answer": "A"}, {"example_id": 97, "predicted_answer": "B"}, {"example_id": 98, "predicted_answer": "C"}, {"example_id": 99, "predicted_answer": "D"}, {"example_id": 100, "predicted_answer": "A"}, {"example_id": 101, "predicted_answer": "B"}, {"example_id": 102, "predicted_answer": "C"}, {"example_id": 103, "predicted_answer": "D"}, {"example_id": 104, "predicted_answer": "A"}, {"example_id": 105, "predicted_answer": "B"}, {"example_id": 106, "predicted_answer": "D"}, {"example_id": 107, "predicted_answer": "A"}, {"example_id": 108, "predicted_answer": "C"}, {"example_id": 109, "predicted_answer": "B"}, {"example_id": 110, "predicted_answer": "C"}, {"example_id": 111, "predicted_answer": "D"}, {"example_id": 112, "predicted_answer": "D"}, {"example_id": 113, "predicted_answer": "E"}, {"example_id": 114, "predicted_answer": "A"}, {"example_id": 115, "predicted_answer": "B"}, {"example_id": 116, "predicted_answer": "A"}, {"example_id": 117, "predicted_answer": "C"}, {"example_id": 118, "predicted_answer": "C"}, {"example_id": 119, "predicted_answer": "D"}, {"example_id": 120, "predicted_answer": "E"}, {"example_id": 121, "predicted_answer": "F"}, {"example_id": 122, "predicted_answer": "B"}, {"example_id": 123, "predicted_answer": "D"}, {"example_id": 124, "predicted_answer": "D"}, {"example_id": 125, "predicted_answer": "D"}, {"example_id": 126, "predicted_answer": "B"}, {"example_id": 127, "predicted_answer": "D"}, {"example_id": 128, "predicted_answer": "D"}, {"example_id": 129, "predicted_answer": "B"}, {"example_id": 130, "predicted_answer": "C"}, {"example_id": 131, "predicted_answer": "D"}, {"example_id": 132, "predicted_answer": "E"}, {"example_id": 133, "predicted_answer": "F"}, {"example_id": 134, "predicted_answer": "A"}, {"example_id": 135, "predicted_answer": "D"}, {"example_id": 136, "predicted_answer": "E"}, {"example_id": 137, "predicted_answer": "B"}, {"example_id": 138, "predicted_answer": "C"}, {"example_id": 139, "predicted_answer": "B"}, {"example_id": 140, "predicted_answer": "A"}, {"example_id": 141, "predicted_answer": "A"}, {"example_id": 142, "predicted_answer": "C"}, {"example_id": 143, "predicted_answer": "D"}, {"example_id": 144, "predicted_answer": "E"}, {"example_id": 145, "predicted_answer": "E"}, {"example_id": 146, "predicted_answer": "G"}, {"example_id": 147, "predicted_answer": "F"}, {"example_id": 148, "predicted_answer": "D"}, {"example_id": 149, "predicted_answer": "D"}, {"example_id": 150, "predicted_answer": "A"}, {"example_id": 151, "predicted_answer": "C"}, {"example_id": 152, "predicted_answer": "A"}, {"example_id": 153, "predicted_answer": "E"}, {"example_id": 154, "predicted_answer": "A"}, {"example_id": 155, "predicted_answer": "A"}, {"example_id": 156, "predicted_answer": "A"}, {"example_id": 157, "predicted_answer": "A"}, {"example_id": 158, "predicted_answer": "A"}, {"example_id": 159, "predicted_answer": "B"}, {"example_id": 160, "predicted_answer": "A"}, {"example_id": 161, "predicted_answer": "E"}, {"example_id": 162, "predicted_answer": "G"}, {"example_id": 163, "predicted_answer": "A"}, {"example_id": 164, "predicted_answer": "G"}, {"example_id": 165, "predicted_answer": "A"}, {"example_id": 166, "predicted_answer": "B"}, {"example_id": 167, "predicted_answer": "B"}, {"example_id": 168, "predicted_answer": "F"}, {"example_id": 169, "predicted_answer": "F"}, {"example_id": 170, "predicted_answer": "G"}, {"example_id": 171, "predicted_answer": "G"}, {"example_id": 172, "predicted_answer": "E"}, {"example_id": 173, "predicted_answer": "F"}, {"example_id": 174, "predicted_answer": "A"}, {"example_id": 175, "predicted_answer": "B"}, {"example_id": 176, "predicted_answer": "B"}, {"example_id": 177, "predicted_answer": "C"}, {"example_id": 178, "predicted_answer": "B"}, {"example_id": 179, "predicted_answer": "F"}, {"example_id": 180, "predicted_answer": "F"}, {"example_id": 181, "predicted_answer": "G"}, {"example_id": 182, "predicted_answer": "E"}, {"example_id": 183, "predicted_answer": "G"}, {"example_id": 184, "predicted_answer": "E"}, {"example_id": 185, "predicted_answer": "B"}, {"example_id": 186, "predicted_answer": "B"}, {"example_id": 187, "predicted_answer": "C"}, {"example_id": 188, "predicted_answer": "F"}, {"example_id": 189, "predicted_answer": "E"}, {"example_id": 190, "predicted_answer": "D"}, {"example_id": 191, "predicted_answer": "D"}, {"example_id": 192, "predicted_answer": "D"}, {"example_id": 193, "predicted_answer": "D"}, {"example_id": 194, "predicted_answer": "C"}, {"example_id": 195, "predicted_answer": "E"}, {"example_id": 196, "predicted_answer": "F"}, {"example_id": 197, "predicted_answer": "B"}, {"example_id": 198, "predicted_answer": "F"}, {"example_id": 199, "predicted_answer": "C"}, {"example_id": 200, "predicted_answer": "C"}, {"example_id": 201, "predicted_answer": "C"}, {"example_id": 202, "predicted_answer": "D"}, {"example_id": 203, "predicted_answer": "E"}, {"example_id": 204, "predicted_answer": "C"}, {"example_id": 205, "predicted_answer": "F"}, {"example_id": 206, "predicted_answer": "F"}, {"example_id": 207, "predicted_answer": "E"}, {"example_id": 208, "predicted_answer": "D"}, {"example_id": 209, "predicted_answer": "E"}, {"example_id": 210, "predicted_answer": "E"}, {"example_id": 211, "predicted_answer": "D"}, {"example_id": 212, "predicted_answer": "D"}, {"example_id": 213, "predicted_answer": "B"}, {"example_id": 214, "predicted_answer": "D"}, {"example_id": 215, "predicted_answer": "A"}, {"example_id": 216, "predicted_answer": "B"}, {"example_id": 217, "predicted_answer": "C"}, {"example_id": 218, "predicted_answer": "C"}, {"example_id": 219, "predicted_answer": "C"}, {"example_id": 220, "predicted_answer": "F"}, {"example_id": 221, "predicted_answer": "E"}, {"example_id": 222, "predicted_answer": "E"}, {"example_id": 223, "predicted_answer": "D"}, {"example_id": 224, "predicted_answer": "D"}, {"example_id": 225, "predicted_answer": "B"}, {"example_id": 226, "predicted_answer": "D"}, {"example_id": 227, "predicted_answer": "C"}, {"example_id": 228, "predicted_answer": "C"}, {"example_id": 229, "predicted_answer": "A"}, {"example_id": 230, "predicted_answer": "B"}, {"example_id": 231, "predicted_answer": "E"}, {"example_id": 232, "predicted_answer": "E"}, {"example_id": 233, "predicted_answer": "C"}, {"example_id": 234, "predicted_answer": "F"}, {"example_id": 235, "predicted_answer": "D"}, {"example_id": 236, "predicted_answer": "F"}, {"example_id": 237, "predicted_answer": "C"}, {"example_id": 238, "predicted_answer": "B"}, {"example_id": 239, "predicted_answer": "A"}, {"example_id": 240, "predicted_answer": "B"}, {"example_id": 241, "predicted_answer": "C"}, {"example_id": 242, "predicted_answer": "B"}, {"example_id": 243, "predicted_answer": "C"}, {"example_id": 244, "predicted_answer": "C"}, {"example_id": 245, "predicted_answer": "D"}, {"example_id": 246, "predicted_answer": "D"}, {"example_id": 247, "predicted_answer": "C"}, {"example_id": 248, "predicted_answer": "C"}, {"example_id": 249, "predicted_answer": "D"}, {"example_id": 250, "predicted_answer": "C"}, {"example_id": 251, "predicted_answer": "C"}, {"example_id": 252, "predicted_answer": "C"}, {"example_id": 253, "predicted_answer": "C"}, {"example_id": 254, "predicted_answer": "C"}, {"example_id": 255, "predicted_answer": "C"}, {"example_id": 256, "predicted_answer": "C"}, {"example_id": 257, "predicted_answer": "E"}, {"example_id": 258, "predicted_answer": "C"}, {"example_id": 259, "predicted_answer": "B"}, {"example_id": 260, "predicted_answer": "B"}, {"example_id": 261, "predicted_answer": "B"}, {"example_id": 262, "predicted_answer": "C"}, {"example_id": 263, "predicted_answer": "C"}, {"example_id": 264, "predicted_answer": "B"}, {"example_id": 265, "predicted_answer": "B"}, {"example_id": 266, "predicted_answer": "B"}, {"example_id": 267, "predicted_answer": "C"}, {"example_id": 268, "predicted_answer": "F"}, {"example_id": 269, "predicted_answer": "F"}, {"example_id": 270, "predicted_answer": "A"}, {"example_id": 271, "predicted_answer": "A"}, {"example_id": 272, "predicted_answer": "E"}, {"example_id": 273, "predicted_answer": "F"}, {"example_id": 274, "predicted_answer": "F"}, {"example_id": 275, "predicted_answer": "D"}, {"example_id": 276, "predicted_answer": "B"}, {"example_id": 277, "predicted_answer": "C"}, {"example_id": 278, "predicted_answer": "F"}, {"example_id": 279, "predicted_answer": "F"}, {"example_id": 280, "predicted_answer": "E"}, {"example_id": 281, "predicted_answer": "C"}, {"example_id": 282, "predicted_answer": "D"}, {"example_id": 283, "predicted_answer": "C"}, {"example_id": 284, "predicted_answer": "E"}, {"example_id": 285, "predicted_answer": "C"}, {"example_id": 286, "predicted_answer": "B"}, {"example_id": 287, "predicted_answer": "D"}, {"example_id": 288, "predicted_answer": "E"}, {"example_id": 289, "predicted_answer": "A"}, {"example_id": 290, "predicted_answer": "D"}, {"example_id": 291, "predicted_answer": "B"}, {"example_id": 292, "predicted_answer": "D"}, {"example_id": 293, "predicted_answer": "E"}, {"example_id": 294, "predicted_answer": "F"}, {"example_id": 295, "predicted_answer": "C"}, {"example_id": 296, "predicted_answer": "A"}, {"example_id": 297, "predicted_answer": "C"}, {"example_id": 298, "predicted_answer": "D"}, {"example_id": 299, "predicted_answer": "B"}, {"example_id": 300, "predicted_answer": "D"}, {"example_id": 301, "predicted_answer": "A"}, {"example_id": 302, "predicted_answer": "C"}, {"example_id": 303, "predicted_answer": "A"}, {"example_id": 304, "predicted_answer": "A"}, {"example_id": 305, "predicted_answer": "E"}, {"example_id": 306, "predicted_answer": "F"}, {"example_id": 307, "predicted_answer": "E"}, {"example_id": 308, "predicted_answer": "D"}, {"example_id": 309, "predicted_answer": "D"}, {"example_id": 310, "predicted_answer": "F"}, {"example_id": 311, "predicted_answer": "D"}, {"example_id": 312, "predicted_answer": "B"}, {"example_id": 313, "predicted_answer": "A"}, {"example_id": 314, "predicted_answer": "B"}, {"example_id": 315, "predicted_answer": "C"}, {"example_id": 316, "predicted_answer": "D"}, {"example_id": 317, "predicted_answer": "B"}, {"example_id": 318, "predicted_answer": "A"}, {"example_id": 319, "predicted_answer": "E"}, {"example_id": 320, "predicted_answer": "C"}, {"example_id": 321, "predicted_answer": "A"}, {"example_id": 322, "predicted_answer": "B"}, {"example_id": 323, "predicted_answer": "D"}, {"example_id": 324, "predicted_answer": "D"}, {"example_id": 325, "predicted_answer": "D"}, {"example_id": 326, "predicted_answer": "F"}, {"example_id": 327, "predicted_answer": "D"}, {"example_id": 328, "predicted_answer": "C"}, {"example_id": 329, "predicted_answer": "C"}, {"example_id": 330, "predicted_answer": "B"}, {"example_id": 331, "predicted_answer": "D"}, {"example_id": 332, "predicted_answer": "F"}, {"example_id": 333, "predicted_answer": "E"}, {"example_id": 334, "predicted_answer": "F"}, {"example_id": 335, "predicted_answer": "E"}, {"example_id": 336, "predicted_answer": "C"}, {"example_id": 337, "predicted_answer": "E"}, {"example_id": 338, "predicted_answer": "E"}, {"example_id": 339, "predicted_answer": "E"}, {"example_id": 340, "predicted_answer": "F"}, {"example_id": 341, "predicted_answer": "F"}, {"example_id": 342, "predicted_answer": "D"}, {"example_id": 343, "predicted_answer": "A"}, {"example_id": 344, "predicted_answer": "C"}, {"example_id": 345, "predicted_answer": "E"}, {"example_id": 346, "predicted_answer": "B"}, {"example_id": 347, "predicted_answer": "B"}, {"example_id": 348, "predicted_answer": "B"}, {"example_id": 349, "predicted_answer": "E"}, {"example_id": 350, "predicted_answer": "F"}, {"example_id": 351, "predicted_answer": "C"}, {"example_id": 352, "predicted_answer": "A"}, {"example_id": 353, "predicted_answer": "B"}, {"example_id": 354, "predicted_answer": "B"}, {"example_id": 355, "predicted_answer": "F"}, {"example_id": 356, "predicted_answer": "C"}, {"example_id": 357, "predicted_answer": "F"}, {"example_id": 358, "predicted_answer": "C"}, {"example_id": 359, "predicted_answer": "C"}, {"example_id": 360, "predicted_answer": "B"}, {"example_id": 361, "predicted_answer": "C"}, {"example_id": 362, "predicted_answer": "D"}, {"example_id": 363, "predicted_answer": "A"}, {"example_id": 364, "predicted_answer": "B"}, {"example_id": 365, "predicted_answer": "B"}, {"example_id": 366, "predicted_answer": "A"}, {"example_id": 367, "predicted_answer": "A"}, {"example_id": 368, "predicted_answer": "B"}, {"example_id": 369, "predicted_answer": "E"}, {"example_id": 370, "predicted_answer": "F"}, {"example_id": 371, "predicted_answer": "E"}, {"example_id": 372, "predicted_answer": "A"}, {"example_id": 373, "predicted_answer": "D"}, {"example_id": 374, "predicted_answer": "B"}, {"example_id": 375, "predicted_answer": "C"}, {"example_id": 376, "predicted_answer": "D"}, {"example_id": 377, "predicted_answer": "A"}, {"example_id": 378, "predicted_answer": "F"}, {"example_id": 379, "predicted_answer": "E"}, {"example_id": 380, "predicted_answer": "A"}, {"example_id": 381, "predicted_answer": "C"}, {"example_id": 382, "predicted_answer": "A"}, {"example_id": 383, "predicted_answer": "F"}, {"example_id": 384, "predicted_answer": "A"}, {"example_id": 385, "predicted_answer": "B"}, {"example_id": 386, "predicted_answer": "F"}, {"example_id": 387, "predicted_answer": "E"}, {"example_id": 388, "predicted_answer": "C"}, {"example_id": 389, "predicted_answer": "B"}, {"example_id": 390, "predicted_answer": "C"}, {"example_id": 391, "predicted_answer": "D"}, {"example_id": 392, "predicted_answer": "A"}, {"example_id": 393, "predicted_answer": "C"}, {"example_id": 394, "predicted_answer": "F"}, {"example_id": 395, "predicted_answer": "E"}, {"example_id": 396, "predicted_answer": "F"}, {"example_id": 397, "predicted_answer": "E"}, {"example_id": 398, "predicted_answer": "D"}, {"example_id": 399, "predicted_answer": "D"}, {"example_id": 400, "predicted_answer": "D"}, {"example_id": 401, "predicted_answer": "B"}, {"example_id": 402, "predicted_answer": "E"}, {"example_id": 403, "predicted_answer": "G"}, {"example_id": 404, "predicted_answer": "G"}, {"example_id": 405, "predicted_answer": "G"}, {"example_id": 406, "predicted_answer": "C"}, {"example_id": 407, "predicted_answer": "G"}, {"example_id": 408, "predicted_answer": "A"}, {"example_id": 409, "predicted_answer": "F"}, {"example_id": 410, "predicted_answer": "B"}, {"example_id": 411, "predicted_answer": "H"}, {"example_id": 412, "predicted_answer": "E"}, {"example_id": 413, "predicted_answer": "F"}, {"example_id": 414, "predicted_answer": "C"}, {"example_id": 415, "predicted_answer": "F"}, {"example_id": 416, "predicted_answer": "C"}, {"example_id": 417, "predicted_answer": "B"}, {"example_id": 418, "predicted_answer": "C"}, {"example_id": 419, "predicted_answer": "G"}, {"example_id": 420, "predicted_answer": "A"}, {"example_id": 421, "predicted_answer": "B"}, {"example_id": 422, "predicted_answer": "H"}, {"example_id": 423, "predicted_answer": "E"}, {"example_id": 424, "predicted_answer": "G"}, {"example_id": 425, "predicted_answer": "G"}, {"example_id": 426, "predicted_answer": "G"}, {"example_id": 427, "predicted_answer": "B"}, {"example_id": 428, "predicted_answer": "E"}, {"example_id": 429, "predicted_answer": "C"}, {"example_id": 430, "predicted_answer": "C"}, {"example_id": 431, "predicted_answer": "E"}, {"example_id": 432, "predicted_answer": "F"}, {"example_id": 433, "predicted_answer": "A"}, {"example_id": 434, "predicted_answer": "D"}, {"example_id": 435, "predicted_answer": "D"}, {"example_id": 436, "predicted_answer": "C"}, {"example_id": 437, "predicted_answer": "H"}, {"example_id": 438, "predicted_answer": "H"}, {"example_id": 439, "predicted_answer": "A"}, {"example_id": 440, "predicted_answer": "F"}, {"example_id": 441, "predicted_answer": "C"}, {"example_id": 442, "predicted_answer": "F"}, {"example_id": 443, "predicted_answer": "G"}, {"example_id": 444, "predicted_answer": "G"}, {"example_id": 445, "predicted_answer": "G"}, {"example_id": 446, "predicted_answer": "D"}, {"example_id": 447, "predicted_answer": "B"}, {"example_id": 448, "predicted_answer": "A"}, {"example_id": 449, "predicted_answer": "A"}, {"example_id": 450, "predicted_answer": "I"}, {"example_id": 451, "predicted_answer": "B"}, {"example_id": 452, "predicted_answer": "I"}, {"example_id": 453, "predicted_answer": "F"}, {"example_id": 454, "predicted_answer": "F"}, {"example_id": 455, "predicted_answer": "J"}, {"example_id": 456, "predicted_answer": "H"}, {"example_id": 457, "predicted_answer": "C"}, {"example_id": 458, "predicted_answer": "A"}, {"example_id": 459, "predicted_answer": "B"}, {"example_id": 460, "predicted_answer": "C"}, {"example_id": 461, "predicted_answer": "I"}, {"example_id": 462, "predicted_answer": "E"}, {"example_id": 463, "predicted_answer": "A"}, {"example_id": 464, "predicted_answer": "E"}, {"example_id": 465, "predicted_answer": "F"}, {"example_id": 466, "predicted_answer": "C"}, {"example_id": 467, "predicted_answer": "E"}, {"example_id": 468, "predicted_answer": "D"}, {"example_id": 469, "predicted_answer": "J"}, {"example_id": 470, "predicted_answer": "A"}, {"example_id": 471, "predicted_answer": "C"}, {"example_id": 472, "predicted_answer": "D"}, {"example_id": 473, "predicted_answer": "J"}, {"example_id": 474, "predicted_answer": "H"}, {"example_id": 475, "predicted_answer": "F"}, {"example_id": 476, "predicted_answer": "E"}, {"example_id": 477, "predicted_answer": "J"}, {"example_id": 478, "predicted_answer": "A"}, {"example_id": 479, "predicted_answer": "D"}, {"example_id": 480, "predicted_answer": "F"}, {"example_id": 481, "predicted_answer": "G"}, {"example_id": 482, "predicted_answer": "A"}, {"example_id": 483, "predicted_answer": "E"}, {"example_id": 484, "predicted_answer": "J"}, {"example_id": 485, "predicted_answer": "C"}, {"example_id": 486, "predicted_answer": "F"}, {"example_id": 487, "predicted_answer": "F"}, {"example_id": 488, "predicted_answer": "B"}, {"example_id": 489, "predicted_answer": "F"}, {"example_id": 490, "predicted_answer": "C"}, {"example_id": 491, "predicted_answer": "D"}, {"example_id": 492, "predicted_answer": "C"}, {"example_id": 493, "predicted_answer": "A"}, {"example_id": 494, "predicted_answer": "H"}, {"example_id": 495, "predicted_answer": "H"}, {"example_id": 496, "predicted_answer": "A"}, {"example_id": 497, "predicted_answer": "H"}, {"example_id": 498, "predicted_answer": "F"}, {"example_id": 499, "predicted_answer": "A"}, {"example_id": 500, "predicted_answer": "B"}]

tests/test_evaluator.py

@@ -0,0 +1,52 @@
+"""Tests for evaluator.score_submission().
+
+Run from the EgoMemReason-Space/ directory:
+    python -m pytest tests/ -q
+"""
+
+import json
+import os
+import pathlib
+import sys
+
+import pytest
+
+ROOT = pathlib.Path(__file__).resolve().parents[1]
+sys.path.insert(0, str(ROOT))
+
+import evaluator
+
+ANN = ROOT / "annotations_private.json"
+ORACLE = ROOT / "tests" / "fixtures" / "oracle_submission.json"
+ALL_A = ROOT / "tests" / "fixtures" / "all_a_submission.json"
+
+
+pytestmark = pytest.mark.skipif(
+    not ANN.exists(),
+    reason=f"{ANN} not present (copy from ../EgoMemReason-EvalAI.archived/)",
+)
+
+
+def test_oracle_scores_100():
+    metrics = evaluator.score_submission(str(ORACLE), str(ANN))
+    for k, v in metrics.items():
+        assert v == 100.0, f"{k} should be 100.0, got {v}"
+
+
+def test_all_a_scores_around_14():
+    # All-A's exact score depends on the A-letter frequency in the dataset
+    # — we measured 14.2% during the EvalAI port. Allow a wide band.
+    metrics = evaluator.score_submission(str(ALL_A), str(ANN))
+    assert 10.0 <= metrics["Overall"] <= 20.0, metrics
+
+
+def test_broken_submission_raises(tmp_path):
+    broken = tmp_path / "broken.json"
+    json.dump(
+        [{"example_id": 1, "predicted_answer": "ZZ"}],   # bogus letter + only 1 row
+        broken.open("w"),
+    )
+    with pytest.raises(ValueError) as exc:
+        evaluator.score_submission(str(broken), str(ANN))
+    assert "must be one of" in str(exc.value)
+    assert "missing" in str(exc.value)