logs/README.md

Push logs — chronological

Persisted output of every metadata-stub push run. Each file is raw stdout from scripts/push_metadata_stubs.py, kept verbatim so we have a paper trail of which slugs went up when and what errored.

Session: 2026-04-19

push_batch_initial_10.log

First sanity run. 10 diverse stubs pushed to validate the renderer on a variety of sources (NEMAR HBN releases, OpenNeuro, MEG, iEEG, various pathologies).

Slugs: ds000117, ds000246, ds000247, EEG2025r1, ds003800, ds002799, EEG2025r10, ds004551, ds004598, ds003061.

Pushed: 10. Failures: 0. Wall clock: ~3m13s (serial, ~20 s/push).

push_bulk_parallel_day1.log

Option C: every remaining slug, parallelized with 12 workers.

• Attempted: 670 (736 total − 10 initial − 56 already on HF from prior runs).
• Pushed: 233 (18:02:37 → 18:09:02, ~1.7 s/push wall clock × 12 threads).
• Hit HF's org-level rate limit of 300 dataset repo creations per day.
• Remaining: ~437 (queued for tomorrow).

Script is idempotent — resume with the same command:

python scripts/push_metadata_stubs.py --all --skip-existing --workers 12

Totals after day 1

• EEGDash/* dataset repos live: 299 (plus the catalog Space).
• Remaining to push: ~437.
• HF rate-limit window resets 24 h after the first repo creation of the day.

Session: 2026-04-20

push_bulk_parallel_day2.log

Second bulk run, kicked off after the 24h reset. Same command, same 12 workers.

• Attempted: 435 remaining (after day 1's 299 + a probe push).
• Pushed: 299. Hit the 300/day quota again (probe earlier in day burned one slot).
• Wall clock: ~6 min (12 workers, ~1.2 s/push thread).

Totals after day 2

• EEGDash/* dataset repos live: 600.
• Remaining: ~136.
• A recurring cron (10 */6 * * *, job id kept in session state) retries the same command until all are pushed; --skip-existing makes each attempt idempotent.

Session: 2026-04-20 (evening) — queued for tomorrow

Hit the 300/day HF quota again mid-afternoon. All pending work is queued via the existing recurring cron (job 586420c6, 10 */6 * * *) — next fires at 00:10, 06:10, 12:10, 18:10 local. The first fire after HF's window refreshes picks up everything via --skip-existing.

Pending when quota reopens (~24 h from the first 429 of day 2)

1. ~136 remaining metadata stubs — the cron will push them automatically (python scripts/push_metadata_stubs.py --all --skip-existing --workers 12).
2. Push the org card at org-readme/README.md. Two ways:
   • Create EEGDash/README as a Space (amazon-style — lets us host images alongside the card):

     from huggingface_hub import HfApi
     api = HfApi()
     api.create_repo("EEGDash/README", repo_type="space", space_sdk="static", exist_ok=True)
     api.upload_file(
         repo_id="EEGDash/README",
         repo_type="space",
         path_or_fileobj="org-readme/README.md",
         path_in_repo="README.md",
     )
     

   • Or just paste the markdown from org-readme/README.md into the description field at https://huggingface.co/organizations/EEGDash/settings.
3. Empty-commit the Space once stubs are done so the on 🤗 cache refreshes.

Reference: current state

Total datasets in CSV	736
Mirrored to HF	600
Remaining	~136
Org card drafted	org-readme/README.md (pushed to Space repo)
Org card published	no (blocked on quota)
HF rate limit	300 repo-creations / 24 h org-wide
First 429 day 2	2026-04-20 ~18:34 UTC
Earliest clean window	~2026-04-21 18:30 UTC

Session: 2026-04-21 — closed ✅

push_bulk_parallel_day3.log

Cron 586420c6 fired at 23:41 local. 24 h window had cleared.

• Attempted: 136 remaining.
• Pushed: 136 / 136 (zero 429s, zero failures).
• Wall clock: ~16 s total (12 workers, <200 ms per push thread).

Totals after day 3 (final)

• EEGDash/* dataset repos live: 736 — complete coverage of the CSV.
• 600 skipped from earlier runs.
• Recurring cron 586420c6 deleted after verification.

day	attempted	pushed	429s	cumulative
2026-04-19	10 + 670	10 + 233	0 + 437	299
2026-04-20	435	299	136	600
2026-04-21	136	136	0	736