Emaad commited on
Commit
4d38d64
Β·
1 Parent(s): adbd260

Add model card: finetuning recipe + GIFT-Eval bar metrics + base-collection CTA (#1)

Browse files

- Add model card: finetuning recipe + GIFT-Eval bar metrics + base-collection CTA (f993da455e005e4d9c903b1bc7fdb7d36fd3b55d)
- Tighten spacing: drop section HRs; turn post-image prose into figcaption (2eb07259fdc8f6758a6f30f37982738a3c36a2ed)

Files changed (3) hide show
  1. .gitattributes +1 -0
  2. README.md +103 -0
  3. assets/bar_metrics_gift_eval.png +3 -0
.gitattributes CHANGED
@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
 
 
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
36
+ assets/bar_metrics_gift_eval.png filter=lfs diff=lfs merge=lfs -text
README.md ADDED
@@ -0,0 +1,103 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ tags:
3
+ - time-series-forecasting
4
+ - foundation-models
5
+ - finetuned
6
+ - time-series
7
+ - timeseries
8
+ - forecasting
9
+ - observability
10
+ - gift-eval
11
+ - safetensors
12
+ - pytorch_model_hub_mixin
13
+ license: apache-2.0
14
+ pipeline_tag: time-series-forecasting
15
+ thumbnail: https://corp.dd-static.net/img/about/presskit/kit/press_kit.png
16
+ base_model: Datadog/Toto-2.0-2.5B
17
+ model-index:
18
+ - name: Toto-2.0-2.5B-FT
19
+ results:
20
+ - task:
21
+ type: time-series-forecasting
22
+ dataset:
23
+ name: GIFT-Eval
24
+ type: GIFT-Eval
25
+ metrics:
26
+ - name: CRPS
27
+ type: CRPS
28
+ value: 0.463
29
+ - name: MASE
30
+ type: MASE
31
+ value: 0.679
32
+ source:
33
+ name: GIFT-Eval Time Series Forecasting Leaderboard
34
+ url: https://huggingface.co/spaces/Salesforce/GIFT-Eval
35
+ ---
36
+
37
+ # Toto-2.0-2.5B-FT
38
+
39
+ > [!WARNING]
40
+ > **This is a benchmarking checkpoint, not a general-purpose model.**
41
+ > Toto-2.0-2.5B-FT is the [Toto 2.0 2.5B](https://huggingface.co/Datadog/Toto-2.0-2.5B) base model finetuned on the GIFT-Eval training split for our **#2-on-GIFT-Eval-leaderboard** submission. It is released for reproducibility only.
42
+ >
43
+ > For real workloads, please use the base [Toto 2.0 collection](https://huggingface.co/collections/Datadog/toto-20). The base checkpoints are pretrained without any public data, generalize to every benchmark we have evaluated, and are what we recommend deploying.
44
+
45
+ ## ✨ What this is
46
+
47
+ A single Toto 2.0 2.5B base checkpoint finetuned on a mix that **includes the GIFT-Eval training split**, used to probe how far the base model can be pushed on a single in-distribution benchmark.
48
+
49
+ <figure>
50
+ <img src="assets/bar_metrics_gift_eval.png" alt="GIFT-Eval bar metrics β€” Toto 2.0 2.5B-FT highlighted">
51
+ <figcaption>On the full GIFT-Eval leaderboard (foundation models + finetuned + ensemble + agentic), Toto-2.0-2.5B-FT places <b>#2 on CRPS rank, MASE rank, and #3 on raw CRPS / MASE</b>, behind only the <a href="https://huggingface.co/Datadog/Toto-2.0-Family-and-Friends">Toto 2.0 Family-and-Friends</a> ensemble.</figcaption>
52
+ </figure>
53
+
54
+ ## πŸ” Finetuning recipe
55
+
56
+ Starting from a fully-decayed [Toto-2.0-2.5B](https://huggingface.co/Datadog/Toto-2.0-2.5B) base checkpoint, we finetuned for 10,000 steps on a mix designed to expose the model to in-distribution structure without overfitting to GIFT-Eval alone:
57
+
58
+ | Source | Share |
59
+ |---|---:|
60
+ | GIFT-Eval Pretrain | 45% |
61
+ | Datadog 5-minute+ observability metrics | 25% |
62
+ | GIFT-Eval train split | 15% |
63
+ | Synthetic (TempoPFN) | 10% |
64
+ | Datadog 10s observability metrics | 2.5% |
65
+ | Datadog 60s observability metrics | 2.5% |
66
+
67
+ The public portion (45% GIFT-Eval Pretrain) is drawn from the Toto 1.0 mix of GIFT-Eval Pretrain and the Chronos pretraining corpus, and is non-leaking with respect to the GIFT-Eval test split.
68
+
69
+ NorMuon and AdamW learning rates were both dropped by roughly an order of magnitude from pretraining (to 0.05 and 0.001 respectively). All other architecture and inference settings match the base 2.5B model.
70
+
71
+ ## ⚑ Quick Start
72
+
73
+ ```python
74
+ import torch
75
+ from toto2 import Toto2Model
76
+
77
+ model = Toto2Model.from_pretrained("Datadog/Toto-2.0-2.5B-FT")
78
+ model = model.to("cuda").eval()
79
+
80
+ # Same forecast() interface as the base 2.5B model.
81
+ ```
82
+
83
+ See the base [Toto-2.0-2.5B](https://huggingface.co/Datadog/Toto-2.0-2.5B) model card for the full inference example.
84
+
85
+ ## πŸ”— Additional Resources
86
+
87
+ - **Technical Report** β€” *(coming soon)*
88
+ - [Blog Post](https://www.datadoghq.com/blog/ai/toto-2/)
89
+ - [Base model: Toto-2.0-2.5B](https://huggingface.co/Datadog/Toto-2.0-2.5B) β€” the unfinetuned checkpoint, which is what we recommend deploying
90
+ - [Toto 2.0 Collection](https://huggingface.co/collections/Datadog/toto-20) β€” all five base sizes (4m β†’ 2.5B)
91
+ - [Toto 2.0 Family-and-Friends](https://huggingface.co/Datadog/Toto-2.0-Family-and-Friends) β€” companion FFORMA-ensemble submission, also benchmark-only
92
+ - [GIFT-Eval benchmark](https://huggingface.co/spaces/Salesforce/GIFT-Eval) β€” leaderboard hosting this submission
93
+ - [GitHub Repository](https://github.com/DataDog/toto)
94
+
95
+ ## πŸ“ License
96
+
97
+ Apache 2.0.
98
+
99
+ ## πŸ“– Citation
100
+
101
+ ```bibtex
102
+ (citation coming soon)
103
+ ```
assets/bar_metrics_gift_eval.png ADDED

Git LFS Details

  • SHA256: 9eef1afe0c18126a9e4813345ae4b0189539c61878b4e0d3428e1205bfe13c5e
  • Pointer size: 131 Bytes
  • Size of remote file: 602 kB