Upload folder using huggingface_hub

starcoder2-7b/base/adapter/README.md

@@ -0,0 +1,207 @@
+---
+base_model: bigcode/starcoder2-7b
+library_name: peft
+pipeline_tag: text-generation
+tags:
+- base_model:adapter:bigcode/starcoder2-7b
+- lora
+- transformers
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.18.1

starcoder2-7b/base/adapter/adapter_config.json

@@ -0,0 +1,46 @@
+{
+  "alora_invocation_tokens": null,
+  "alpha_pattern": {},
+  "arrow_config": null,
+  "auto_mapping": null,
+  "base_model_name_or_path": "bigcode/starcoder2-7b",
+  "bias": "none",
+  "corda_config": null,
+  "ensure_weight_tying": true,
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 32,
+  "lora_bias": false,
+  "lora_dropout": 0.05,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": [
+    "lm_head",
+    "embed_tokens"
+  ],
+  "peft_type": "LORA",
+  "peft_version": "0.18.1",
+  "qalora_group_size": 16,
+  "r": 16,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "v_proj",
+    "k_proj",
+    "q_proj",
+    "o_proj"
+  ],
+  "target_parameters": null,
+  "task_type": "CAUSAL_LM",
+  "trainable_token_indices": null,
+  "use_dora": false,
+  "use_qalora": false,
+  "use_rslora": false
+}

starcoder2-7b/base/adapter/adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e8722c41c3297c10da9786673bd5f7f3556116c882e76e37035eedd5ae393938
+size 2804312360

starcoder2-7b/base/audit_results.json

@@ -0,0 +1,137 @@
+{
+  "delta": 1e-05,
+  "num_canaries": 500,
+  "num_members": 250,
+  "paper_guess_fraction": 0.2,
+  "paper_guess_steps": 20,
+  "loss": {
+    "auc": 1.0,
+    "empirical_epsilon": {
+      "0.05": 3.4791953936219215,
+      "0.01": 3.023197554051876
+    },
+    "empirical_epsilon_details": {
+      "0.05": {
+        "epsilon": 3.4791953936219215,
+        "num_guesses": 100,
+        "correct_guesses": 100,
+        "candidate_num_guesses": [
+          5,
+          10,
+          15,
+          20,
+          25,
+          30,
+          35,
+          40,
+          45,
+          50,
+          55,
+          60,
+          65,
+          70,
+          75,
+          80,
+          85,
+          90,
+          95,
+          100
+        ],
+        "direction": "lower"
+      },
+      "0.01": {
+        "epsilon": 3.023197554051876,
+        "num_guesses": 100,
+        "correct_guesses": 100,
+        "candidate_num_guesses": [
+          5,
+          10,
+          15,
+          20,
+          25,
+          30,
+          35,
+          40,
+          45,
+          50,
+          55,
+          60,
+          65,
+          70,
+          75,
+          80,
+          85,
+          90,
+          95,
+          100
+        ],
+        "direction": "lower"
+      }
+    }
+  },
+  "embedding": {
+    "auc": 0.916224,
+    "empirical_epsilon": {
+      "0.05": 3.4791953936219215,
+      "0.01": 3.023197554051876
+    },
+    "empirical_epsilon_details": {
+      "0.05": {
+        "epsilon": 3.4791953936219215,
+        "num_guesses": 100,
+        "correct_guesses": 100,
+        "candidate_num_guesses": [
+          5,
+          10,
+          15,
+          20,
+          25,
+          30,
+          35,
+          40,
+          45,
+          50,
+          55,
+          60,
+          65,
+          70,
+          75,
+          80,
+          85,
+          90,
+          95,
+          100
+        ],
+        "direction": "lower"
+      },
+      "0.01": {
+        "epsilon": 3.023197554051876,
+        "num_guesses": 100,
+        "correct_guesses": 100,
+        "candidate_num_guesses": [
+          5,
+          10,
+          15,
+          20,
+          25,
+          30,
+          35,
+          40,
+          45,
+          50,
+          55,
+          60,
+          65,
+          70,
+          75,
+          80,
+          85,
+          90,
+          95,
+          100
+        ],
+        "direction": "lower"
+      }
+    }
+  }
+}

starcoder2-7b/base/audit_scores.npz

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:df7a9ad49c78c2b7e30773bfe7d47ce312de900740e940d72ac0b3117dff09bf
+size 12784

starcoder2-7b/base/codecarbon.csv

@@ -0,0 +1,2 @@
+timestamp,project_name,run_id,experiment_id,duration,emissions,emissions_rate,cpu_power,gpu_power,ram_power,cpu_energy,gpu_energy,ram_energy,energy_consumed,water_consumed,country_name,country_iso_code,region,cloud_provider,cloud_region,os,python_version,codecarbon_version,cpu_count,cpu_model,gpu_count,gpu_model,longitude,latitude,ram_total_size,tracking_mode,cpu_utilization_percent,gpu_utilization_percent,ram_utilization_percent,ram_used_gb,on_cloud,pue,wue
+2026-03-21T12:05:04,codedp-starcoder2-7b-cpt-base,499ee9b6-ab5e-4714-be7f-0b592eb40909,5b0fa12a-3dd7-45bb-9766-cc326314d9f1,3820.65162669681,0.5568149811716148,0.0001457382236267942,179.3439937196561,2316.658494927477,70.0,0.18368298514637418,2.457235179675422,0.07169108754482326,2.7126092523666183,0.0,United States,USA,california,,,Linux-5.15.0-157-generic-x86_64-with-glibc2.39,3.12.13,3.2.5,224,Intel(R) Xeon(R) Platinum 8480C,4,4 x NVIDIA H200,-121.9552,37.3541,2015.5625190734863,machine,2.2497114375655825,96.45330535152151,2.3057974816369358,46.78091819539025,N,1.0,0.0

starcoder2-7b/base/epochs/epoch_001/adapter/README.md

@@ -0,0 +1,207 @@
+---
+base_model: bigcode/starcoder2-7b
+library_name: peft
+pipeline_tag: text-generation
+tags:
+- base_model:adapter:bigcode/starcoder2-7b
+- lora
+- transformers
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.18.1

starcoder2-7b/base/epochs/epoch_001/adapter/adapter_config.json

@@ -0,0 +1,46 @@
+{
+  "alora_invocation_tokens": null,
+  "alpha_pattern": {},
+  "arrow_config": null,
+  "auto_mapping": null,
+  "base_model_name_or_path": "bigcode/starcoder2-7b",
+  "bias": "none",
+  "corda_config": null,
+  "ensure_weight_tying": true,
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 32,
+  "lora_bias": false,
+  "lora_dropout": 0.05,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": [
+    "lm_head",
+    "embed_tokens"
+  ],
+  "peft_type": "LORA",
+  "peft_version": "0.18.1",
+  "qalora_group_size": 16,
+  "r": 16,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "v_proj",
+    "k_proj",
+    "q_proj",
+    "o_proj"
+  ],
+  "target_parameters": null,
+  "task_type": "CAUSAL_LM",
+  "trainable_token_indices": null,
+  "use_dora": false,
+  "use_qalora": false,
+  "use_rslora": false
+}

starcoder2-7b/base/epochs/epoch_001/adapter/adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8ab7c236acdfacb363251d6f1c092cab8b23cfee5381584cd586426a1ecf16d0
+size 2804312360

starcoder2-7b/base/epochs/epoch_001/audit_results.json

@@ -0,0 +1,137 @@
+{
+  "delta": 1e-05,
+  "num_canaries": 500,
+  "num_members": 250,
+  "paper_guess_fraction": 0.2,
+  "paper_guess_steps": 20,
+  "loss": {
+    "auc": 0.998072,
+    "empirical_epsilon": {
+      "0.05": 3.4791953936219215,
+      "0.01": 3.023197554051876
+    },
+    "empirical_epsilon_details": {
+      "0.05": {
+        "epsilon": 3.4791953936219215,
+        "num_guesses": 100,
+        "correct_guesses": 100,
+        "candidate_num_guesses": [
+          5,
+          10,
+          15,
+          20,
+          25,
+          30,
+          35,
+          40,
+          45,
+          50,
+          55,
+          60,
+          65,
+          70,
+          75,
+          80,
+          85,
+          90,
+          95,
+          100
+        ],
+        "direction": "lower"
+      },
+      "0.01": {
+        "epsilon": 3.023197554051876,
+        "num_guesses": 100,
+        "correct_guesses": 100,
+        "candidate_num_guesses": [
+          5,
+          10,
+          15,
+          20,
+          25,
+          30,
+          35,
+          40,
+          45,
+          50,
+          55,
+          60,
+          65,
+          70,
+          75,
+          80,
+          85,
+          90,
+          95,
+          100
+        ],
+        "direction": "lower"
+      }
+    }
+  },
+  "embedding": {
+    "auc": 0.96,
+    "empirical_epsilon": {
+      "0.05": 3.4791953936219215,
+      "0.01": 3.023197554051876
+    },
+    "empirical_epsilon_details": {
+      "0.05": {
+        "epsilon": 3.4791953936219215,
+        "num_guesses": 100,
+        "correct_guesses": 100,
+        "candidate_num_guesses": [
+          5,
+          10,
+          15,
+          20,
+          25,
+          30,
+          35,
+          40,
+          45,
+          50,
+          55,
+          60,
+          65,
+          70,
+          75,
+          80,
+          85,
+          90,
+          95,
+          100
+        ],
+        "direction": "lower"
+      },
+      "0.01": {
+        "epsilon": 3.023197554051876,
+        "num_guesses": 100,
+        "correct_guesses": 100,
+        "candidate_num_guesses": [
+          5,
+          10,
+          15,
+          20,
+          25,
+          30,
+          35,
+          40,
+          45,
+          50,
+          55,
+          60,
+          65,
+          70,
+          75,
+          80,
+          85,
+          90,
+          95,
+          100
+        ],
+        "direction": "lower"
+      }
+    }
+  }
+}

starcoder2-7b/base/epochs/epoch_001/audit_scores.npz

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ad12d1a5732cbbe84adb0a8ad0d9063e66354e84483f65f8d0ad9cbad178b42f
+size 12784

starcoder2-7b/base/epochs/epoch_002/adapter/README.md

@@ -0,0 +1,207 @@
+---
+base_model: bigcode/starcoder2-7b
+library_name: peft
+pipeline_tag: text-generation
+tags:
+- base_model:adapter:bigcode/starcoder2-7b
+- lora
+- transformers
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.18.1

starcoder2-7b/base/epochs/epoch_002/adapter/adapter_config.json

@@ -0,0 +1,46 @@
+{
+  "alora_invocation_tokens": null,
+  "alpha_pattern": {},
+  "arrow_config": null,
+  "auto_mapping": null,
+  "base_model_name_or_path": "bigcode/starcoder2-7b",
+  "bias": "none",
+  "corda_config": null,
+  "ensure_weight_tying": true,
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 32,
+  "lora_bias": false,
+  "lora_dropout": 0.05,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": [
+    "lm_head",
+    "embed_tokens"
+  ],
+  "peft_type": "LORA",
+  "peft_version": "0.18.1",
+  "qalora_group_size": 16,
+  "r": 16,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "v_proj",
+    "k_proj",
+    "q_proj",
+    "o_proj"
+  ],
+  "target_parameters": null,
+  "task_type": "CAUSAL_LM",
+  "trainable_token_indices": null,
+  "use_dora": false,
+  "use_qalora": false,
+  "use_rslora": false
+}

starcoder2-7b/base/epochs/epoch_002/adapter/adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e8722c41c3297c10da9786673bd5f7f3556116c882e76e37035eedd5ae393938
+size 2804312360

starcoder2-7b/base/epochs/epoch_002/audit_results.json

@@ -0,0 +1,137 @@
+{
+  "delta": 1e-05,
+  "num_canaries": 500,
+  "num_members": 250,
+  "paper_guess_fraction": 0.2,
+  "paper_guess_steps": 20,
+  "loss": {
+    "auc": 1.0,
+    "empirical_epsilon": {
+      "0.05": 3.4791953936219215,
+      "0.01": 3.023197554051876
+    },
+    "empirical_epsilon_details": {
+      "0.05": {
+        "epsilon": 3.4791953936219215,
+        "num_guesses": 100,
+        "correct_guesses": 100,
+        "candidate_num_guesses": [
+          5,
+          10,
+          15,
+          20,
+          25,
+          30,
+          35,
+          40,
+          45,
+          50,
+          55,
+          60,
+          65,
+          70,
+          75,
+          80,
+          85,
+          90,
+          95,
+          100
+        ],
+        "direction": "lower"
+      },
+      "0.01": {
+        "epsilon": 3.023197554051876,
+        "num_guesses": 100,
+        "correct_guesses": 100,
+        "candidate_num_guesses": [
+          5,
+          10,
+          15,
+          20,
+          25,
+          30,
+          35,
+          40,
+          45,
+          50,
+          55,
+          60,
+          65,
+          70,
+          75,
+          80,
+          85,
+          90,
+          95,
+          100
+        ],
+        "direction": "lower"
+      }
+    }
+  },
+  "embedding": {
+    "auc": 0.916224,
+    "empirical_epsilon": {
+      "0.05": 3.4791953936219215,
+      "0.01": 3.023197554051876
+    },
+    "empirical_epsilon_details": {
+      "0.05": {
+        "epsilon": 3.4791953936219215,
+        "num_guesses": 100,
+        "correct_guesses": 100,
+        "candidate_num_guesses": [
+          5,
+          10,
+          15,
+          20,
+          25,
+          30,
+          35,
+          40,
+          45,
+          50,
+          55,
+          60,
+          65,
+          70,
+          75,
+          80,
+          85,
+          90,
+          95,
+          100
+        ],
+        "direction": "lower"
+      },
+      "0.01": {
+        "epsilon": 3.023197554051876,
+        "num_guesses": 100,
+        "correct_guesses": 100,
+        "candidate_num_guesses": [
+          5,
+          10,
+          15,
+          20,
+          25,
+          30,
+          35,
+          40,
+          45,
+          50,
+          55,
+          60,
+          65,
+          70,
+          75,
+          80,
+          85,
+          90,
+          95,
+          100
+        ],
+        "direction": "lower"
+      }
+    }
+  }
+}

starcoder2-7b/base/epochs/epoch_002/audit_scores.npz

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:df7a9ad49c78c2b7e30773bfe7d47ce312de900740e940d72ac0b3117dff09bf
+size 12784

starcoder2-7b/base/metrics.jsonl

@@ -0,0 +1,55 @@
+{"timestamp": 1774091071.9870658, "event": "train_step", "step": 10, "epoch": 1, "metrics": {"train/step_loss": 1.5507386380975896, "train/step_real_loss": 1.2866992354393005, "train/lr": 4.761904761904762e-05, "train/step_canary_loss": 10.0, "perf/step_duration_sec": 8.244300998747349, "perf/samples_per_sec": 8.00553012438873, "perf/tokens_per_sec": 6440.812872803661, "perf/logical_batch_size": 66.0, "perf/logical_token_count": 53100.0, "perf/gradient_accumulation_steps": 8.0, "system/cuda_memory_allocated_gb": 17.816345691680908, "system/cuda_max_memory_allocated_gb": 82.16860580444336}}
+{"timestamp": 1774091154.380765, "event": "train_step", "step": 20, "epoch": 1, "metrics": {"train/step_loss": 1.4504926845208923, "train/step_real_loss": 1.2006134390830994, "train/lr": 9.523809523809524e-05, "train/step_canary_loss": 6.78125, "perf/step_duration_sec": 8.244972918182611, "perf/samples_per_sec": 8.126163744242886, "perf/tokens_per_sec": 6422.095078472549, "perf/logical_batch_size": 67.0, "perf/logical_token_count": 52950.0, "perf/gradient_accumulation_steps": 8.0, "system/cuda_memory_allocated_gb": 17.911073684692383, "system/cuda_max_memory_allocated_gb": 82.16860580444336}}
+{"timestamp": 1774091235.6533117, "event": "train_step", "step": 30, "epoch": 1, "metrics": {"train/step_loss": 0.9861254319548607, "train/step_real_loss": 0.9861254319548607, "train/lr": 9.98706541985615e-05, "perf/step_duration_sec": 7.973596462979913, "perf/samples_per_sec": 8.026490968929942, "perf/tokens_per_sec": 7148.9697609673985, "perf/logical_batch_size": 64.0, "perf/logical_token_count": 57003.0, "perf/gradient_accumulation_steps": 8.0, "system/cuda_memory_allocated_gb": 17.816345691680908, "system/cuda_max_memory_allocated_gb": 82.16860580444336}}
+{"timestamp": 1774091318.662035, "event": "train_step", "step": 40, "epoch": 1, "metrics": {"train/step_loss": 1.0557814985513687, "train/step_real_loss": 1.0557814985513687, "train/lr": 9.942439201095397e-05, "perf/step_duration_sec": 8.119743634015322, "perf/samples_per_sec": 7.882022251526572, "perf/tokens_per_sec": 6340.347961767047, "perf/logical_batch_size": 64.0, "perf/logical_token_count": 51482.0, "perf/gradient_accumulation_steps": 8.0, "system/cuda_memory_allocated_gb": 17.816345691680908, "system/cuda_max_memory_allocated_gb": 88.38552379608154}}
+{"timestamp": 1774091402.0365882, "event": "train_step", "step": 50, "epoch": 1, "metrics": {"train/step_loss": 1.017115435233483, "train/step_real_loss": 0.9500000476837158, "train/lr": 9.866246608261724e-05, "train/step_canary_loss": 5.3125, "perf/step_duration_sec": 8.119036318734288, "perf/samples_per_sec": 8.005876245437603, "perf/tokens_per_sec": 6307.645142790008, "perf/logical_batch_size": 65.0, "perf/logical_token_count": 51212.0, "perf/gradient_accumulation_steps": 8.0, "system/cuda_memory_allocated_gb": 17.816345691680908, "system/cuda_max_memory_allocated_gb": 88.38552379608154}}
+{"timestamp": 1774091429.1958544, "event": "eval_step", "step": 50, "epoch": 1, "metrics": {"eval/loss": 0.8033588177513802, "eval/duration_sec": 27.156014366075397}}
+{"timestamp": 1774091511.9092546, "event": "train_step", "step": 60, "epoch": 1, "metrics": {"train/step_loss": 1.2237938749256418, "train/step_real_loss": 0.9667061120271683, "train/lr": 9.7589742682592e-05, "train/step_canary_loss": 6.708333333333333, "perf/step_duration_sec": 8.495171755552292, "perf/samples_per_sec": 7.886832889071372, "perf/tokens_per_sec": 6228.008276044599, "perf/logical_batch_size": 67.0, "perf/logical_token_count": 52908.0, "perf/gradient_accumulation_steps": 8.0, "system/cuda_memory_allocated_gb": 17.816345691680908, "system/cuda_max_memory_allocated_gb": 88.38552379608154}}
+{"timestamp": 1774091593.5344725, "event": "train_step", "step": 70, "epoch": 1, "metrics": {"train/step_loss": 1.0743033794256358, "train/step_real_loss": 0.9387456253170967, "train/lr": 9.621307308142384e-05, "train/step_canary_loss": 9.75, "perf/step_duration_sec": 8.257235972210765, "perf/samples_per_sec": 7.8718835478062665, "perf/tokens_per_sec": 6568.057420488078, "perf/logical_batch_size": 65.0, "perf/logical_token_count": 54234.0, "perf/gradient_accumulation_steps": 8.0, "system/cuda_memory_allocated_gb": 17.816345691680908, "system/cuda_max_memory_allocated_gb": 88.38552379608154}}
+{"timestamp": 1774091676.5485113, "event": "train_step", "step": 80, "epoch": 1, "metrics": {"train/step_loss": 0.8626528829336166, "train/step_real_loss": 0.8626528829336166, "train/lr": 9.454124979346391e-05, "perf/step_duration_sec": 8.386640733107924, "perf/samples_per_sec": 7.631184169765056, "perf/tokens_per_sec": 6658.446662625317, "perf/logical_batch_size": 64.0, "perf/logical_token_count": 55842.0, "perf/gradient_accumulation_steps": 8.0, "system/cuda_memory_allocated_gb": 17.816345691680908, "system/cuda_max_memory_allocated_gb": 88.38552379608154}}
+{"timestamp": 1774091757.879373, "event": "train_step", "step": 90, "epoch": 1, "metrics": {"train/step_loss": 0.9340124875307083, "train/step_real_loss": 0.9340124875307083, "train/lr": 9.258495042083221e-05, "perf/step_duration_sec": 8.493970146402717, "perf/samples_per_sec": 7.534756880103312, "perf/tokens_per_sec": 5785.162786428072, "perf/logical_batch_size": 64.0, "perf/logical_token_count": 49139.0, "perf/gradient_accumulation_steps": 8.0, "system/cuda_memory_allocated_gb": 17.816345691680908, "system/cuda_max_memory_allocated_gb": 88.38552379608154}}
+{"timestamp": 1774091839.6918592, "event": "train_step", "step": 100, "epoch": 1, "metrics": {"train/step_loss": 0.9577739440477812, "train/step_real_loss": 0.880453996360302, "train/lr": 9.035666945770107e-05, "train/step_canary_loss": 5.90625, "perf/step_duration_sec": 8.234961070120335, "perf/samples_per_sec": 7.893176354633353, "perf/tokens_per_sec": 6771.616711381143, "perf/logical_batch_size": 65.0, "perf/logical_token_count": 55764.0, "perf/gradient_accumulation_steps": 8.0, "system/cuda_memory_allocated_gb": 17.816345691680908, "system/cuda_max_memory_allocated_gb": 88.38552379608154}}
+{"timestamp": 1774091866.8768253, "event": "eval_step", "step": 100, "epoch": 1, "metrics": {"eval/loss": 0.7754902612717671, "eval/duration_sec": 27.182465851306915}}
+{"timestamp": 1774091950.1670244, "event": "train_step", "step": 110, "epoch": 1, "metrics": {"train/step_loss": 0.9567192720644402, "train/step_real_loss": 0.860762245953083, "train/lr": 8.787063849045118e-05, "train/step_canary_loss": 4.02734375, "perf/step_duration_sec": 8.036363879218698, "perf/samples_per_sec": 8.21266943507498, "perf/tokens_per_sec": 6478.054102878841, "perf/logical_batch_size": 66.0, "perf/logical_token_count": 52060.0, "perf/gradient_accumulation_steps": 8.0, "system/cuda_memory_allocated_gb": 17.816345691680908, "system/cuda_max_memory_allocated_gb": 88.38552379608154}}
+{"timestamp": 1774092031.8751383, "event": "train_step", "step": 120, "epoch": 1, "metrics": {"train/step_loss": 0.9170611575245857, "train/step_real_loss": 0.9170611575245857, "train/lr": 8.5142735303366e-05, "perf/step_duration_sec": 7.79720464348793, "perf/samples_per_sec": 8.208069805305357, "perf/tokens_per_sec": 6600.4423832818775, "perf/logical_batch_size": 64.0, "perf/logical_token_count": 51465.0, "perf/gradient_accumulation_steps": 8.0, "system/cuda_memory_allocated_gb": 17.816345691680908, "system/cuda_max_memory_allocated_gb": 88.38552379608154}}
+{"timestamp": 1774092114.4179726, "event": "train_step", "step": 130, "epoch": 1, "metrics": {"train/step_loss": 0.9158061426697355, "train/step_real_loss": 0.8711828961968422, "train/lr": 8.219038247038819e-05, "train/step_canary_loss": 2.34375, "perf/step_duration_sec": 8.22744246572256, "perf/samples_per_sec": 8.021933945447975, "perf/tokens_per_sec": 6342.311139506375, "perf/logical_batch_size": 66.0, "perf/logical_token_count": 52181.0, "perf/gradient_accumulation_steps": 8.0, "system/cuda_memory_allocated_gb": 17.816345691680908, "system/cuda_max_memory_allocated_gb": 88.38552379608154}}
+{"timestamp": 1774092196.3902268, "event": "train_step", "step": 140, "epoch": 1, "metrics": {"train/step_loss": 0.9856266433542425, "train/step_real_loss": 0.8489470109343529, "train/lr": 7.903243608061246e-05, "train/step_canary_loss": 5.359375, "perf/step_duration_sec": 8.333684524521232, "perf/samples_per_sec": 7.9196662419605675, "perf/tokens_per_sec": 6461.007714123103, "perf/logical_batch_size": 66.0, "perf/logical_token_count": 53844.0, "perf/gradient_accumulation_steps": 8.0, "system/cuda_memory_allocated_gb": 17.816345691680908, "system/cuda_max_memory_allocated_gb": 88.38552379608154}}
+{"timestamp": 1774092278.982639, "event": "train_step", "step": 150, "epoch": 1, "metrics": {"train/step_loss": 0.94221530854702, "train/step_real_loss": 0.94221530854702, "train/lr": 7.568906530820282e-05, "perf/step_duration_sec": 8.388992108404636, "perf/samples_per_sec": 7.629045202686584, "perf/tokens_per_sec": 6081.421861022835, "perf/logical_batch_size": 64.0, "perf/logical_token_count": 51017.0, "perf/gradient_accumulation_steps": 8.0, "system/cuda_memory_allocated_gb": 17.816345691680908, "system/cuda_max_memory_allocated_gb": 88.38552379608154}}
+{"timestamp": 1774092306.1516783, "event": "eval_step", "step": 150, "epoch": 1, "metrics": {"eval/loss": 0.7622188641422096, "eval/duration_sec": 27.166858203709126}}
+{"timestamp": 1774092388.1349308, "event": "train_step", "step": 160, "epoch": 1, "metrics": {"train/step_loss": 0.9667639210820198, "train/step_real_loss": 0.9667639210820198, "train/lr": 7.21816235958972e-05, "perf/step_duration_sec": 8.235355000942945, "perf/samples_per_sec": 7.771371117902267, "perf/tokens_per_sec": 6008.24129552819, "perf/logical_batch_size": 64.0, "perf/logical_token_count": 49480.0, "perf/gradient_accumulation_steps": 8.0, "system/cuda_memory_allocated_gb": 17.816345691680908, "system/cuda_max_memory_allocated_gb": 88.38552379608154}}
+{"timestamp": 1774092469.9282126, "event": "train_step", "step": 170, "epoch": 1, "metrics": {"train/step_loss": 0.9262635037302971, "train/step_real_loss": 0.9262635037302971, "train/lr": 6.853251227482479e-05, "perf/step_duration_sec": 7.9527543764561415, "perf/samples_per_sec": 8.047526299752175, "perf/tokens_per_sec": 6280.842791759705, "perf/logical_batch_size": 64.0, "perf/logical_token_count": 49950.0, "perf/gradient_accumulation_steps": 8.0, "system/cuda_memory_allocated_gb": 17.816345691680908, "system/cuda_max_memory_allocated_gb": 88.38552379608154}}
+{"timestamp": 1774092551.943189, "event": "train_step", "step": 180, "epoch": 1, "metrics": {"train/step_loss": 0.8705217910535408, "train/step_real_loss": 0.8467002063989639, "train/lr": 6.476503749166904e-05, "train/step_canary_loss": 1.6328125, "perf/step_duration_sec": 8.540382800623775, "perf/samples_per_sec": 7.7279908337574135, "perf/tokens_per_sec": 6385.076790236751, "perf/logical_batch_size": 66.0, "perf/logical_token_count": 54531.0, "perf/gradient_accumulation_steps": 8.0, "system/cuda_memory_allocated_gb": 17.816345691680908, "system/cuda_max_memory_allocated_gb": 88.38552379608154}}
+{"timestamp": 1774092634.1860914, "event": "train_step", "step": 190, "epoch": 1, "metrics": {"train/step_loss": 0.8329041105050307, "train/step_real_loss": 0.830781526863575, "train/lr": 6.090326135695403e-05, "train/step_canary_loss": 0.96875, "perf/step_duration_sec": 7.938739079982042, "perf/samples_per_sec": 8.18769824088324, "perf/tokens_per_sec": 6963.07051322375, "perf/logical_batch_size": 65.0, "perf/logical_token_count": 55278.0, "perf/gradient_accumulation_steps": 8.0, "system/cuda_memory_allocated_gb": 17.816345691680908, "system/cuda_max_memory_allocated_gb": 88.38552379608154}}
+{"timestamp": 1774092717.69117, "event": "train_step", "step": 200, "epoch": 1, "metrics": {"train/step_loss": 0.9225348509274997, "train/step_real_loss": 0.8622424304485321, "train/lr": 5.697184826514057e-05, "train/step_canary_loss": 4.78125, "perf/step_duration_sec": 8.257816776633263, "perf/samples_per_sec": 7.871329887571168, "perf/tokens_per_sec": 6772.4922352662325, "perf/logical_batch_size": 65.0, "perf/logical_token_count": 55926.0, "perf/gradient_accumulation_steps": 8.0, "system/cuda_memory_allocated_gb": 17.816345691680908, "system/cuda_max_memory_allocated_gb": 88.38552379608154}}
+{"timestamp": 1774092744.8800561, "event": "eval_step", "step": 200, "epoch": 1, "metrics": {"eval/loss": 0.7524948388166811, "eval/duration_sec": 27.18629032932222}}
+{"timestamp": 1774092830.2611096, "event": "train_epoch", "step": 207, "epoch": 1, "metrics": {"train/epoch_loss": 1.036294524908722, "train/epoch_real_loss": 0.957591299003593, "train/epoch_canary_loss": 5.154934052784185, "perf/epoch_duration_sec": 1814.8832574244589, "perf/epoch_samples_per_sec": 29.762244915220545, "perf/epoch_tokens_per_sec": 24175.732417227537, "perf/epoch_samples": 54015.0, "perf/epoch_tokens": 43876132.0, "system/cuda_epoch_peak_memory_gb": 88.38552379608154, "eval/loss": 0.7516984458284817, "eval/duration_sec": 27.281295781955123}}
+{"timestamp": 1774092840.935869, "event": "audit_epoch", "step": 207, "epoch": 1, "metrics": {"audit/delta": 1e-05, "audit/num_canaries": 500.0, "audit/num_members": 250.0, "audit/paper_guess_fraction": 0.2, "audit/paper_guess_steps": 20.0, "audit/loss/auc": 0.998072, "audit/loss/empirical_epsilon/0.05": 3.4791953936219215, "audit/loss/empirical_epsilon/0.01": 3.023197554051876, "audit/loss/empirical_epsilon_details/0.05/epsilon": 3.4791953936219215, "audit/loss/empirical_epsilon_details/0.05/num_guesses": 100.0, "audit/loss/empirical_epsilon_details/0.05/correct_guesses": 100.0, "audit/loss/empirical_epsilon_details/0.01/epsilon": 3.023197554051876, "audit/loss/empirical_epsilon_details/0.01/num_guesses": 100.0, "audit/loss/empirical_epsilon_details/0.01/correct_guesses": 100.0, "audit/embedding/auc": 0.96, "audit/embedding/empirical_epsilon/0.05": 3.4791953936219215, "audit/embedding/empirical_epsilon/0.01": 3.023197554051876, "audit/embedding/empirical_epsilon_details/0.05/epsilon": 3.4791953936219215, "audit/embedding/empirical_epsilon_details/0.05/num_guesses": 100.0, "audit/embedding/empirical_epsilon_details/0.05/correct_guesses": 100.0, "audit/embedding/empirical_epsilon_details/0.01/epsilon": 3.023197554051876, "audit/embedding/empirical_epsilon_details/0.01/num_guesses": 100.0, "audit/embedding/empirical_epsilon_details/0.01/correct_guesses": 100.0, "perf/audit_duration_sec": 6.3129072319716215}}
+{"timestamp": 1774092866.0795767, "event": "train_step", "step": 210, "epoch": 2, "metrics": {"train/step_loss": 0.8786039505944108, "train/step_real_loss": 0.8291560411453247, "train/lr": 5.29959073680547e-05, "train/step_canary_loss": 2.4609375, "perf/step_duration_sec": 8.105620637536049, "perf/samples_per_sec": 8.142498020985931, "perf/tokens_per_sec": 6755.805933775599, "perf/logical_batch_size": 66.0, "perf/logical_token_count": 54760.0, "perf/gradient_accumulation_steps": 8.0, "system/cuda_memory_allocated_gb": 17.911073684692383, "system/cuda_max_memory_allocated_gb": 75.98184061050415}}
+{"timestamp": 1774092947.7881637, "event": "train_step", "step": 220, "epoch": 2, "metrics": {"train/step_loss": 0.8338463682394761, "train/step_real_loss": 0.8286867365241051, "train/lr": 4.9000832207739676e-05, "train/step_canary_loss": 1.1640625, "perf/step_duration_sec": 8.100939376279712, "perf/samples_per_sec": 8.023760823383757, "perf/tokens_per_sec": 6522.083124668936, "perf/logical_batch_size": 65.0, "perf/logical_token_count": 52835.0, "perf/gradient_accumulation_steps": 8.0, "system/cuda_memory_allocated_gb": 17.816345691680908, "system/cuda_max_memory_allocated_gb": 82.1704511642456}}
+{"timestamp": 1774093029.3670137, "event": "train_step", "step": 230, "epoch": 2, "metrics": {"train/step_loss": 0.7926744809517494, "train/step_real_loss": 0.8034120723605156, "train/lr": 4.501213853296425e-05, "train/step_canary_loss": 0.10546875, "perf/step_duration_sec": 8.09845926053822, "perf/samples_per_sec": 8.026218063073905, "perf/tokens_per_sec": 6708.683497950849, "perf/logical_batch_size": 65.0, "perf/logical_token_count": 54330.0, "perf/gradient_accumulation_steps": 8.0, "system/cuda_memory_allocated_gb": 17.816345691680908, "system/cuda_max_memory_allocated_gb": 82.1704511642456}}
+{"timestamp": 1774093111.4853563, "event": "train_step", "step": 240, "epoch": 2, "metrics": {"train/step_loss": 0.8129148510369387, "train/step_real_loss": 0.7906041964888573, "train/lr": 4.1055301335220955e-05, "train/step_canary_loss": 1.52685546875, "perf/step_duration_sec": 8.505769751966, "perf/samples_per_sec": 7.75943881913156, "perf/tokens_per_sec": 6383.901937558238, "perf/logical_batch_size": 66.0, "perf/logical_token_count": 54300.0, "perf/gradient_accumulation_steps": 8.0, "system/cuda_memory_allocated_gb": 17.816345691680908, "system/cuda_max_memory_allocated_gb": 82.1704511642456}}
+{"timestamp": 1774093192.88336, "event": "train_step", "step": 250, "epoch": 2, "metrics": {"train/step_loss": 0.8410789425556476, "train/step_real_loss": 0.850802831351757, "train/lr": 3.715559214503298e-05, "train/step_canary_loss": 0.21875, "perf/step_duration_sec": 8.232771871611476, "perf/samples_per_sec": 7.895275250385015, "perf/tokens_per_sec": 6654.745309889904, "perf/logical_batch_size": 65.0, "perf/logical_token_count": 54787.0, "perf/gradient_accumulation_steps": 8.0, "system/cuda_memory_allocated_gb": 17.816345691680908, "system/cuda_max_memory_allocated_gb": 82.1704511642456}}
+{"timestamp": 1774093220.0732572, "event": "eval_step", "step": 250, "epoch": 2, "metrics": {"eval/loss": 0.748522541389383, "eval/duration_sec": 27.187080297619104}}
+{"timestamp": 1774093302.2765276, "event": "train_step", "step": 260, "epoch": 2, "metrics": {"train/step_loss": 0.8724462010643699, "train/step_real_loss": 0.8488068357110023, "train/lr": 3.33379176277258e-05, "train/step_canary_loss": 1.62890625, "perf/step_duration_sec": 8.470736568793654, "perf/samples_per_sec": 7.7915302245551095, "perf/tokens_per_sec": 6277.49423773815, "perf/logical_batch_size": 66.0, "perf/logical_token_count": 53175.0, "perf/gradient_accumulation_steps": 8.0, "system/cuda_memory_allocated_gb": 17.911073684692383, "system/cuda_max_memory_allocated_gb": 82.1704511642456}}
+{"timestamp": 1774093384.391679, "event": "train_step", "step": 270, "epoch": 2, "metrics": {"train/step_loss": 0.8640658855438232, "train/step_real_loss": 0.7926059737801552, "train/lr": 2.962666050951997e-05, "train/step_canary_loss": 5.4375, "perf/step_duration_sec": 7.947357261553407, "perf/samples_per_sec": 8.178819431516908, "perf/tokens_per_sec": 6407.161314659091, "perf/logical_batch_size": 65.0, "perf/logical_token_count": 50920.0, "perf/gradient_accumulation_steps": 8.0, "system/cuda_memory_allocated_gb": 17.816345691680908, "system/cuda_max_memory_allocated_gb": 82.1704511642456}}
+{"timestamp": 1774093468.0585487, "event": "train_step", "step": 280, "epoch": 2, "metrics": {"train/step_loss": 0.7910019425245431, "train/step_real_loss": 0.8018507286906242, "train/lr": 2.604552384991855e-05, "train/step_canary_loss": 0.0966796875, "perf/step_duration_sec": 8.62639987282455, "perf/samples_per_sec": 7.53500892124967, "perf/tokens_per_sec": 6275.039506402552, "perf/logical_batch_size": 65.0, "perf/logical_token_count": 54131.0, "perf/gradient_accumulation_steps": 8.0, "system/cuda_memory_allocated_gb": 17.816345691680908, "system/cuda_max_memory_allocated_gb": 82.1704511642456}}
+{"timestamp": 1774093550.7132335, "event": "train_step", "step": 290, "epoch": 2, "metrics": {"train/step_loss": 0.8430270507480159, "train/step_real_loss": 0.8613760396838188, "train/lr": 2.2617379654990623e-05, "train/step_canary_loss": 0.255859375, "perf/step_duration_sec": 8.335046991705894, "perf/samples_per_sec": 7.9183716739300705, "perf/tokens_per_sec": 6261.392413496007, "perf/logical_batch_size": 66.0, "perf/logical_token_count": 52189.0, "perf/gradient_accumulation_steps": 8.0, "system/cuda_memory_allocated_gb": 17.816345691680908, "system/cuda_max_memory_allocated_gb": 82.1704511642456}}
+{"timestamp": 1774093632.5524132, "event": "train_step", "step": 300, "epoch": 2, "metrics": {"train/step_loss": 0.7691241015087474, "train/step_real_loss": 0.784599058330059, "train/lr": 1.936412279842705e-05, "train/step_canary_loss": 0.27392578125, "perf/step_duration_sec": 8.091827008873224, "perf/samples_per_sec": 8.156378025336753, "perf/tokens_per_sec": 5984.062677922065, "perf/logical_batch_size": 66.0, "perf/logical_token_count": 48422.0, "perf/gradient_accumulation_steps": 8.0, "system/cuda_memory_allocated_gb": 17.911073684692383, "system/cuda_max_memory_allocated_gb": 82.1704511642456}}
+{"timestamp": 1774093659.719083, "event": "eval_step", "step": 300, "epoch": 2, "metrics": {"eval/loss": 0.7462242173737493, "eval/duration_sec": 27.16416385397315}}
+{"timestamp": 1774093742.5612013, "event": "train_step", "step": 310, "epoch": 2, "metrics": {"train/step_loss": 0.7514625552928809, "train/step_real_loss": 0.7710395082831383, "train/lr": 1.6306531183346385e-05, "train/step_canary_loss": 0.125, "perf/step_duration_sec": 8.240195842459798, "perf/samples_per_sec": 8.009518373327666, "perf/tokens_per_sec": 6369.751520897312, "perf/logical_batch_size": 66.0, "perf/logical_token_count": 52488.0, "perf/gradient_accumulation_steps": 8.0, "system/cuda_memory_allocated_gb": 17.816345691680908, "system/cuda_max_memory_allocated_gb": 82.1704511642456}}
+{"timestamp": 1774093824.76255, "event": "train_step", "step": 320, "epoch": 2, "metrics": {"train/step_loss": 0.8317307600608239, "train/step_real_loss": 0.8416290208697319, "train/lr": 1.3464133037968912e-05, "train/step_canary_loss": 0.1982421875, "perf/step_duration_sec": 7.953533122316003, "perf/samples_per_sec": 8.172468637569782, "perf/tokens_per_sec": 6226.666720107783, "perf/logical_batch_size": 65.0, "perf/logical_token_count": 49524.0, "perf/gradient_accumulation_steps": 8.0, "system/cuda_memory_allocated_gb": 17.816345691680908, "system/cuda_max_memory_allocated_gb": 82.1704511642456}}
+{"timestamp": 1774093907.919859, "event": "train_step", "step": 330, "epoch": 2, "metrics": {"train/step_loss": 0.8719267258277306, "train/step_real_loss": 0.8635779246687889, "train/lr": 1.0855082192715294e-05, "train/step_canary_loss": 1.40625, "perf/step_duration_sec": 8.101611092686653, "perf/samples_per_sec": 8.023095561656334, "perf/tokens_per_sec": 5623.572827524039, "perf/logical_batch_size": 65.0, "perf/logical_token_count": 45560.0, "perf/gradient_accumulation_steps": 8.0, "system/cuda_memory_allocated_gb": 17.816345691680908, "system/cuda_max_memory_allocated_gb": 82.1704511642456}}
+{"timestamp": 1774093989.7239172, "event": "train_step", "step": 340, "epoch": 2, "metrics": {"train/step_loss": 0.8476224361043988, "train/step_real_loss": 0.8228105828166008, "train/lr": 8.49604213531004e-06, "train/step_canary_loss": 1.6416015625, "perf/step_duration_sec": 8.274557610973716, "perf/samples_per_sec": 7.976257233676254, "perf/tokens_per_sec": 6674.435371234426, "perf/logical_batch_size": 66.0, "perf/logical_token_count": 55228.0, "perf/gradient_accumulation_steps": 8.0, "system/cuda_memory_allocated_gb": 17.816345691680908, "system/cuda_max_memory_allocated_gb": 82.1704511642456}}
+{"timestamp": 1774094071.9116077, "event": "train_step", "step": 350, "epoch": 2, "metrics": {"train/step_loss": 0.8573094416355741, "train/step_real_loss": 0.816773310303688, "train/lr": 6.402079584406673e-06, "train/step_canary_loss": 1.376171875, "perf/step_duration_sec": 8.637357847765088, "perf/samples_per_sec": 7.988554048140279, "perf/tokens_per_sec": 6212.547974249382, "perf/logical_batch_size": 69.0, "perf/logical_token_count": 53660.0, "perf/gradient_accumulation_steps": 8.0, "system/cuda_memory_allocated_gb": 17.816345691680908, "system/cuda_max_memory_allocated_gb": 82.1704511642456}}
+{"timestamp": 1774094099.091823, "event": "eval_step", "step": 350, "epoch": 2, "metrics": {"eval/loss": 0.7456388631942629, "eval/duration_sec": 27.17722495831549}}
+{"timestamp": 1774094181.5654633, "event": "train_step", "step": 360, "epoch": 2, "metrics": {"train/step_loss": 0.8446246770712046, "train/step_real_loss": 0.8453097268939018, "train/lr": 4.586568261458729e-06, "train/step_canary_loss": 0.80078125, "perf/step_duration_sec": 7.976690696552396, "perf/samples_per_sec": 8.148742689508275, "perf/tokens_per_sec": 6948.119478163339, "perf/logical_batch_size": 65.0, "perf/logical_token_count": 55423.0, "perf/gradient_accumulation_steps": 8.0, "system/cuda_memory_allocated_gb": 17.816345691680908, "system/cuda_max_memory_allocated_gb": 82.1704511642456}}
+{"timestamp": 1774094263.5117958, "event": "train_step", "step": 370, "epoch": 2, "metrics": {"train/step_loss": 0.8721330687403679, "train/step_real_loss": 0.8721330687403679, "train/lr": 3.06110347542643e-06, "perf/step_duration_sec": 8.37641635723412, "perf/samples_per_sec": 7.640498904370688, "perf/tokens_per_sec": 5830.058812420977, "perf/logical_batch_size": 64.0, "perf/logical_token_count": 48835.0, "perf/gradient_accumulation_steps": 8.0, "system/cuda_memory_allocated_gb": 17.816345691680908, "system/cuda_max_memory_allocated_gb": 82.1704511642456}}
+{"timestamp": 1774094346.1630964, "event": "train_step", "step": 380, "epoch": 2, "metrics": {"train/step_loss": 0.81929341004445, "train/step_real_loss": 0.8280055150389671, "train/lr": 1.8354280658494649e-06, "train/step_canary_loss": 0.26171875, "perf/step_duration_sec": 7.9715049508959055, "perf/samples_per_sec": 8.154043734576712, "perf/tokens_per_sec": 6787.676898315027, "perf/logical_batch_size": 65.0, "perf/logical_token_count": 54108.0, "perf/gradient_accumulation_steps": 8.0, "system/cuda_memory_allocated_gb": 17.816345691680908, "system/cuda_max_memory_allocated_gb": 82.1704511642456}}
+{"timestamp": 1774094429.2324018, "event": "train_step", "step": 390, "epoch": 2, "metrics": {"train/step_loss": 0.8554338663816452, "train/step_real_loss": 0.8554338663816452, "train/lr": 9.17370177272775e-07, "perf/step_duration_sec": 8.24701151996851, "perf/samples_per_sec": 7.760386880148844, "perf/tokens_per_sec": 6142.34621563781, "perf/logical_batch_size": 64.0, "perf/logical_token_count": 50656.0, "perf/gradient_accumulation_steps": 8.0, "system/cuda_memory_allocated_gb": 17.816345691680908, "system/cuda_max_memory_allocated_gb": 82.1704511642456}}
+{"timestamp": 1774094512.2904398, "event": "train_step", "step": 400, "epoch": 2, "metrics": {"train/step_loss": 0.7998756021261215, "train/step_real_loss": 0.7998756021261215, "train/lr": 3.127932624475638e-07, "perf/step_duration_sec": 7.943094424903393, "perf/samples_per_sec": 8.0573132555677, "perf/tokens_per_sec": 6869.86671452841, "perf/logical_batch_size": 64.0, "perf/logical_token_count": 54568.0, "perf/gradient_accumulation_steps": 8.0, "system/cuda_memory_allocated_gb": 17.816345691680908, "system/cuda_max_memory_allocated_gb": 82.1704511642456}}
+{"timestamp": 1774094539.4831598, "event": "eval_step", "step": 400, "epoch": 2, "metrics": {"eval/loss": 0.7455017728538349, "eval/duration_sec": 27.190383709967136}}
+{"timestamp": 1774094621.4459202, "event": "train_step", "step": 410, "epoch": 2, "metrics": {"train/step_loss": 0.854436350591255, "train/step_real_loss": 0.8517185375094414, "train/lr": 2.5558633627303928e-08, "train/step_canary_loss": 0.94140625, "perf/step_duration_sec": 8.378037076443434, "perf/samples_per_sec": 7.877740262760654, "perf/tokens_per_sec": 6647.619184760483, "perf/logical_batch_size": 66.0, "perf/logical_token_count": 55694.0, "perf/gradient_accumulation_steps": 8.0, "system/cuda_memory_allocated_gb": 17.816345691680908, "system/cuda_max_memory_allocated_gb": 88.38552379608154}}
+{"timestamp": 1774094682.1799755, "event": "train_epoch", "step": 414, "epoch": 2, "metrics": {"train/epoch_loss": 0.840315752633972, "train/epoch_real_loss": 0.8362452851041504, "train/epoch_canary_loss": 1.0832555509625612, "perf/epoch_duration_sec": 1814.1021996028721, "perf/epoch_samples_per_sec": 29.765136722628178, "perf/epoch_tokens_per_sec": 24186.19579955584, "perf/epoch_samples": 53997.0, "perf/epoch_tokens": 43876231.0, "system/cuda_epoch_peak_memory_gb": 88.38552379608154, "eval/loss": 0.7454980848164394, "eval/duration_sec": 27.13153049722314}}
+{"timestamp": 1774094693.161264, "event": "audit_epoch", "step": 414, "epoch": 2, "metrics": {"audit/delta": 1e-05, "audit/num_canaries": 500.0, "audit/num_members": 250.0, "audit/paper_guess_fraction": 0.2, "audit/paper_guess_steps": 20.0, "audit/loss/auc": 1.0, "audit/loss/empirical_epsilon/0.05": 3.4791953936219215, "audit/loss/empirical_epsilon/0.01": 3.023197554051876, "audit/loss/empirical_epsilon_details/0.05/epsilon": 3.4791953936219215, "audit/loss/empirical_epsilon_details/0.05/num_guesses": 100.0, "audit/loss/empirical_epsilon_details/0.05/correct_guesses": 100.0, "audit/loss/empirical_epsilon_details/0.01/epsilon": 3.023197554051876, "audit/loss/empirical_epsilon_details/0.01/num_guesses": 100.0, "audit/loss/empirical_epsilon_details/0.01/correct_guesses": 100.0, "audit/embedding/auc": 0.916224, "audit/embedding/empirical_epsilon/0.05": 3.4791953936219215, "audit/embedding/empirical_epsilon/0.01": 3.023197554051876, "audit/embedding/empirical_epsilon_details/0.05/epsilon": 3.4791953936219215, "audit/embedding/empirical_epsilon_details/0.05/num_guesses": 100.0, "audit/embedding/empirical_epsilon_details/0.05/correct_guesses": 100.0, "audit/embedding/empirical_epsilon_details/0.01/epsilon": 3.023197554051876, "audit/embedding/empirical_epsilon_details/0.01/num_guesses": 100.0, "audit/embedding/empirical_epsilon_details/0.01/correct_guesses": 100.0, "perf/audit_duration_sec": 6.722778998315334}}
+{"timestamp": 1774094704.3854914, "event": "audit_final", "step": 414, "epoch": 2, "metrics": {"audit/delta": 1e-05, "audit/num_canaries": 500.0, "audit/num_members": 250.0, "audit/paper_guess_fraction": 0.2, "audit/paper_guess_steps": 20.0, "audit/loss/auc": 1.0, "audit/loss/empirical_epsilon/0.05": 3.4791953936219215, "audit/loss/empirical_epsilon/0.01": 3.023197554051876, "audit/loss/empirical_epsilon_details/0.05/epsilon": 3.4791953936219215, "audit/loss/empirical_epsilon_details/0.05/num_guesses": 100.0, "audit/loss/empirical_epsilon_details/0.05/correct_guesses": 100.0, "audit/loss/empirical_epsilon_details/0.01/epsilon": 3.023197554051876, "audit/loss/empirical_epsilon_details/0.01/num_guesses": 100.0, "audit/loss/empirical_epsilon_details/0.01/correct_guesses": 100.0, "audit/embedding/auc": 0.916224, "audit/embedding/empirical_epsilon/0.05": 3.4791953936219215, "audit/embedding/empirical_epsilon/0.01": 3.023197554051876, "audit/embedding/empirical_epsilon_details/0.05/epsilon": 3.4791953936219215, "audit/embedding/empirical_epsilon_details/0.05/num_guesses": 100.0, "audit/embedding/empirical_epsilon_details/0.05/correct_guesses": 100.0, "audit/embedding/empirical_epsilon_details/0.01/epsilon": 3.023197554051876, "audit/embedding/empirical_epsilon_details/0.01/num_guesses": 100.0, "audit/embedding/empirical_epsilon_details/0.01/correct_guesses": 100.0}}
+{"timestamp": 1774094704.914049, "event": "energy_final", "step": 414, "epoch": null, "metrics": {"energy/codecarbon/duration": 3820.65162669681, "energy/codecarbon/emissions": 0.5568149811716148, "energy/codecarbon/emissions_rate": 0.0001457382236267942, "energy/codecarbon/cpu_power": 179.3439937196561, "energy/codecarbon/gpu_power": 2316.658494927477, "energy/codecarbon/ram_power": 70.0, "energy/codecarbon/cpu_energy": 0.18368298514637418, "energy/codecarbon/gpu_energy": 2.457235179675422, "energy/codecarbon/ram_energy": 0.07169108754482326, "energy/codecarbon/energy_consumed": 2.7126092523666183, "energy/codecarbon/water_consumed": 0.0, "energy/codecarbon/cpu_count": 224.0, "energy/codecarbon/gpu_count": 4.0, "energy/codecarbon/longitude": -121.9552, "energy/codecarbon/latitude": 37.3541, "energy/codecarbon/ram_total_size": 2015.5625190734863, "energy/codecarbon/cpu_utilization_percent": 2.2497114375655825, "energy/codecarbon/gpu_utilization_percent": 96.45330535152151, "energy/codecarbon/ram_utilization_percent": 2.3057974816369358, "energy/codecarbon/ram_used_gb": 46.78091819539025, "energy/codecarbon/pue": 1.0, "energy/codecarbon/wue": 0.0}}

starcoder2-7b/base/pretrain_lm_head.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c9a9b40bb4ecc60cfc3a7e2f7cd31be2121d8f32f92d1a4ecdac554172fb4e5b
+size 457594472

starcoder2-7b/base/resolved_config.yaml

@@ -0,0 +1,100 @@
+model:
+  name: bigcode/starcoder2-7b
+  tokenizer_name: bigcode/starcoder2-7b
+  max_length: 1024
+  dtype: bfloat16
+  trust_remote_code: true
+  use_fast_tokenizer: true
+  cache_dir: null
+  local_files_only: false
+  low_cpu_mem_usage: true
+  tie_word_embeddings: true
+  gradient_checkpointing: false
+  use_chat_template: false
+dataset:
+  name: melihcatal/codedp-cpt
+  split: train
+  mode: cpt
+  text_column: text
+  validation_ratio: 0.05
+  max_samples: -1
+lora:
+  enabled: true
+  r: 16
+  alpha: 32
+  dropout: 0.05
+  target_modules:
+  - q_proj
+  - k_proj
+  - v_proj
+  - o_proj
+  modules_to_save:
+  - lm_head
+  bias: none
+training:
+  seed: 42
+  epochs: 2
+  warmup_steps: null
+  warmup_ratio: 0.05
+  mixed_precision: false
+  mixed_precision_dtype: bfloat16
+  batch_size: 8
+  eval_batch_size: 8
+  eval_every_steps: 50
+  eval_every_epochs: 1
+  learning_rate: 0.0001
+  optimizer: adamw
+  lr_scheduler: cosine
+  adam_beta1: 0.9
+  adam_beta2: 0.999
+  adam_epsilon: 1.0e-08
+  sgd_momentum: 0.9
+  weight_decay: 0.01
+  max_grad_norm: 1.0
+  log_every: 10
+  gradient_accumulation_steps: 8
+  num_workers: 4
+  output_dir: runs/cpt/starcoder2-7b/base
+distributed:
+  strategy: dpddp
+  backend: nccl
+  devices: null
+dp:
+  module_validator: auto
+  target_delta: 1.0e-05
+  noise_multiplier: null
+  max_grad_norm: 1.0
+  grad_sample_mode: ghost
+  secure_mode: false
+  enabled: false
+  target_epsilon: 8.0
+audit:
+  enabled: true
+  run_every_epoch: true
+  epoch_device: cuda
+  q_canary: auto
+  num_canaries: 500
+  prefix_length: 49
+  num_digits: 12
+  batch_size: 32
+  delta: 1.0e-05
+  p_values:
+  - 0.05
+  - 0.01
+  paper_guess_fraction: 0.2
+  paper_guess_steps: 20
+  enable_holdout_empirical_epsilon: false
+  holdout_seed: 42
+  tie_seed: 42
+tracking:
+  enabled: true
+  tensorboard: true
+  wandb: false
+  wandb_project: codedp-finetune-h200-audit
+  wandb_run_name: starcoder2-7b-cpt-base
+  wandb_mode: online
+  codecarbon: true
+  codecarbon_output_file: codecarbon.csv
+  codecarbon_measure_power_secs: 15
+  codecarbon_country_iso_code: null
+  codecarbon_project_name: codedp-starcoder2-7b-cpt-base

starcoder2-7b/base/scalars.csv

@@ -0,0 +1,613 @@
+timestamp,event,step,epoch,key,value
+1774091071.9870658,train_step,10,1,train/step_loss,1.5507386380975896
+1774091071.9870658,train_step,10,1,train/step_real_loss,1.2866992354393005
+1774091071.9870658,train_step,10,1,train/lr,4.761904761904762e-05
+1774091071.9870658,train_step,10,1,train/step_canary_loss,10.0
+1774091071.9870658,train_step,10,1,perf/step_duration_sec,8.244300998747349
+1774091071.9870658,train_step,10,1,perf/samples_per_sec,8.00553012438873
+1774091071.9870658,train_step,10,1,perf/tokens_per_sec,6440.812872803661
+1774091071.9870658,train_step,10,1,perf/logical_batch_size,66.0
+1774091071.9870658,train_step,10,1,perf/logical_token_count,53100.0
+1774091071.9870658,train_step,10,1,perf/gradient_accumulation_steps,8.0
+1774091071.9870658,train_step,10,1,system/cuda_memory_allocated_gb,17.816345691680908
+1774091071.9870658,train_step,10,1,system/cuda_max_memory_allocated_gb,82.16860580444336
+1774091154.380765,train_step,20,1,train/step_loss,1.4504926845208923
+1774091154.380765,train_step,20,1,train/step_real_loss,1.2006134390830994
+1774091154.380765,train_step,20,1,train/lr,9.523809523809524e-05
+1774091154.380765,train_step,20,1,train/step_canary_loss,6.78125
+1774091154.380765,train_step,20,1,perf/step_duration_sec,8.244972918182611
+1774091154.380765,train_step,20,1,perf/samples_per_sec,8.126163744242886
+1774091154.380765,train_step,20,1,perf/tokens_per_sec,6422.095078472549
+1774091154.380765,train_step,20,1,perf/logical_batch_size,67.0
+1774091154.380765,train_step,20,1,perf/logical_token_count,52950.0
+1774091154.380765,train_step,20,1,perf/gradient_accumulation_steps,8.0
+1774091154.380765,train_step,20,1,system/cuda_memory_allocated_gb,17.911073684692383
+1774091154.380765,train_step,20,1,system/cuda_max_memory_allocated_gb,82.16860580444336
+1774091235.6533117,train_step,30,1,train/step_loss,0.9861254319548607
+1774091235.6533117,train_step,30,1,train/step_real_loss,0.9861254319548607
+1774091235.6533117,train_step,30,1,train/lr,9.98706541985615e-05
+1774091235.6533117,train_step,30,1,perf/step_duration_sec,7.973596462979913
+1774091235.6533117,train_step,30,1,perf/samples_per_sec,8.026490968929942
+1774091235.6533117,train_step,30,1,perf/tokens_per_sec,7148.9697609673985
+1774091235.6533117,train_step,30,1,perf/logical_batch_size,64.0
+1774091235.6533117,train_step,30,1,perf/logical_token_count,57003.0
+1774091235.6533117,train_step,30,1,perf/gradient_accumulation_steps,8.0
+1774091235.6533117,train_step,30,1,system/cuda_memory_allocated_gb,17.816345691680908
+1774091235.6533117,train_step,30,1,system/cuda_max_memory_allocated_gb,82.16860580444336
+1774091318.662035,train_step,40,1,train/step_loss,1.0557814985513687
+1774091318.662035,train_step,40,1,train/step_real_loss,1.0557814985513687
+1774091318.662035,train_step,40,1,train/lr,9.942439201095397e-05
+1774091318.662035,train_step,40,1,perf/step_duration_sec,8.119743634015322
+1774091318.662035,train_step,40,1,perf/samples_per_sec,7.882022251526572
+1774091318.662035,train_step,40,1,perf/tokens_per_sec,6340.347961767047
+1774091318.662035,train_step,40,1,perf/logical_batch_size,64.0
+1774091318.662035,train_step,40,1,perf/logical_token_count,51482.0
+1774091318.662035,train_step,40,1,perf/gradient_accumulation_steps,8.0
+1774091318.662035,train_step,40,1,system/cuda_memory_allocated_gb,17.816345691680908
+1774091318.662035,train_step,40,1,system/cuda_max_memory_allocated_gb,88.38552379608154
+1774091402.0365882,train_step,50,1,train/step_loss,1.017115435233483
+1774091402.0365882,train_step,50,1,train/step_real_loss,0.9500000476837158
+1774091402.0365882,train_step,50,1,train/lr,9.866246608261724e-05
+1774091402.0365882,train_step,50,1,train/step_canary_loss,5.3125
+1774091402.0365882,train_step,50,1,perf/step_duration_sec,8.119036318734288
+1774091402.0365882,train_step,50,1,perf/samples_per_sec,8.005876245437603
+1774091402.0365882,train_step,50,1,perf/tokens_per_sec,6307.645142790008
+1774091402.0365882,train_step,50,1,perf/logical_batch_size,65.0
+1774091402.0365882,train_step,50,1,perf/logical_token_count,51212.0
+1774091402.0365882,train_step,50,1,perf/gradient_accumulation_steps,8.0
+1774091402.0365882,train_step,50,1,system/cuda_memory_allocated_gb,17.816345691680908
+1774091402.0365882,train_step,50,1,system/cuda_max_memory_allocated_gb,88.38552379608154
+1774091429.1958544,eval_step,50,1,eval/loss,0.8033588177513802
+1774091429.1958544,eval_step,50,1,eval/duration_sec,27.156014366075397
+1774091511.9092546,train_step,60,1,train/step_loss,1.2237938749256418
+1774091511.9092546,train_step,60,1,train/step_real_loss,0.9667061120271683
+1774091511.9092546,train_step,60,1,train/lr,9.7589742682592e-05
+1774091511.9092546,train_step,60,1,train/step_canary_loss,6.708333333333333
+1774091511.9092546,train_step,60,1,perf/step_duration_sec,8.495171755552292
+1774091511.9092546,train_step,60,1,perf/samples_per_sec,7.886832889071372
+1774091511.9092546,train_step,60,1,perf/tokens_per_sec,6228.008276044599
+1774091511.9092546,train_step,60,1,perf/logical_batch_size,67.0
+1774091511.9092546,train_step,60,1,perf/logical_token_count,52908.0
+1774091511.9092546,train_step,60,1,perf/gradient_accumulation_steps,8.0
+1774091511.9092546,train_step,60,1,system/cuda_memory_allocated_gb,17.816345691680908
+1774091511.9092546,train_step,60,1,system/cuda_max_memory_allocated_gb,88.38552379608154
+1774091593.5344725,train_step,70,1,train/step_loss,1.0743033794256358
+1774091593.5344725,train_step,70,1,train/step_real_loss,0.9387456253170967
+1774091593.5344725,train_step,70,1,train/lr,9.621307308142384e-05
+1774091593.5344725,train_step,70,1,train/step_canary_loss,9.75
+1774091593.5344725,train_step,70,1,perf/step_duration_sec,8.257235972210765
+1774091593.5344725,train_step,70,1,perf/samples_per_sec,7.8718835478062665
+1774091593.5344725,train_step,70,1,perf/tokens_per_sec,6568.057420488078
+1774091593.5344725,train_step,70,1,perf/logical_batch_size,65.0
+1774091593.5344725,train_step,70,1,perf/logical_token_count,54234.0
+1774091593.5344725,train_step,70,1,perf/gradient_accumulation_steps,8.0
+1774091593.5344725,train_step,70,1,system/cuda_memory_allocated_gb,17.816345691680908
+1774091593.5344725,train_step,70,1,system/cuda_max_memory_allocated_gb,88.38552379608154
+1774091676.5485113,train_step,80,1,train/step_loss,0.8626528829336166
+1774091676.5485113,train_step,80,1,train/step_real_loss,0.8626528829336166
+1774091676.5485113,train_step,80,1,train/lr,9.454124979346391e-05
+1774091676.5485113,train_step,80,1,perf/step_duration_sec,8.386640733107924
+1774091676.5485113,train_step,80,1,perf/samples_per_sec,7.631184169765056
+1774091676.5485113,train_step,80,1,perf/tokens_per_sec,6658.446662625317
+1774091676.5485113,train_step,80,1,perf/logical_batch_size,64.0
+1774091676.5485113,train_step,80,1,perf/logical_token_count,55842.0
+1774091676.5485113,train_step,80,1,perf/gradient_accumulation_steps,8.0
+1774091676.5485113,train_step,80,1,system/cuda_memory_allocated_gb,17.816345691680908
+1774091676.5485113,train_step,80,1,system/cuda_max_memory_allocated_gb,88.38552379608154
+1774091757.879373,train_step,90,1,train/step_loss,0.9340124875307083
+1774091757.879373,train_step,90,1,train/step_real_loss,0.9340124875307083
+1774091757.879373,train_step,90,1,train/lr,9.258495042083221e-05
+1774091757.879373,train_step,90,1,perf/step_duration_sec,8.493970146402717
+1774091757.879373,train_step,90,1,perf/samples_per_sec,7.534756880103312
+1774091757.879373,train_step,90,1,perf/tokens_per_sec,5785.162786428072
+1774091757.879373,train_step,90,1,perf/logical_batch_size,64.0
+1774091757.879373,train_step,90,1,perf/logical_token_count,49139.0
+1774091757.879373,train_step,90,1,perf/gradient_accumulation_steps,8.0
+1774091757.879373,train_step,90,1,system/cuda_memory_allocated_gb,17.816345691680908
+1774091757.879373,train_step,90,1,system/cuda_max_memory_allocated_gb,88.38552379608154
+1774091839.6918592,train_step,100,1,train/step_loss,0.9577739440477812
+1774091839.6918592,train_step,100,1,train/step_real_loss,0.880453996360302
+1774091839.6918592,train_step,100,1,train/lr,9.035666945770107e-05
+1774091839.6918592,train_step,100,1,train/step_canary_loss,5.90625
+1774091839.6918592,train_step,100,1,perf/step_duration_sec,8.234961070120335
+1774091839.6918592,train_step,100,1,perf/samples_per_sec,7.893176354633353
+1774091839.6918592,train_step,100,1,perf/tokens_per_sec,6771.616711381143
+1774091839.6918592,train_step,100,1,perf/logical_batch_size,65.0
+1774091839.6918592,train_step,100,1,perf/logical_token_count,55764.0
+1774091839.6918592,train_step,100,1,perf/gradient_accumulation_steps,8.0
+1774091839.6918592,train_step,100,1,system/cuda_memory_allocated_gb,17.816345691680908
+1774091839.6918592,train_step,100,1,system/cuda_max_memory_allocated_gb,88.38552379608154
+1774091866.8768253,eval_step,100,1,eval/loss,0.7754902612717671
+1774091866.8768253,eval_step,100,1,eval/duration_sec,27.182465851306915
+1774091950.1670244,train_step,110,1,train/step_loss,0.9567192720644402
+1774091950.1670244,train_step,110,1,train/step_real_loss,0.860762245953083
+1774091950.1670244,train_step,110,1,train/lr,8.787063849045118e-05
+1774091950.1670244,train_step,110,1,train/step_canary_loss,4.02734375
+1774091950.1670244,train_step,110,1,perf/step_duration_sec,8.036363879218698
+1774091950.1670244,train_step,110,1,perf/samples_per_sec,8.21266943507498
+1774091950.1670244,train_step,110,1,perf/tokens_per_sec,6478.054102878841
+1774091950.1670244,train_step,110,1,perf/logical_batch_size,66.0
+1774091950.1670244,train_step,110,1,perf/logical_token_count,52060.0
+1774091950.1670244,train_step,110,1,perf/gradient_accumulation_steps,8.0
+1774091950.1670244,train_step,110,1,system/cuda_memory_allocated_gb,17.816345691680908
+1774091950.1670244,train_step,110,1,system/cuda_max_memory_allocated_gb,88.38552379608154
+1774092031.8751383,train_step,120,1,train/step_loss,0.9170611575245857
+1774092031.8751383,train_step,120,1,train/step_real_loss,0.9170611575245857
+1774092031.8751383,train_step,120,1,train/lr,8.5142735303366e-05
+1774092031.8751383,train_step,120,1,perf/step_duration_sec,7.79720464348793
+1774092031.8751383,train_step,120,1,perf/samples_per_sec,8.208069805305357
+1774092031.8751383,train_step,120,1,perf/tokens_per_sec,6600.4423832818775
+1774092031.8751383,train_step,120,1,perf/logical_batch_size,64.0
+1774092031.8751383,train_step,120,1,perf/logical_token_count,51465.0
+1774092031.8751383,train_step,120,1,perf/gradient_accumulation_steps,8.0
+1774092031.8751383,train_step,120,1,system/cuda_memory_allocated_gb,17.816345691680908
+1774092031.8751383,train_step,120,1,system/cuda_max_memory_allocated_gb,88.38552379608154
+1774092114.4179726,train_step,130,1,train/step_loss,0.9158061426697355
+1774092114.4179726,train_step,130,1,train/step_real_loss,0.8711828961968422
+1774092114.4179726,train_step,130,1,train/lr,8.219038247038819e-05
+1774092114.4179726,train_step,130,1,train/step_canary_loss,2.34375
+1774092114.4179726,train_step,130,1,perf/step_duration_sec,8.22744246572256
+1774092114.4179726,train_step,130,1,perf/samples_per_sec,8.021933945447975
+1774092114.4179726,train_step,130,1,perf/tokens_per_sec,6342.311139506375
+1774092114.4179726,train_step,130,1,perf/logical_batch_size,66.0
+1774092114.4179726,train_step,130,1,perf/logical_token_count,52181.0
+1774092114.4179726,train_step,130,1,perf/gradient_accumulation_steps,8.0
+1774092114.4179726,train_step,130,1,system/cuda_memory_allocated_gb,17.816345691680908
+1774092114.4179726,train_step,130,1,system/cuda_max_memory_allocated_gb,88.38552379608154
+1774092196.3902268,train_step,140,1,train/step_loss,0.9856266433542425
+1774092196.3902268,train_step,140,1,train/step_real_loss,0.8489470109343529
+1774092196.3902268,train_step,140,1,train/lr,7.903243608061246e-05
+1774092196.3902268,train_step,140,1,train/step_canary_loss,5.359375
+1774092196.3902268,train_step,140,1,perf/step_duration_sec,8.333684524521232
+1774092196.3902268,train_step,140,1,perf/samples_per_sec,7.9196662419605675
+1774092196.3902268,train_step,140,1,perf/tokens_per_sec,6461.007714123103
+1774092196.3902268,train_step,140,1,perf/logical_batch_size,66.0
+1774092196.3902268,train_step,140,1,perf/logical_token_count,53844.0
+1774092196.3902268,train_step,140,1,perf/gradient_accumulation_steps,8.0
+1774092196.3902268,train_step,140,1,system/cuda_memory_allocated_gb,17.816345691680908
+1774092196.3902268,train_step,140,1,system/cuda_max_memory_allocated_gb,88.38552379608154
+1774092278.982639,train_step,150,1,train/step_loss,0.94221530854702
+1774092278.982639,train_step,150,1,train/step_real_loss,0.94221530854702
+1774092278.982639,train_step,150,1,train/lr,7.568906530820282e-05
+1774092278.982639,train_step,150,1,perf/step_duration_sec,8.388992108404636
+1774092278.982639,train_step,150,1,perf/samples_per_sec,7.629045202686584
+1774092278.982639,train_step,150,1,perf/tokens_per_sec,6081.421861022835
+1774092278.982639,train_step,150,1,perf/logical_batch_size,64.0
+1774092278.982639,train_step,150,1,perf/logical_token_count,51017.0
+1774092278.982639,train_step,150,1,perf/gradient_accumulation_steps,8.0
+1774092278.982639,train_step,150,1,system/cuda_memory_allocated_gb,17.816345691680908
+1774092278.982639,train_step,150,1,system/cuda_max_memory_allocated_gb,88.38552379608154
+1774092306.1516783,eval_step,150,1,eval/loss,0.7622188641422096
+1774092306.1516783,eval_step,150,1,eval/duration_sec,27.166858203709126
+1774092388.1349308,train_step,160,1,train/step_loss,0.9667639210820198
+1774092388.1349308,train_step,160,1,train/step_real_loss,0.9667639210820198
+1774092388.1349308,train_step,160,1,train/lr,7.21816235958972e-05
+1774092388.1349308,train_step,160,1,perf/step_duration_sec,8.235355000942945
+1774092388.1349308,train_step,160,1,perf/samples_per_sec,7.771371117902267
+1774092388.1349308,train_step,160,1,perf/tokens_per_sec,6008.24129552819
+1774092388.1349308,train_step,160,1,perf/logical_batch_size,64.0
+1774092388.1349308,train_step,160,1,perf/logical_token_count,49480.0
+1774092388.1349308,train_step,160,1,perf/gradient_accumulation_steps,8.0
+1774092388.1349308,train_step,160,1,system/cuda_memory_allocated_gb,17.816345691680908
+1774092388.1349308,train_step,160,1,system/cuda_max_memory_allocated_gb,88.38552379608154
+1774092469.9282126,train_step,170,1,train/step_loss,0.9262635037302971
+1774092469.9282126,train_step,170,1,train/step_real_loss,0.9262635037302971
+1774092469.9282126,train_step,170,1,train/lr,6.853251227482479e-05
+1774092469.9282126,train_step,170,1,perf/step_duration_sec,7.9527543764561415
+1774092469.9282126,train_step,170,1,perf/samples_per_sec,8.047526299752175
+1774092469.9282126,train_step,170,1,perf/tokens_per_sec,6280.842791759705
+1774092469.9282126,train_step,170,1,perf/logical_batch_size,64.0
+1774092469.9282126,train_step,170,1,perf/logical_token_count,49950.0
+1774092469.9282126,train_step,170,1,perf/gradient_accumulation_steps,8.0
+1774092469.9282126,train_step,170,1,system/cuda_memory_allocated_gb,17.816345691680908
+1774092469.9282126,train_step,170,1,system/cuda_max_memory_allocated_gb,88.38552379608154
+1774092551.943189,train_step,180,1,train/step_loss,0.8705217910535408
+1774092551.943189,train_step,180,1,train/step_real_loss,0.8467002063989639
+1774092551.943189,train_step,180,1,train/lr,6.476503749166904e-05
+1774092551.943189,train_step,180,1,train/step_canary_loss,1.6328125
+1774092551.943189,train_step,180,1,perf/step_duration_sec,8.540382800623775
+1774092551.943189,train_step,180,1,perf/samples_per_sec,7.7279908337574135
+1774092551.943189,train_step,180,1,perf/tokens_per_sec,6385.076790236751
+1774092551.943189,train_step,180,1,perf/logical_batch_size,66.0
+1774092551.943189,train_step,180,1,perf/logical_token_count,54531.0
+1774092551.943189,train_step,180,1,perf/gradient_accumulation_steps,8.0
+1774092551.943189,train_step,180,1,system/cuda_memory_allocated_gb,17.816345691680908
+1774092551.943189,train_step,180,1,system/cuda_max_memory_allocated_gb,88.38552379608154
+1774092634.1860914,train_step,190,1,train/step_loss,0.8329041105050307
+1774092634.1860914,train_step,190,1,train/step_real_loss,0.830781526863575
+1774092634.1860914,train_step,190,1,train/lr,6.090326135695403e-05
+1774092634.1860914,train_step,190,1,train/step_canary_loss,0.96875
+1774092634.1860914,train_step,190,1,perf/step_duration_sec,7.938739079982042
+1774092634.1860914,train_step,190,1,perf/samples_per_sec,8.18769824088324
+1774092634.1860914,train_step,190,1,perf/tokens_per_sec,6963.07051322375
+1774092634.1860914,train_step,190,1,perf/logical_batch_size,65.0
+1774092634.1860914,train_step,190,1,perf/logical_token_count,55278.0
+1774092634.1860914,train_step,190,1,perf/gradient_accumulation_steps,8.0
+1774092634.1860914,train_step,190,1,system/cuda_memory_allocated_gb,17.816345691680908
+1774092634.1860914,train_step,190,1,system/cuda_max_memory_allocated_gb,88.38552379608154
+1774092717.69117,train_step,200,1,train/step_loss,0.9225348509274997
+1774092717.69117,train_step,200,1,train/step_real_loss,0.8622424304485321
+1774092717.69117,train_step,200,1,train/lr,5.697184826514057e-05
+1774092717.69117,train_step,200,1,train/step_canary_loss,4.78125
+1774092717.69117,train_step,200,1,perf/step_duration_sec,8.257816776633263
+1774092717.69117,train_step,200,1,perf/samples_per_sec,7.871329887571168
+1774092717.69117,train_step,200,1,perf/tokens_per_sec,6772.4922352662325
+1774092717.69117,train_step,200,1,perf/logical_batch_size,65.0
+1774092717.69117,train_step,200,1,perf/logical_token_count,55926.0
+1774092717.69117,train_step,200,1,perf/gradient_accumulation_steps,8.0
+1774092717.69117,train_step,200,1,system/cuda_memory_allocated_gb,17.816345691680908
+1774092717.69117,train_step,200,1,system/cuda_max_memory_allocated_gb,88.38552379608154
+1774092744.8800561,eval_step,200,1,eval/loss,0.7524948388166811
+1774092744.8800561,eval_step,200,1,eval/duration_sec,27.18629032932222
+1774092830.2611096,train_epoch,207,1,train/epoch_loss,1.036294524908722
+1774092830.2611096,train_epoch,207,1,train/epoch_real_loss,0.957591299003593
+1774092830.2611096,train_epoch,207,1,train/epoch_canary_loss,5.154934052784185
+1774092830.2611096,train_epoch,207,1,perf/epoch_duration_sec,1814.8832574244589
+1774092830.2611096,train_epoch,207,1,perf/epoch_samples_per_sec,29.762244915220545
+1774092830.2611096,train_epoch,207,1,perf/epoch_tokens_per_sec,24175.732417227537
+1774092830.2611096,train_epoch,207,1,perf/epoch_samples,54015.0
+1774092830.2611096,train_epoch,207,1,perf/epoch_tokens,43876132.0
+1774092830.2611096,train_epoch,207,1,system/cuda_epoch_peak_memory_gb,88.38552379608154
+1774092830.2611096,train_epoch,207,1,eval/loss,0.7516984458284817
+1774092830.2611096,train_epoch,207,1,eval/duration_sec,27.281295781955123
+1774092840.935869,audit_epoch,207,1,audit/delta,1e-05
+1774092840.935869,audit_epoch,207,1,audit/num_canaries,500.0
+1774092840.935869,audit_epoch,207,1,audit/num_members,250.0
+1774092840.935869,audit_epoch,207,1,audit/paper_guess_fraction,0.2
+1774092840.935869,audit_epoch,207,1,audit/paper_guess_steps,20.0
+1774092840.935869,audit_epoch,207,1,audit/loss/auc,0.998072
+1774092840.935869,audit_epoch,207,1,audit/loss/empirical_epsilon/0.05,3.4791953936219215
+1774092840.935869,audit_epoch,207,1,audit/loss/empirical_epsilon/0.01,3.023197554051876
+1774092840.935869,audit_epoch,207,1,audit/loss/empirical_epsilon_details/0.05/epsilon,3.4791953936219215
+1774092840.935869,audit_epoch,207,1,audit/loss/empirical_epsilon_details/0.05/num_guesses,100.0
+1774092840.935869,audit_epoch,207,1,audit/loss/empirical_epsilon_details/0.05/correct_guesses,100.0
+1774092840.935869,audit_epoch,207,1,audit/loss/empirical_epsilon_details/0.01/epsilon,3.023197554051876
+1774092840.935869,audit_epoch,207,1,audit/loss/empirical_epsilon_details/0.01/num_guesses,100.0
+1774092840.935869,audit_epoch,207,1,audit/loss/empirical_epsilon_details/0.01/correct_guesses,100.0
+1774092840.935869,audit_epoch,207,1,audit/embedding/auc,0.96
+1774092840.935869,audit_epoch,207,1,audit/embedding/empirical_epsilon/0.05,3.4791953936219215
+1774092840.935869,audit_epoch,207,1,audit/embedding/empirical_epsilon/0.01,3.023197554051876
+1774092840.935869,audit_epoch,207,1,audit/embedding/empirical_epsilon_details/0.05/epsilon,3.4791953936219215
+1774092840.935869,audit_epoch,207,1,audit/embedding/empirical_epsilon_details/0.05/num_guesses,100.0
+1774092840.935869,audit_epoch,207,1,audit/embedding/empirical_epsilon_details/0.05/correct_guesses,100.0
+1774092840.935869,audit_epoch,207,1,audit/embedding/empirical_epsilon_details/0.01/epsilon,3.023197554051876
+1774092840.935869,audit_epoch,207,1,audit/embedding/empirical_epsilon_details/0.01/num_guesses,100.0
+1774092840.935869,audit_epoch,207,1,audit/embedding/empirical_epsilon_details/0.01/correct_guesses,100.0
+1774092840.935869,audit_epoch,207,1,perf/audit_duration_sec,6.3129072319716215
+1774092866.0795767,train_step,210,2,train/step_loss,0.8786039505944108
+1774092866.0795767,train_step,210,2,train/step_real_loss,0.8291560411453247
+1774092866.0795767,train_step,210,2,train/lr,5.29959073680547e-05
+1774092866.0795767,train_step,210,2,train/step_canary_loss,2.4609375
+1774092866.0795767,train_step,210,2,perf/step_duration_sec,8.105620637536049
+1774092866.0795767,train_step,210,2,perf/samples_per_sec,8.142498020985931
+1774092866.0795767,train_step,210,2,perf/tokens_per_sec,6755.805933775599
+1774092866.0795767,train_step,210,2,perf/logical_batch_size,66.0
+1774092866.0795767,train_step,210,2,perf/logical_token_count,54760.0
+1774092866.0795767,train_step,210,2,perf/gradient_accumulation_steps,8.0
+1774092866.0795767,train_step,210,2,system/cuda_memory_allocated_gb,17.911073684692383
+1774092866.0795767,train_step,210,2,system/cuda_max_memory_allocated_gb,75.98184061050415
+1774092947.7881637,train_step,220,2,train/step_loss,0.8338463682394761
+1774092947.7881637,train_step,220,2,train/step_real_loss,0.8286867365241051
+1774092947.7881637,train_step,220,2,train/lr,4.9000832207739676e-05
+1774092947.7881637,train_step,220,2,train/step_canary_loss,1.1640625
+1774092947.7881637,train_step,220,2,perf/step_duration_sec,8.100939376279712
+1774092947.7881637,train_step,220,2,perf/samples_per_sec,8.023760823383757
+1774092947.7881637,train_step,220,2,perf/tokens_per_sec,6522.083124668936
+1774092947.7881637,train_step,220,2,perf/logical_batch_size,65.0
+1774092947.7881637,train_step,220,2,perf/logical_token_count,52835.0
+1774092947.7881637,train_step,220,2,perf/gradient_accumulation_steps,8.0
+1774092947.7881637,train_step,220,2,system/cuda_memory_allocated_gb,17.816345691680908
+1774092947.7881637,train_step,220,2,system/cuda_max_memory_allocated_gb,82.1704511642456
+1774093029.3670137,train_step,230,2,train/step_loss,0.7926744809517494
+1774093029.3670137,train_step,230,2,train/step_real_loss,0.8034120723605156
+1774093029.3670137,train_step,230,2,train/lr,4.501213853296425e-05
+1774093029.3670137,train_step,230,2,train/step_canary_loss,0.10546875
+1774093029.3670137,train_step,230,2,perf/step_duration_sec,8.09845926053822
+1774093029.3670137,train_step,230,2,perf/samples_per_sec,8.026218063073905
+1774093029.3670137,train_step,230,2,perf/tokens_per_sec,6708.683497950849
+1774093029.3670137,train_step,230,2,perf/logical_batch_size,65.0
+1774093029.3670137,train_step,230,2,perf/logical_token_count,54330.0
+1774093029.3670137,train_step,230,2,perf/gradient_accumulation_steps,8.0
+1774093029.3670137,train_step,230,2,system/cuda_memory_allocated_gb,17.816345691680908
+1774093029.3670137,train_step,230,2,system/cuda_max_memory_allocated_gb,82.1704511642456
+1774093111.4853563,train_step,240,2,train/step_loss,0.8129148510369387
+1774093111.4853563,train_step,240,2,train/step_real_loss,0.7906041964888573
+1774093111.4853563,train_step,240,2,train/lr,4.1055301335220955e-05
+1774093111.4853563,train_step,240,2,train/step_canary_loss,1.52685546875
+1774093111.4853563,train_step,240,2,perf/step_duration_sec,8.505769751966
+1774093111.4853563,train_step,240,2,perf/samples_per_sec,7.75943881913156
+1774093111.4853563,train_step,240,2,perf/tokens_per_sec,6383.901937558238
+1774093111.4853563,train_step,240,2,perf/logical_batch_size,66.0
+1774093111.4853563,train_step,240,2,perf/logical_token_count,54300.0
+1774093111.4853563,train_step,240,2,perf/gradient_accumulation_steps,8.0
+1774093111.4853563,train_step,240,2,system/cuda_memory_allocated_gb,17.816345691680908
+1774093111.4853563,train_step,240,2,system/cuda_max_memory_allocated_gb,82.1704511642456
+1774093192.88336,train_step,250,2,train/step_loss,0.8410789425556476
+1774093192.88336,train_step,250,2,train/step_real_loss,0.850802831351757
+1774093192.88336,train_step,250,2,train/lr,3.715559214503298e-05
+1774093192.88336,train_step,250,2,train/step_canary_loss,0.21875
+1774093192.88336,train_step,250,2,perf/step_duration_sec,8.232771871611476
+1774093192.88336,train_step,250,2,perf/samples_per_sec,7.895275250385015
+1774093192.88336,train_step,250,2,perf/tokens_per_sec,6654.745309889904
+1774093192.88336,train_step,250,2,perf/logical_batch_size,65.0
+1774093192.88336,train_step,250,2,perf/logical_token_count,54787.0
+1774093192.88336,train_step,250,2,perf/gradient_accumulation_steps,8.0
+1774093192.88336,train_step,250,2,system/cuda_memory_allocated_gb,17.816345691680908
+1774093192.88336,train_step,250,2,system/cuda_max_memory_allocated_gb,82.1704511642456
+1774093220.0732572,eval_step,250,2,eval/loss,0.748522541389383
+1774093220.0732572,eval_step,250,2,eval/duration_sec,27.187080297619104
+1774093302.2765276,train_step,260,2,train/step_loss,0.8724462010643699
+1774093302.2765276,train_step,260,2,train/step_real_loss,0.8488068357110023
+1774093302.2765276,train_step,260,2,train/lr,3.33379176277258e-05
+1774093302.2765276,train_step,260,2,train/step_canary_loss,1.62890625
+1774093302.2765276,train_step,260,2,perf/step_duration_sec,8.470736568793654
+1774093302.2765276,train_step,260,2,perf/samples_per_sec,7.7915302245551095
+1774093302.2765276,train_step,260,2,perf/tokens_per_sec,6277.49423773815
+1774093302.2765276,train_step,260,2,perf/logical_batch_size,66.0
+1774093302.2765276,train_step,260,2,perf/logical_token_count,53175.0
+1774093302.2765276,train_step,260,2,perf/gradient_accumulation_steps,8.0
+1774093302.2765276,train_step,260,2,system/cuda_memory_allocated_gb,17.911073684692383
+1774093302.2765276,train_step,260,2,system/cuda_max_memory_allocated_gb,82.1704511642456
+1774093384.391679,train_step,270,2,train/step_loss,0.8640658855438232
+1774093384.391679,train_step,270,2,train/step_real_loss,0.7926059737801552
+1774093384.391679,train_step,270,2,train/lr,2.962666050951997e-05
+1774093384.391679,train_step,270,2,train/step_canary_loss,5.4375
+1774093384.391679,train_step,270,2,perf/step_duration_sec,7.947357261553407
+1774093384.391679,train_step,270,2,perf/samples_per_sec,8.178819431516908
+1774093384.391679,train_step,270,2,perf/tokens_per_sec,6407.161314659091
+1774093384.391679,train_step,270,2,perf/logical_batch_size,65.0
+1774093384.391679,train_step,270,2,perf/logical_token_count,50920.0
+1774093384.391679,train_step,270,2,perf/gradient_accumulation_steps,8.0
+1774093384.391679,train_step,270,2,system/cuda_memory_allocated_gb,17.816345691680908
+1774093384.391679,train_step,270,2,system/cuda_max_memory_allocated_gb,82.1704511642456
+1774093468.0585487,train_step,280,2,train/step_loss,0.7910019425245431
+1774093468.0585487,train_step,280,2,train/step_real_loss,0.8018507286906242
+1774093468.0585487,train_step,280,2,train/lr,2.604552384991855e-05
+1774093468.0585487,train_step,280,2,train/step_canary_loss,0.0966796875
+1774093468.0585487,train_step,280,2,perf/step_duration_sec,8.62639987282455
+1774093468.0585487,train_step,280,2,perf/samples_per_sec,7.53500892124967
+1774093468.0585487,train_step,280,2,perf/tokens_per_sec,6275.039506402552
+1774093468.0585487,train_step,280,2,perf/logical_batch_size,65.0
+1774093468.0585487,train_step,280,2,perf/logical_token_count,54131.0
+1774093468.0585487,train_step,280,2,perf/gradient_accumulation_steps,8.0
+1774093468.0585487,train_step,280,2,system/cuda_memory_allocated_gb,17.816345691680908
+1774093468.0585487,train_step,280,2,system/cuda_max_memory_allocated_gb,82.1704511642456
+1774093550.7132335,train_step,290,2,train/step_loss,0.8430270507480159
+1774093550.7132335,train_step,290,2,train/step_real_loss,0.8613760396838188
+1774093550.7132335,train_step,290,2,train/lr,2.2617379654990623e-05
+1774093550.7132335,train_step,290,2,train/step_canary_loss,0.255859375
+1774093550.7132335,train_step,290,2,perf/step_duration_sec,8.335046991705894
+1774093550.7132335,train_step,290,2,perf/samples_per_sec,7.9183716739300705
+1774093550.7132335,train_step,290,2,perf/tokens_per_sec,6261.392413496007
+1774093550.7132335,train_step,290,2,perf/logical_batch_size,66.0
+1774093550.7132335,train_step,290,2,perf/logical_token_count,52189.0
+1774093550.7132335,train_step,290,2,perf/gradient_accumulation_steps,8.0
+1774093550.7132335,train_step,290,2,system/cuda_memory_allocated_gb,17.816345691680908
+1774093550.7132335,train_step,290,2,system/cuda_max_memory_allocated_gb,82.1704511642456
+1774093632.5524132,train_step,300,2,train/step_loss,0.7691241015087474
+1774093632.5524132,train_step,300,2,train/step_real_loss,0.784599058330059
+1774093632.5524132,train_step,300,2,train/lr,1.936412279842705e-05
+1774093632.5524132,train_step,300,2,train/step_canary_loss,0.27392578125
+1774093632.5524132,train_step,300,2,perf/step_duration_sec,8.091827008873224
+1774093632.5524132,train_step,300,2,perf/samples_per_sec,8.156378025336753
+1774093632.5524132,train_step,300,2,perf/tokens_per_sec,5984.062677922065
+1774093632.5524132,train_step,300,2,perf/logical_batch_size,66.0
+1774093632.5524132,train_step,300,2,perf/logical_token_count,48422.0
+1774093632.5524132,train_step,300,2,perf/gradient_accumulation_steps,8.0
+1774093632.5524132,train_step,300,2,system/cuda_memory_allocated_gb,17.911073684692383
+1774093632.5524132,train_step,300,2,system/cuda_max_memory_allocated_gb,82.1704511642456
+1774093659.719083,eval_step,300,2,eval/loss,0.7462242173737493
+1774093659.719083,eval_step,300,2,eval/duration_sec,27.16416385397315
+1774093742.5612013,train_step,310,2,train/step_loss,0.7514625552928809
+1774093742.5612013,train_step,310,2,train/step_real_loss,0.7710395082831383
+1774093742.5612013,train_step,310,2,train/lr,1.6306531183346385e-05
+1774093742.5612013,train_step,310,2,train/step_canary_loss,0.125
+1774093742.5612013,train_step,310,2,perf/step_duration_sec,8.240195842459798
+1774093742.5612013,train_step,310,2,perf/samples_per_sec,8.009518373327666
+1774093742.5612013,train_step,310,2,perf/tokens_per_sec,6369.751520897312
+1774093742.5612013,train_step,310,2,perf/logical_batch_size,66.0
+1774093742.5612013,train_step,310,2,perf/logical_token_count,52488.0
+1774093742.5612013,train_step,310,2,perf/gradient_accumulation_steps,8.0
+1774093742.5612013,train_step,310,2,system/cuda_memory_allocated_gb,17.816345691680908
+1774093742.5612013,train_step,310,2,system/cuda_max_memory_allocated_gb,82.1704511642456
+1774093824.76255,train_step,320,2,train/step_loss,0.8317307600608239
+1774093824.76255,train_step,320,2,train/step_real_loss,0.8416290208697319
+1774093824.76255,train_step,320,2,train/lr,1.3464133037968912e-05
+1774093824.76255,train_step,320,2,train/step_canary_loss,0.1982421875
+1774093824.76255,train_step,320,2,perf/step_duration_sec,7.953533122316003
+1774093824.76255,train_step,320,2,perf/samples_per_sec,8.172468637569782
+1774093824.76255,train_step,320,2,perf/tokens_per_sec,6226.666720107783
+1774093824.76255,train_step,320,2,perf/logical_batch_size,65.0
+1774093824.76255,train_step,320,2,perf/logical_token_count,49524.0
+1774093824.76255,train_step,320,2,perf/gradient_accumulation_steps,8.0
+1774093824.76255,train_step,320,2,system/cuda_memory_allocated_gb,17.816345691680908
+1774093824.76255,train_step,320,2,system/cuda_max_memory_allocated_gb,82.1704511642456
+1774093907.919859,train_step,330,2,train/step_loss,0.8719267258277306
+1774093907.919859,train_step,330,2,train/step_real_loss,0.8635779246687889
+1774093907.919859,train_step,330,2,train/lr,1.0855082192715294e-05
+1774093907.919859,train_step,330,2,train/step_canary_loss,1.40625
+1774093907.919859,train_step,330,2,perf/step_duration_sec,8.101611092686653
+1774093907.919859,train_step,330,2,perf/samples_per_sec,8.023095561656334
+1774093907.919859,train_step,330,2,perf/tokens_per_sec,5623.572827524039
+1774093907.919859,train_step,330,2,perf/logical_batch_size,65.0
+1774093907.919859,train_step,330,2,perf/logical_token_count,45560.0
+1774093907.919859,train_step,330,2,perf/gradient_accumulation_steps,8.0
+1774093907.919859,train_step,330,2,system/cuda_memory_allocated_gb,17.816345691680908
+1774093907.919859,train_step,330,2,system/cuda_max_memory_allocated_gb,82.1704511642456
+1774093989.7239172,train_step,340,2,train/step_loss,0.8476224361043988
+1774093989.7239172,train_step,340,2,train/step_real_loss,0.8228105828166008
+1774093989.7239172,train_step,340,2,train/lr,8.49604213531004e-06
+1774093989.7239172,train_step,340,2,train/step_canary_loss,1.6416015625
+1774093989.7239172,train_step,340,2,perf/step_duration_sec,8.274557610973716
+1774093989.7239172,train_step,340,2,perf/samples_per_sec,7.976257233676254
+1774093989.7239172,train_step,340,2,perf/tokens_per_sec,6674.435371234426
+1774093989.7239172,train_step,340,2,perf/logical_batch_size,66.0
+1774093989.7239172,train_step,340,2,perf/logical_token_count,55228.0
+1774093989.7239172,train_step,340,2,perf/gradient_accumulation_steps,8.0
+1774093989.7239172,train_step,340,2,system/cuda_memory_allocated_gb,17.816345691680908
+1774093989.7239172,train_step,340,2,system/cuda_max_memory_allocated_gb,82.1704511642456
+1774094071.9116077,train_step,350,2,train/step_loss,0.8573094416355741
+1774094071.9116077,train_step,350,2,train/step_real_loss,0.816773310303688
+1774094071.9116077,train_step,350,2,train/lr,6.402079584406673e-06
+1774094071.9116077,train_step,350,2,train/step_canary_loss,1.376171875
+1774094071.9116077,train_step,350,2,perf/step_duration_sec,8.637357847765088
+1774094071.9116077,train_step,350,2,perf/samples_per_sec,7.988554048140279
+1774094071.9116077,train_step,350,2,perf/tokens_per_sec,6212.547974249382
+1774094071.9116077,train_step,350,2,perf/logical_batch_size,69.0
+1774094071.9116077,train_step,350,2,perf/logical_token_count,53660.0
+1774094071.9116077,train_step,350,2,perf/gradient_accumulation_steps,8.0
+1774094071.9116077,train_step,350,2,system/cuda_memory_allocated_gb,17.816345691680908
+1774094071.9116077,train_step,350,2,system/cuda_max_memory_allocated_gb,82.1704511642456
+1774094099.091823,eval_step,350,2,eval/loss,0.7456388631942629
+1774094099.091823,eval_step,350,2,eval/duration_sec,27.17722495831549
+1774094181.5654633,train_step,360,2,train/step_loss,0.8446246770712046
+1774094181.5654633,train_step,360,2,train/step_real_loss,0.8453097268939018
+1774094181.5654633,train_step,360,2,train/lr,4.586568261458729e-06
+1774094181.5654633,train_step,360,2,train/step_canary_loss,0.80078125
+1774094181.5654633,train_step,360,2,perf/step_duration_sec,7.976690696552396
+1774094181.5654633,train_step,360,2,perf/samples_per_sec,8.148742689508275
+1774094181.5654633,train_step,360,2,perf/tokens_per_sec,6948.119478163339
+1774094181.5654633,train_step,360,2,perf/logical_batch_size,65.0
+1774094181.5654633,train_step,360,2,perf/logical_token_count,55423.0
+1774094181.5654633,train_step,360,2,perf/gradient_accumulation_steps,8.0
+1774094181.5654633,train_step,360,2,system/cuda_memory_allocated_gb,17.816345691680908
+1774094181.5654633,train_step,360,2,system/cuda_max_memory_allocated_gb,82.1704511642456
+1774094263.5117958,train_step,370,2,train/step_loss,0.8721330687403679
+1774094263.5117958,train_step,370,2,train/step_real_loss,0.8721330687403679
+1774094263.5117958,train_step,370,2,train/lr,3.06110347542643e-06
+1774094263.5117958,train_step,370,2,perf/step_duration_sec,8.37641635723412
+1774094263.5117958,train_step,370,2,perf/samples_per_sec,7.640498904370688
+1774094263.5117958,train_step,370,2,perf/tokens_per_sec,5830.058812420977
+1774094263.5117958,train_step,370,2,perf/logical_batch_size,64.0
+1774094263.5117958,train_step,370,2,perf/logical_token_count,48835.0
+1774094263.5117958,train_step,370,2,perf/gradient_accumulation_steps,8.0
+1774094263.5117958,train_step,370,2,system/cuda_memory_allocated_gb,17.816345691680908
+1774094263.5117958,train_step,370,2,system/cuda_max_memory_allocated_gb,82.1704511642456
+1774094346.1630964,train_step,380,2,train/step_loss,0.81929341004445
+1774094346.1630964,train_step,380,2,train/step_real_loss,0.8280055150389671
+1774094346.1630964,train_step,380,2,train/lr,1.8354280658494649e-06
+1774094346.1630964,train_step,380,2,train/step_canary_loss,0.26171875
+1774094346.1630964,train_step,380,2,perf/step_duration_sec,7.9715049508959055
+1774094346.1630964,train_step,380,2,perf/samples_per_sec,8.154043734576712
+1774094346.1630964,train_step,380,2,perf/tokens_per_sec,6787.676898315027
+1774094346.1630964,train_step,380,2,perf/logical_batch_size,65.0
+1774094346.1630964,train_step,380,2,perf/logical_token_count,54108.0
+1774094346.1630964,train_step,380,2,perf/gradient_accumulation_steps,8.0
+1774094346.1630964,train_step,380,2,system/cuda_memory_allocated_gb,17.816345691680908
+1774094346.1630964,train_step,380,2,system/cuda_max_memory_allocated_gb,82.1704511642456
+1774094429.2324018,train_step,390,2,train/step_loss,0.8554338663816452
+1774094429.2324018,train_step,390,2,train/step_real_loss,0.8554338663816452
+1774094429.2324018,train_step,390,2,train/lr,9.17370177272775e-07
+1774094429.2324018,train_step,390,2,perf/step_duration_sec,8.24701151996851
+1774094429.2324018,train_step,390,2,perf/samples_per_sec,7.760386880148844
+1774094429.2324018,train_step,390,2,perf/tokens_per_sec,6142.34621563781
+1774094429.2324018,train_step,390,2,perf/logical_batch_size,64.0
+1774094429.2324018,train_step,390,2,perf/logical_token_count,50656.0
+1774094429.2324018,train_step,390,2,perf/gradient_accumulation_steps,8.0
+1774094429.2324018,train_step,390,2,system/cuda_memory_allocated_gb,17.816345691680908
+1774094429.2324018,train_step,390,2,system/cuda_max_memory_allocated_gb,82.1704511642456
+1774094512.2904398,train_step,400,2,train/step_loss,0.7998756021261215
+1774094512.2904398,train_step,400,2,train/step_real_loss,0.7998756021261215
+1774094512.2904398,train_step,400,2,train/lr,3.127932624475638e-07
+1774094512.2904398,train_step,400,2,perf/step_duration_sec,7.943094424903393
+1774094512.2904398,train_step,400,2,perf/samples_per_sec,8.0573132555677
+1774094512.2904398,train_step,400,2,perf/tokens_per_sec,6869.86671452841
+1774094512.2904398,train_step,400,2,perf/logical_batch_size,64.0
+1774094512.2904398,train_step,400,2,perf/logical_token_count,54568.0
+1774094512.2904398,train_step,400,2,perf/gradient_accumulation_steps,8.0
+1774094512.2904398,train_step,400,2,system/cuda_memory_allocated_gb,17.816345691680908
+1774094512.2904398,train_step,400,2,system/cuda_max_memory_allocated_gb,82.1704511642456
+1774094539.4831598,eval_step,400,2,eval/loss,0.7455017728538349
+1774094539.4831598,eval_step,400,2,eval/duration_sec,27.190383709967136
+1774094621.4459202,train_step,410,2,train/step_loss,0.854436350591255
+1774094621.4459202,train_step,410,2,train/step_real_loss,0.8517185375094414
+1774094621.4459202,train_step,410,2,train/lr,2.5558633627303928e-08
+1774094621.4459202,train_step,410,2,train/step_canary_loss,0.94140625
+1774094621.4459202,train_step,410,2,perf/step_duration_sec,8.378037076443434
+1774094621.4459202,train_step,410,2,perf/samples_per_sec,7.877740262760654
+1774094621.4459202,train_step,410,2,perf/tokens_per_sec,6647.619184760483
+1774094621.4459202,train_step,410,2,perf/logical_batch_size,66.0
+1774094621.4459202,train_step,410,2,perf/logical_token_count,55694.0
+1774094621.4459202,train_step,410,2,perf/gradient_accumulation_steps,8.0
+1774094621.4459202,train_step,410,2,system/cuda_memory_allocated_gb,17.816345691680908
+1774094621.4459202,train_step,410,2,system/cuda_max_memory_allocated_gb,88.38552379608154
+1774094682.1799755,train_epoch,414,2,train/epoch_loss,0.840315752633972
+1774094682.1799755,train_epoch,414,2,train/epoch_real_loss,0.8362452851041504
+1774094682.1799755,train_epoch,414,2,train/epoch_canary_loss,1.0832555509625612
+1774094682.1799755,train_epoch,414,2,perf/epoch_duration_sec,1814.1021996028721
+1774094682.1799755,train_epoch,414,2,perf/epoch_samples_per_sec,29.765136722628178
+1774094682.1799755,train_epoch,414,2,perf/epoch_tokens_per_sec,24186.19579955584
+1774094682.1799755,train_epoch,414,2,perf/epoch_samples,53997.0
+1774094682.1799755,train_epoch,414,2,perf/epoch_tokens,43876231.0
+1774094682.1799755,train_epoch,414,2,system/cuda_epoch_peak_memory_gb,88.38552379608154
+1774094682.1799755,train_epoch,414,2,eval/loss,0.7454980848164394
+1774094682.1799755,train_epoch,414,2,eval/duration_sec,27.13153049722314
+1774094693.161264,audit_epoch,414,2,audit/delta,1e-05
+1774094693.161264,audit_epoch,414,2,audit/num_canaries,500.0
+1774094693.161264,audit_epoch,414,2,audit/num_members,250.0
+1774094693.161264,audit_epoch,414,2,audit/paper_guess_fraction,0.2
+1774094693.161264,audit_epoch,414,2,audit/paper_guess_steps,20.0
+1774094693.161264,audit_epoch,414,2,audit/loss/auc,1.0
+1774094693.161264,audit_epoch,414,2,audit/loss/empirical_epsilon/0.05,3.4791953936219215
+1774094693.161264,audit_epoch,414,2,audit/loss/empirical_epsilon/0.01,3.023197554051876
+1774094693.161264,audit_epoch,414,2,audit/loss/empirical_epsilon_details/0.05/epsilon,3.4791953936219215
+1774094693.161264,audit_epoch,414,2,audit/loss/empirical_epsilon_details/0.05/num_guesses,100.0
+1774094693.161264,audit_epoch,414,2,audit/loss/empirical_epsilon_details/0.05/correct_guesses,100.0
+1774094693.161264,audit_epoch,414,2,audit/loss/empirical_epsilon_details/0.01/epsilon,3.023197554051876
+1774094693.161264,audit_epoch,414,2,audit/loss/empirical_epsilon_details/0.01/num_guesses,100.0
+1774094693.161264,audit_epoch,414,2,audit/loss/empirical_epsilon_details/0.01/correct_guesses,100.0
+1774094693.161264,audit_epoch,414,2,audit/embedding/auc,0.916224
+1774094693.161264,audit_epoch,414,2,audit/embedding/empirical_epsilon/0.05,3.4791953936219215
+1774094693.161264,audit_epoch,414,2,audit/embedding/empirical_epsilon/0.01,3.023197554051876
+1774094693.161264,audit_epoch,414,2,audit/embedding/empirical_epsilon_details/0.05/epsilon,3.4791953936219215
+1774094693.161264,audit_epoch,414,2,audit/embedding/empirical_epsilon_details/0.05/num_guesses,100.0
+1774094693.161264,audit_epoch,414,2,audit/embedding/empirical_epsilon_details/0.05/correct_guesses,100.0
+1774094693.161264,audit_epoch,414,2,audit/embedding/empirical_epsilon_details/0.01/epsilon,3.023197554051876
+1774094693.161264,audit_epoch,414,2,audit/embedding/empirical_epsilon_details/0.01/num_guesses,100.0
+1774094693.161264,audit_epoch,414,2,audit/embedding/empirical_epsilon_details/0.01/correct_guesses,100.0
+1774094693.161264,audit_epoch,414,2,perf/audit_duration_sec,6.722778998315334
+1774094704.3854914,audit_final,414,2,audit/delta,1e-05
+1774094704.3854914,audit_final,414,2,audit/num_canaries,500.0
+1774094704.3854914,audit_final,414,2,audit/num_members,250.0
+1774094704.3854914,audit_final,414,2,audit/paper_guess_fraction,0.2
+1774094704.3854914,audit_final,414,2,audit/paper_guess_steps,20.0
+1774094704.3854914,audit_final,414,2,audit/loss/auc,1.0
+1774094704.3854914,audit_final,414,2,audit/loss/empirical_epsilon/0.05,3.4791953936219215
+1774094704.3854914,audit_final,414,2,audit/loss/empirical_epsilon/0.01,3.023197554051876
+1774094704.3854914,audit_final,414,2,audit/loss/empirical_epsilon_details/0.05/epsilon,3.4791953936219215
+1774094704.3854914,audit_final,414,2,audit/loss/empirical_epsilon_details/0.05/num_guesses,100.0
+1774094704.3854914,audit_final,414,2,audit/loss/empirical_epsilon_details/0.05/correct_guesses,100.0
+1774094704.3854914,audit_final,414,2,audit/loss/empirical_epsilon_details/0.01/epsilon,3.023197554051876
+1774094704.3854914,audit_final,414,2,audit/loss/empirical_epsilon_details/0.01/num_guesses,100.0
+1774094704.3854914,audit_final,414,2,audit/loss/empirical_epsilon_details/0.01/correct_guesses,100.0
+1774094704.3854914,audit_final,414,2,audit/embedding/auc,0.916224
+1774094704.3854914,audit_final,414,2,audit/embedding/empirical_epsilon/0.05,3.4791953936219215
+1774094704.3854914,audit_final,414,2,audit/embedding/empirical_epsilon/0.01,3.023197554051876
+1774094704.3854914,audit_final,414,2,audit/embedding/empirical_epsilon_details/0.05/epsilon,3.4791953936219215
+1774094704.3854914,audit_final,414,2,audit/embedding/empirical_epsilon_details/0.05/num_guesses,100.0
+1774094704.3854914,audit_final,414,2,audit/embedding/empirical_epsilon_details/0.05/correct_guesses,100.0
+1774094704.3854914,audit_final,414,2,audit/embedding/empirical_epsilon_details/0.01/epsilon,3.023197554051876
+1774094704.3854914,audit_final,414,2,audit/embedding/empirical_epsilon_details/0.01/num_guesses,100.0
+1774094704.3854914,audit_final,414,2,audit/embedding/empirical_epsilon_details/0.01/correct_guesses,100.0
+1774094704.914049,energy_final,414,,energy/codecarbon/duration,3820.65162669681
+1774094704.914049,energy_final,414,,energy/codecarbon/emissions,0.5568149811716148
+1774094704.914049,energy_final,414,,energy/codecarbon/emissions_rate,0.0001457382236267942
+1774094704.914049,energy_final,414,,energy/codecarbon/cpu_power,179.3439937196561
+1774094704.914049,energy_final,414,,energy/codecarbon/gpu_power,2316.658494927477
+1774094704.914049,energy_final,414,,energy/codecarbon/ram_power,70.0
+1774094704.914049,energy_final,414,,energy/codecarbon/cpu_energy,0.18368298514637418
+1774094704.914049,energy_final,414,,energy/codecarbon/gpu_energy,2.457235179675422
+1774094704.914049,energy_final,414,,energy/codecarbon/ram_energy,0.07169108754482326
+1774094704.914049,energy_final,414,,energy/codecarbon/energy_consumed,2.7126092523666183
+1774094704.914049,energy_final,414,,energy/codecarbon/water_consumed,0.0
+1774094704.914049,energy_final,414,,energy/codecarbon/cpu_count,224.0
+1774094704.914049,energy_final,414,,energy/codecarbon/gpu_count,4.0
+1774094704.914049,energy_final,414,,energy/codecarbon/longitude,-121.9552
+1774094704.914049,energy_final,414,,energy/codecarbon/latitude,37.3541
+1774094704.914049,energy_final,414,,energy/codecarbon/ram_total_size,2015.5625190734863
+1774094704.914049,energy_final,414,,energy/codecarbon/cpu_utilization_percent,2.2497114375655825
+1774094704.914049,energy_final,414,,energy/codecarbon/gpu_utilization_percent,96.45330535152151
+1774094704.914049,energy_final,414,,energy/codecarbon/ram_utilization_percent,2.3057974816369358
+1774094704.914049,energy_final,414,,energy/codecarbon/ram_used_gb,46.78091819539025
+1774094704.914049,energy_final,414,,energy/codecarbon/pue,1.0
+1774094704.914049,energy_final,414,,energy/codecarbon/wue,0.0

starcoder2-7b/base/summary.json

@@ -0,0 +1,71 @@
+{
+  "audit/delta": 1e-05,
+  "audit/embedding/auc": 0.916224,
+  "audit/embedding/empirical_epsilon/0.01": 3.023197554051876,
+  "audit/embedding/empirical_epsilon/0.05": 3.4791953936219215,
+  "audit/embedding/empirical_epsilon_details/0.01/correct_guesses": 100.0,
+  "audit/embedding/empirical_epsilon_details/0.01/epsilon": 3.023197554051876,
+  "audit/embedding/empirical_epsilon_details/0.01/num_guesses": 100.0,
+  "audit/embedding/empirical_epsilon_details/0.05/correct_guesses": 100.0,
+  "audit/embedding/empirical_epsilon_details/0.05/epsilon": 3.4791953936219215,
+  "audit/embedding/empirical_epsilon_details/0.05/num_guesses": 100.0,
+  "audit/loss/auc": 1.0,
+  "audit/loss/empirical_epsilon/0.01": 3.023197554051876,
+  "audit/loss/empirical_epsilon/0.05": 3.4791953936219215,
+  "audit/loss/empirical_epsilon_details/0.01/correct_guesses": 100.0,
+  "audit/loss/empirical_epsilon_details/0.01/epsilon": 3.023197554051876,
+  "audit/loss/empirical_epsilon_details/0.01/num_guesses": 100.0,
+  "audit/loss/empirical_epsilon_details/0.05/correct_guesses": 100.0,
+  "audit/loss/empirical_epsilon_details/0.05/epsilon": 3.4791953936219215,
+  "audit/loss/empirical_epsilon_details/0.05/num_guesses": 100.0,
+  "audit/num_canaries": 500.0,
+  "audit/num_members": 250.0,
+  "audit/paper_guess_fraction": 0.2,
+  "audit/paper_guess_steps": 20.0,
+  "energy/codecarbon/cpu_count": 224.0,
+  "energy/codecarbon/cpu_energy": 0.18368298514637418,
+  "energy/codecarbon/cpu_power": 179.3439937196561,
+  "energy/codecarbon/cpu_utilization_percent": 2.2497114375655825,
+  "energy/codecarbon/duration": 3820.65162669681,
+  "energy/codecarbon/emissions": 0.5568149811716148,
+  "energy/codecarbon/emissions_rate": 0.0001457382236267942,
+  "energy/codecarbon/energy_consumed": 2.7126092523666183,
+  "energy/codecarbon/gpu_count": 4.0,
+  "energy/codecarbon/gpu_energy": 2.457235179675422,
+  "energy/codecarbon/gpu_power": 2316.658494927477,
+  "energy/codecarbon/gpu_utilization_percent": 96.45330535152151,
+  "energy/codecarbon/latitude": 37.3541,
+  "energy/codecarbon/longitude": -121.9552,
+  "energy/codecarbon/pue": 1.0,
+  "energy/codecarbon/ram_energy": 0.07169108754482326,
+  "energy/codecarbon/ram_power": 70.0,
+  "energy/codecarbon/ram_total_size": 2015.5625190734863,
+  "energy/codecarbon/ram_used_gb": 46.78091819539025,
+  "energy/codecarbon/ram_utilization_percent": 2.3057974816369358,
+  "energy/codecarbon/water_consumed": 0.0,
+  "energy/codecarbon/wue": 0.0,
+  "eval/duration_sec": 27.13153049722314,
+  "eval/loss": 0.7454980848164394,
+  "perf/audit_duration_sec": 6.722778998315334,
+  "perf/epoch_duration_sec": 1814.1021996028721,
+  "perf/epoch_samples": 53997.0,
+  "perf/epoch_samples_per_sec": 29.765136722628178,
+  "perf/epoch_tokens": 43876231.0,
+  "perf/epoch_tokens_per_sec": 24186.19579955584,
+  "perf/gradient_accumulation_steps": 8.0,
+  "perf/logical_batch_size": 66.0,
+  "perf/logical_token_count": 55694.0,
+  "perf/samples_per_sec": 7.877740262760654,
+  "perf/step_duration_sec": 8.378037076443434,
+  "perf/tokens_per_sec": 6647.619184760483,
+  "system/cuda_epoch_peak_memory_gb": 88.38552379608154,
+  "system/cuda_max_memory_allocated_gb": 88.38552379608154,
+  "system/cuda_memory_allocated_gb": 17.816345691680908,
+  "train/epoch_canary_loss": 1.0832555509625612,
+  "train/epoch_loss": 0.840315752633972,
+  "train/epoch_real_loss": 0.8362452851041504,
+  "train/lr": 2.5558633627303928e-08,
+  "train/step_canary_loss": 0.94140625,
+  "train/step_loss": 0.854436350591255,
+  "train/step_real_loss": 0.8517185375094414
+}

starcoder2-7b/base/tensorboard/events.out.tfevents.1774090870.364c4f8de9dd.6145.0

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:da29212bc469bf3917a2aa87b8ff1081b72b125dbbaefea65a979c0832e71373
+size 41006

starcoder2-7b/base/tokenizer/tokenizer_config.json

@@ -0,0 +1,516 @@
+{
+  "add_prefix_space": false,
+  "backend": "tokenizers",
+  "bos_token": "<|endoftext|>",
+  "clean_up_tokenization_spaces": true,
+  "eos_token": "<|endoftext|>",
+  "errors": "replace",
+  "extra_special_tokens": {
+    "extra_special_token_0": "865331112869",
+    "extra_special_token_1": "569765693871",
+    "extra_special_token_2": "485177821815",
+    "extra_special_token_3": "135441121756",
+    "extra_special_token_4": "367459894796",
+    "extra_special_token_5": "877482678543",
+    "extra_special_token_6": "457919547633",
+    "extra_special_token_7": "765474393376",
+    "extra_special_token_8": "114848338811",
+    "extra_special_token_9": "746285987371",
+    "extra_special_token_10": "649291669397",
+    "extra_special_token_11": "927914615679",
+    "extra_special_token_12": "445925149649",
+    "extra_special_token_13": "691587454538",
+    "extra_special_token_14": "143777992227",
+    "extra_special_token_15": "997981281989",
+    "extra_special_token_16": "425949483533",
+    "extra_special_token_17": "982993456429",
+    "extra_special_token_18": "718726519731",
+    "extra_special_token_19": "172599315861",
+    "extra_special_token_20": "643489267333",
+    "extra_special_token_21": "282322838685",
+    "extra_special_token_22": "781653545886",
+    "extra_special_token_23": "796415361892",
+    "extra_special_token_24": "841991688488",
+    "extra_special_token_25": "211411365397",
+    "extra_special_token_26": "698218415444",
+    "extra_special_token_27": "355977139358",
+    "extra_special_token_28": "682564697312",
+    "extra_special_token_29": "383837596997",
+    "extra_special_token_30": "689362171782",
+    "extra_special_token_31": "749966767285",
+    "extra_special_token_32": "753159165157",
+    "extra_special_token_33": "795693824762",
+    "extra_special_token_34": "669689115557",
+    "extra_special_token_35": "327491773134",
+    "extra_special_token_36": "983569279932",
+    "extra_special_token_37": "612128769512",
+    "extra_special_token_38": "374327157578",
+    "extra_special_token_39": "311632789559",
+    "extra_special_token_40": "523918658846",
+    "extra_special_token_41": "765981581453",
+    "extra_special_token_42": "794825141891",
+    "extra_special_token_43": "873898736873",
+    "extra_special_token_44": "447445629421",
+    "extra_special_token_45": "473822473819",
+    "extra_special_token_46": "181439694557",
+    "extra_special_token_47": "592538279337",
+    "extra_special_token_48": "668134915514",
+    "extra_special_token_49": "643692393748",
+    "extra_special_token_50": "696651276628",
+    "extra_special_token_51": "853859348234",
+    "extra_special_token_52": "778466723723",
+    "extra_special_token_53": "929826356991",
+    "extra_special_token_54": "272362973463",
+    "extra_special_token_55": "694235616268",
+    "extra_special_token_56": "281673864127",
+    "extra_special_token_57": "479676316326",
+    "extra_special_token_58": "646979124677",
+    "extra_special_token_59": "922327493433",
+    "extra_special_token_60": "883685933161",
+    "extra_special_token_61": "264259917554",
+    "extra_special_token_62": "836746273134",
+    "extra_special_token_63": "658481324922",
+    "extra_special_token_64": "481884157827",
+    "extra_special_token_65": "587787496812",
+    "extra_special_token_66": "579184949249",
+    "extra_special_token_67": "912193598348",
+    "extra_special_token_68": "529679678956",
+    "extra_special_token_69": "795838284624",
+    "extra_special_token_70": "159337222655",
+    "extra_special_token_71": "173781362446",
+    "extra_special_token_72": "773687856563",
+    "extra_special_token_73": "535787224917",
+    "extra_special_token_74": "351885857332",
+    "extra_special_token_75": "578827344666",
+    "extra_special_token_76": "198462689911",
+    "extra_special_token_77": "722618266242",
+    "extra_special_token_78": "952872416512",
+    "extra_special_token_79": "517778845323",
+    "extra_special_token_80": "749665846687",
+    "extra_special_token_81": "661436365453",
+    "extra_special_token_82": "259666844669",
+    "extra_special_token_83": "242851284913",
+    "extra_special_token_84": "514532995959",
+    "extra_special_token_85": "161588262349",
+    "extra_special_token_86": "742765629356",
+    "extra_special_token_87": "225164373623",
+    "extra_special_token_88": "676539973863",
+    "extra_special_token_89": "826214551218",
+    "extra_special_token_90": "182345464792",
+    "extra_special_token_91": "232776999554",
+    "extra_special_token_92": "337326533813",
+    "extra_special_token_93": "676676697292",
+    "extra_special_token_94": "929185622831",
+    "extra_special_token_95": "545512344383",
+    "extra_special_token_96": "499444466686",
+    "extra_special_token_97": "314697386682",
+    "extra_special_token_98": "517379856925",
+    "extra_special_token_99": "379557332953",
+    "extra_special_token_100": "614797267726",
+    "extra_special_token_101": "429781429464",
+    "extra_special_token_102": "922466849763",
+    "extra_special_token_103": "721737645236",
+    "extra_special_token_104": "479227349997",
+    "extra_special_token_105": "136931728327",
+    "extra_special_token_106": "259533577263",
+    "extra_special_token_107": "488538864842",
+    "extra_special_token_108": "937495658852",
+    "extra_special_token_109": "489991411364",
+    "extra_special_token_110": "499148455254",
+    "extra_special_token_111": "441373944925",
+    "extra_special_token_112": "899151413682",
+    "extra_special_token_113": "467893531755",
+    "extra_special_token_114": "527117488925",
+    "extra_special_token_115": "928335588653",
+    "extra_special_token_116": "374439448821",
+    "extra_special_token_117": "879425227932",
+    "extra_special_token_118": "867678158885",
+    "extra_special_token_119": "399749397872",
+    "extra_special_token_120": "129693547287",
+    "extra_special_token_121": "689285841825",
+    "extra_special_token_122": "771619544974",
+    "extra_special_token_123": "724883568652",
+    "extra_special_token_124": "516968424863",
+    "extra_special_token_125": "733737988257",
+    "extra_special_token_126": "852347289392",
+    "extra_special_token_127": "296953381169",
+    "extra_special_token_128": "377273562477",
+    "extra_special_token_129": "262296912232",
+    "extra_special_token_130": "547149832394",
+    "extra_special_token_131": "298464134954",
+    "extra_special_token_132": "216667245274",
+    "extra_special_token_133": "843998562287",
+    "extra_special_token_134": "572154333646",
+    "extra_special_token_135": "124589118494",
+    "extra_special_token_136": "841824384614",
+    "extra_special_token_137": "232896526252",
+    "extra_special_token_138": "295448593321",
+    "extra_special_token_139": "123741461297",
+    "extra_special_token_140": "653573457168",
+    "extra_special_token_141": "196735786156",
+    "extra_special_token_142": "377338713663",
+    "extra_special_token_143": "964342468552",
+    "extra_special_token_144": "586855179568",
+    "extra_special_token_145": "484773717614",
+    "extra_special_token_146": "894885246797",
+    "extra_special_token_147": "677896358599",
+    "extra_special_token_148": "848845611563",
+    "extra_special_token_149": "851852651677",
+    "extra_special_token_150": "398549545767",
+    "extra_special_token_151": "454244839926",
+    "extra_special_token_152": "799364566435",
+    "extra_special_token_153": "967114116556",
+    "extra_special_token_154": "817378986438",
+    "extra_special_token_155": "233795848681",
+    "extra_special_token_156": "824387273757",
+    "extra_special_token_157": "916198946615",
+    "extra_special_token_158": "563117729724",
+    "extra_special_token_159": "951794811935",
+    "extra_special_token_160": "374598961236",
+    "extra_special_token_161": "922867396683",
+    "extra_special_token_162": "765737843639",
+    "extra_special_token_163": "175469284871",
+    "extra_special_token_164": "231853711778",
+    "extra_special_token_165": "662426712668",
+    "extra_special_token_166": "711412347158",
+    "extra_special_token_167": "753466987363",
+    "extra_special_token_168": "513361312532",
+    "extra_special_token_169": "712992815957",
+    "extra_special_token_170": "971621888444",
+    "extra_special_token_171": "829235161526",
+    "extra_special_token_172": "585544633356",
+    "extra_special_token_173": "582471228164",
+    "extra_special_token_174": "678666359123",
+    "extra_special_token_175": "557533689478",
+    "extra_special_token_176": "632962475133",
+    "extra_special_token_177": "484489193824",
+    "extra_special_token_178": "489562189822",
+    "extra_special_token_179": "589547936288",
+    "extra_special_token_180": "363214487524",
+    "extra_special_token_181": "244885399387",
+    "extra_special_token_182": "431751228368",
+    "extra_special_token_183": "433581868192",
+    "extra_special_token_184": "486391569221",
+    "extra_special_token_185": "185438575221",
+    "extra_special_token_186": "126574388585",
+    "extra_special_token_187": "741757479784",
+    "extra_special_token_188": "529854679937",
+    "extra_special_token_189": "996116119839",
+    "extra_special_token_190": "616248973917",
+    "extra_special_token_191": "763531783491",
+    "extra_special_token_192": "955456118295",
+    "extra_special_token_193": "364196983365",
+    "extra_special_token_194": "195792996468",
+    "extra_special_token_195": "151859598873",
+    "extra_special_token_196": "399223169721",
+    "extra_special_token_197": "938488813964",
+    "extra_special_token_198": "961981959227",
+    "extra_special_token_199": "183368827562",
+    "extra_special_token_200": "533417736566",
+    "extra_special_token_201": "786391632558",
+    "extra_special_token_202": "665661658354",
+    "extra_special_token_203": "693281533643",
+    "extra_special_token_204": "475794684356",
+    "extra_special_token_205": "652154162978",
+    "extra_special_token_206": "753233719644",
+    "extra_special_token_207": "668514843129",
+    "extra_special_token_208": "819162623892",
+    "extra_special_token_209": "941169431859",
+    "extra_special_token_210": "877385381798",
+    "extra_special_token_211": "752644929761",
+    "extra_special_token_212": "881136466196",
+    "extra_special_token_213": "275597777299",
+    "extra_special_token_214": "731681792655",
+    "extra_special_token_215": "961133895172",
+    "extra_special_token_216": "864718285734",
+    "extra_special_token_217": "963852916563",
+    "extra_special_token_218": "319584985416",
+    "extra_special_token_219": "563365646341",
+    "extra_special_token_220": "811371928234",
+    "extra_special_token_221": "837131396371",
+    "extra_special_token_222": "267514771964",
+    "extra_special_token_223": "944513428457",
+    "extra_special_token_224": "117298239631",
+    "extra_special_token_225": "158142752582",
+    "extra_special_token_226": "252867443568",
+    "extra_special_token_227": "839269684865",
+    "extra_special_token_228": "612788593128",
+    "extra_special_token_229": "145669731981",
+    "extra_special_token_230": "121557291859",
+    "extra_special_token_231": "245416776926",
+    "extra_special_token_232": "799417897197",
+    "extra_special_token_233": "997958836435",
+    "extra_special_token_234": "892336777248",
+    "extra_special_token_235": "158929292238",
+    "extra_special_token_236": "581976444672",
+    "extra_special_token_237": "897784492783",
+    "extra_special_token_238": "492373714791",
+    "extra_special_token_239": "512659818733",
+    "extra_special_token_240": "881112998642",
+    "extra_special_token_241": "619454958782",
+    "extra_special_token_242": "431149748713",
+    "extra_special_token_243": "624221476921",
+    "extra_special_token_244": "125866399464",
+    "extra_special_token_245": "339882449689",
+    "extra_special_token_246": "186198784585",
+    "extra_special_token_247": "943193294691",
+    "extra_special_token_248": "955668961269",
+    "extra_special_token_249": "232787996724",
+    "extra_special_token_250": "215671314196",
+    "extra_special_token_251": "286173241916",
+    "extra_special_token_252": "745977673725",
+    "extra_special_token_253": "556976448182",
+    "extra_special_token_254": "599961512792",
+    "extra_special_token_255": "766294538337",
+    "extra_special_token_256": "934912591213",
+    "extra_special_token_257": "295118729589",
+    "extra_special_token_258": "529455466433",
+    "extra_special_token_259": "196119929397",
+    "extra_special_token_260": "379571934299",
+    "extra_special_token_261": "251789649997",
+    "extra_special_token_262": "564544131355",
+    "extra_special_token_263": "244371196654",
+    "extra_special_token_264": "384598329253",
+    "extra_special_token_265": "887753195844",
+    "extra_special_token_266": "364947325679",
+    "extra_special_token_267": "655517954651",
+    "extra_special_token_268": "673948786567",
+    "extra_special_token_269": "857231548835",
+    "extra_special_token_270": "816115936673",
+    "extra_special_token_271": "644234165531",
+    "extra_special_token_272": "182782912224",
+    "extra_special_token_273": "234316622259",
+    "extra_special_token_274": "421369185549",
+    "extra_special_token_275": "434632855397",
+    "extra_special_token_276": "921889371893",
+    "extra_special_token_277": "415956914763",
+    "extra_special_token_278": "598916996413",
+    "extra_special_token_279": "773671349113",
+    "extra_special_token_280": "952465217972",
+    "extra_special_token_281": "117657531962",
+    "extra_special_token_282": "729825168745",
+    "extra_special_token_283": "691315125346",
+    "extra_special_token_284": "768461952319",
+    "extra_special_token_285": "664847713559",
+    "extra_special_token_286": "953267689786",
+    "extra_special_token_287": "886464195129",
+    "extra_special_token_288": "824488329416",
+    "extra_special_token_289": "837873762491",
+    "extra_special_token_290": "532833541879",
+    "extra_special_token_291": "669183782449",
+    "extra_special_token_292": "941976537588",
+    "extra_special_token_293": "739394546916",
+    "extra_special_token_294": "267954879268",
+    "extra_special_token_295": "637551427887",
+    "extra_special_token_296": "217756494954",
+    "extra_special_token_297": "524444658383",
+    "extra_special_token_298": "117783274348",
+    "extra_special_token_299": "138218735276",
+    "extra_special_token_300": "814611949491",
+    "extra_special_token_301": "711641973413",
+    "extra_special_token_302": "499156317423",
+    "extra_special_token_303": "515856611931",
+    "extra_special_token_304": "454164859837",
+    "extra_special_token_305": "345271433112",
+    "extra_special_token_306": "462294118988",
+    "extra_special_token_307": "511785788222",
+    "extra_special_token_308": "497294727353",
+    "extra_special_token_309": "866519986723",
+    "extra_special_token_310": "334513529294",
+    "extra_special_token_311": "549946382131",
+    "extra_special_token_312": "284445431422",
+    "extra_special_token_313": "396521188476",
+    "extra_special_token_314": "421435255895",
+    "extra_special_token_315": "133373659361",
+    "extra_special_token_316": "322683334381",
+    "extra_special_token_317": "228358422847",
+    "extra_special_token_318": "291762694874",
+    "extra_special_token_319": "143182978129",
+    "extra_special_token_320": "511923256573",
+    "extra_special_token_321": "327158398268",
+    "extra_special_token_322": "879764613759",
+    "extra_special_token_323": "564395222747",
+    "extra_special_token_324": "451161679736",
+    "extra_special_token_325": "538631466654",
+    "extra_special_token_326": "221762325616",
+    "extra_special_token_327": "218391991184",
+    "extra_special_token_328": "322589379462",
+    "extra_special_token_329": "876537814263",
+    "extra_special_token_330": "152676556624",
+    "extra_special_token_331": "332522971941",
+    "extra_special_token_332": "884354318946",
+    "extra_special_token_333": "513349618943",
+    "extra_special_token_334": "116639746413",
+    "extra_special_token_335": "635185846287",
+    "extra_special_token_336": "993832498489",
+    "extra_special_token_337": "813981174797",
+    "extra_special_token_338": "438745114173",
+    "extra_special_token_339": "983493951323",
+    "extra_special_token_340": "724492262421",
+    "extra_special_token_341": "622553389126",
+    "extra_special_token_342": "889965243135",
+    "extra_special_token_343": "364492359246",
+    "extra_special_token_344": "154962668224",
+    "extra_special_token_345": "179564995814",
+    "extra_special_token_346": "418412875665",
+    "extra_special_token_347": "718951851413",
+    "extra_special_token_348": "699446724178",
+    "extra_special_token_349": "624266421831",
+    "extra_special_token_350": "815458725125",
+    "extra_special_token_351": "455423278865",
+    "extra_special_token_352": "393741199486",
+    "extra_special_token_353": "328552864359",
+    "extra_special_token_354": "211662639865",
+    "extra_special_token_355": "218784516525",
+    "extra_special_token_356": "762486672996",
+    "extra_special_token_357": "142799718159",
+    "extra_special_token_358": "858146415154",
+    "extra_special_token_359": "767858144912",
+    "extra_special_token_360": "571317457151",
+    "extra_special_token_361": "635127952696",
+    "extra_special_token_362": "116427191984",
+    "extra_special_token_363": "268921994538",
+    "extra_special_token_364": "523937669294",
+    "extra_special_token_365": "165429152138",
+    "extra_special_token_366": "739246183345",
+    "extra_special_token_367": "591464355756",
+    "extra_special_token_368": "212985874612",
+    "extra_special_token_369": "191887635211",
+    "extra_special_token_370": "967214577653",
+    "extra_special_token_371": "119342152414",
+    "extra_special_token_372": "946444632795",
+    "extra_special_token_373": "618423867817",
+    "extra_special_token_374": "228565148417",
+    "extra_special_token_375": "729116422489",
+    "extra_special_token_376": "527874729936",
+    "extra_special_token_377": "739784153482",
+    "extra_special_token_378": "387763951128",
+    "extra_special_token_379": "331369926711",
+    "extra_special_token_380": "562716493614",
+    "extra_special_token_381": "739667844957",
+    "extra_special_token_382": "562389434565",
+    "extra_special_token_383": "256497188281",
+    "extra_special_token_384": "859927364588",
+    "extra_special_token_385": "417668946583",
+    "extra_special_token_386": "357621613582",
+    "extra_special_token_387": "438435178228",
+    "extra_special_token_388": "485692541169",
+    "extra_special_token_389": "825815739116",
+    "extra_special_token_390": "342221452223",
+    "extra_special_token_391": "697747991249",
+    "extra_special_token_392": "716763689965",
+    "extra_special_token_393": "141499982867",
+    "extra_special_token_394": "818479319499",
+    "extra_special_token_395": "336813343298",
+    "extra_special_token_396": "594688742928",
+    "extra_special_token_397": "472129283475",
+    "extra_special_token_398": "514354144759",
+    "extra_special_token_399": "349249721685",
+    "extra_special_token_400": "546276298359",
+    "extra_special_token_401": "353755529131",
+    "extra_special_token_402": "315534574435",
+    "extra_special_token_403": "523723475786",
+    "extra_special_token_404": "215826764872",
+    "extra_special_token_405": "367968398551",
+    "extra_special_token_406": "569853653352",
+    "extra_special_token_407": "389715484387",
+    "extra_special_token_408": "293847485454",
+    "extra_special_token_409": "714738141818",
+    "extra_special_token_410": "178478368922",
+    "extra_special_token_411": "581493616981",
+    "extra_special_token_412": "589439538674",
+    "extra_special_token_413": "846657726193",
+    "extra_special_token_414": "722339992679",
+    "extra_special_token_415": "138154781148",
+    "extra_special_token_416": "757785319772",
+    "extra_special_token_417": "492516914298",
+    "extra_special_token_418": "919181521716",
+    "extra_special_token_419": "985781138935",
+    "extra_special_token_420": "476969195485",
+    "extra_special_token_421": "313145133463",
+    "extra_special_token_422": "758963111966",
+    "extra_special_token_423": "147541537162",
+    "extra_special_token_424": "557163366873",
+    "extra_special_token_425": "144373897488",
+    "extra_special_token_426": "522515164754",
+    "extra_special_token_427": "724964923582",
+    "extra_special_token_428": "284776712475",
+    "extra_special_token_429": "375429755114",
+    "extra_special_token_430": "181233596124",
+    "extra_special_token_431": "948585673431",
+    "extra_special_token_432": "243165586174",
+    "extra_special_token_433": "396847976144",
+    "extra_special_token_434": "997724962668",
+    "extra_special_token_435": "558837194455",
+    "extra_special_token_436": "163165456396",
+    "extra_special_token_437": "378749551722",
+    "extra_special_token_438": "161238482259",
+    "extra_special_token_439": "754978243758",
+    "extra_special_token_440": "195388849133",
+    "extra_special_token_441": "229775525672",
+    "extra_special_token_442": "262437452884",
+    "extra_special_token_443": "441377892146",
+    "extra_special_token_444": "451885565366",
+    "extra_special_token_445": "981277526855",
+    "extra_special_token_446": "762495822823",
+    "extra_special_token_447": "368763327262",
+    "extra_special_token_448": "757422791351",
+    "extra_special_token_449": "636324136426",
+    "extra_special_token_450": "214193645583",
+    "extra_special_token_451": "412843856172",
+    "extra_special_token_452": "179386156569",
+    "extra_special_token_453": "756916173536",
+    "extra_special_token_454": "892697125149",
+    "extra_special_token_455": "625334487352",
+    "extra_special_token_456": "941861857715",
+    "extra_special_token_457": "887417525236",
+    "extra_special_token_458": "649516938598",
+    "extra_special_token_459": "717628619782",
+    "extra_special_token_460": "438124184139",
+    "extra_special_token_461": "547563892268",
+    "extra_special_token_462": "856317483891",
+    "extra_special_token_463": "313313831273",
+    "extra_special_token_464": "371496153876",
+    "extra_special_token_465": "587541149322",
+    "extra_special_token_466": "265847332563",
+    "extra_special_token_467": "449549215429",
+    "extra_special_token_468": "163497196769",
+    "extra_special_token_469": "861342291298",
+    "extra_special_token_470": "268433315926",
+    "extra_special_token_471": "774679513717",
+    "extra_special_token_472": "851254219729",
+    "extra_special_token_473": "583527834464",
+    "extra_special_token_474": "488496781997",
+    "extra_special_token_475": "556814553861",
+    "extra_special_token_476": "482829231639",
+    "extra_special_token_477": "618878266619",
+    "extra_special_token_478": "147444452794",
+    "extra_special_token_479": "949235426629",
+    "extra_special_token_480": "357299947518",
+    "extra_special_token_481": "175528632226",
+    "extra_special_token_482": "645527857972",
+    "extra_special_token_483": "186872457894",
+    "extra_special_token_484": "552738847828",
+    "extra_special_token_485": "626748382482",
+    "extra_special_token_486": "921894985642",
+    "extra_special_token_487": "943878645871",
+    "extra_special_token_488": "859289776479",
+    "extra_special_token_489": "614583493135",
+    "extra_special_token_490": "933775286797",
+    "extra_special_token_491": "332234613346",
+    "extra_special_token_492": "325196781219",
+    "extra_special_token_493": "142526557681",
+    "extra_special_token_494": "356722692178",
+    "extra_special_token_495": "449318681694",
+    "extra_special_token_496": "687284547244",
+    "extra_special_token_497": "947262995132",
+    "extra_special_token_498": "893974619684",
+    "extra_special_token_499": "797238311233"
+  },
+  "is_local": false,
+  "model_max_length": 1000000000000000019884624838656,
+  "pad_token": "<|endoftext|>",
+  "tokenizer_class": "GPT2Tokenizer",
+  "unk_token": "<|endoftext|>",
+  "vocab_size": 49152
+}

starcoder2-7b/base/train.log

@@ -0,0 +1,49 @@
+2026-03-21 11:04:31,986 [INFO] new_opacus_codex.train_steps: epoch=1 step=10 loss=1.5025
+2026-03-21 11:05:54,380 [INFO] new_opacus_codex.train_steps: epoch=1 step=20 loss=1.3712
+2026-03-21 11:07:15,652 [INFO] new_opacus_codex.train_steps: epoch=1 step=30 loss=1.1814
+2026-03-21 11:08:38,661 [INFO] new_opacus_codex.train_steps: epoch=1 step=40 loss=1.1834
+2026-03-21 11:10:02,035 [INFO] new_opacus_codex.train_steps: epoch=1 step=50 loss=1.1336
+2026-03-21 11:10:29,195 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=1 step=50 eval_loss=0.8034 duration_sec=27.16
+2026-03-21 11:11:51,908 [INFO] new_opacus_codex.train_steps: epoch=1 step=60 loss=1.1219
+2026-03-21 11:13:13,533 [INFO] new_opacus_codex.train_steps: epoch=1 step=70 loss=1.0407
+2026-03-21 11:14:36,548 [INFO] new_opacus_codex.train_steps: epoch=1 step=80 loss=1.0129
+2026-03-21 11:15:57,878 [INFO] new_opacus_codex.train_steps: epoch=1 step=90 loss=1.0161
+2026-03-21 11:17:19,691 [INFO] new_opacus_codex.train_steps: epoch=1 step=100 loss=1.0200
+2026-03-21 11:17:46,876 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=1 step=100 eval_loss=0.7755 duration_sec=27.18
+2026-03-21 11:19:10,166 [INFO] new_opacus_codex.train_steps: epoch=1 step=110 loss=1.0177
+2026-03-21 11:20:31,874 [INFO] new_opacus_codex.train_steps: epoch=1 step=120 loss=0.9631
+2026-03-21 11:21:54,417 [INFO] new_opacus_codex.train_steps: epoch=1 step=130 loss=0.9569
+2026-03-21 11:23:16,389 [INFO] new_opacus_codex.train_steps: epoch=1 step=140 loss=0.9588
+2026-03-21 11:24:38,982 [INFO] new_opacus_codex.train_steps: epoch=1 step=150 loss=0.9273
+2026-03-21 11:25:06,151 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=1 step=150 eval_loss=0.7622 duration_sec=27.17
+2026-03-21 11:26:28,134 [INFO] new_opacus_codex.train_steps: epoch=1 step=160 loss=0.9226
+2026-03-21 11:27:49,927 [INFO] new_opacus_codex.train_steps: epoch=1 step=170 loss=0.9244
+2026-03-21 11:29:11,942 [INFO] new_opacus_codex.train_steps: epoch=1 step=180 loss=0.9250
+2026-03-21 11:30:34,185 [INFO] new_opacus_codex.train_steps: epoch=1 step=190 loss=0.8691
+2026-03-21 11:31:57,690 [INFO] new_opacus_codex.train_steps: epoch=1 step=200 loss=0.8916
+2026-03-21 11:32:24,879 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=1 step=200 eval_loss=0.7525 duration_sec=27.19
+2026-03-21 11:34:26,079 [INFO] new_opacus_codex.train_steps: epoch=2 step=210 loss=0.9168
+2026-03-21 11:35:47,787 [INFO] new_opacus_codex.train_steps: epoch=2 step=220 loss=0.8558
+2026-03-21 11:37:09,366 [INFO] new_opacus_codex.train_steps: epoch=2 step=230 loss=0.8524
+2026-03-21 11:38:31,484 [INFO] new_opacus_codex.train_steps: epoch=2 step=240 loss=0.8580
+2026-03-21 11:39:52,882 [INFO] new_opacus_codex.train_steps: epoch=2 step=250 loss=0.8491
+2026-03-21 11:40:20,073 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=2 step=250 eval_loss=0.7485 duration_sec=27.19
+2026-03-21 11:41:42,276 [INFO] new_opacus_codex.train_steps: epoch=2 step=260 loss=0.8442
+2026-03-21 11:43:04,391 [INFO] new_opacus_codex.train_steps: epoch=2 step=270 loss=0.8571
+2026-03-21 11:44:28,058 [INFO] new_opacus_codex.train_steps: epoch=2 step=280 loss=0.8479
+2026-03-21 11:45:50,712 [INFO] new_opacus_codex.train_steps: epoch=2 step=290 loss=0.8434
+2026-03-21 11:47:12,551 [INFO] new_opacus_codex.train_steps: epoch=2 step=300 loss=0.8405
+2026-03-21 11:47:39,718 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=2 step=300 eval_loss=0.7462 duration_sec=27.16
+2026-03-21 11:49:02,560 [INFO] new_opacus_codex.train_steps: epoch=2 step=310 loss=0.8507
+2026-03-21 11:50:24,762 [INFO] new_opacus_codex.train_steps: epoch=2 step=320 loss=0.8479
+2026-03-21 11:51:47,919 [INFO] new_opacus_codex.train_steps: epoch=2 step=330 loss=0.8466
+2026-03-21 11:53:09,723 [INFO] new_opacus_codex.train_steps: epoch=2 step=340 loss=0.8369
+2026-03-21 11:54:31,911 [INFO] new_opacus_codex.train_steps: epoch=2 step=350 loss=0.8474
+2026-03-21 11:54:59,091 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=2 step=350 eval_loss=0.7456 duration_sec=27.18
+2026-03-21 11:56:21,565 [INFO] new_opacus_codex.train_steps: epoch=2 step=360 loss=0.8568
+2026-03-21 11:57:43,511 [INFO] new_opacus_codex.train_steps: epoch=2 step=370 loss=0.8227
+2026-03-21 11:59:06,162 [INFO] new_opacus_codex.train_steps: epoch=2 step=380 loss=0.8342
+2026-03-21 12:00:29,231 [INFO] new_opacus_codex.train_steps: epoch=2 step=390 loss=0.8265
+2026-03-21 12:01:52,289 [INFO] new_opacus_codex.train_steps: epoch=2 step=400 loss=0.8197
+2026-03-21 12:02:19,482 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=2 step=400 eval_loss=0.7455 duration_sec=27.19
+2026-03-21 12:03:41,445 [INFO] new_opacus_codex.train_steps: epoch=2 step=410 loss=0.8286

starcoder2-7b/dp3/adapter/README.md

@@ -0,0 +1,207 @@
+---
+base_model: bigcode/starcoder2-7b
+library_name: peft
+pipeline_tag: text-generation
+tags:
+- base_model:adapter:bigcode/starcoder2-7b
+- lora
+- transformers
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.18.1

starcoder2-7b/dp3/adapter/adapter_config.json

@@ -0,0 +1,46 @@
+{
+  "alora_invocation_tokens": null,
+  "alpha_pattern": {},
+  "arrow_config": null,
+  "auto_mapping": null,
+  "base_model_name_or_path": "bigcode/starcoder2-7b",
+  "bias": "none",
+  "corda_config": null,
+  "ensure_weight_tying": true,
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 32,
+  "lora_bias": false,
+  "lora_dropout": 0.05,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": [
+    "lm_head",
+    "embed_tokens"
+  ],
+  "peft_type": "LORA",
+  "peft_version": "0.18.1",
+  "qalora_group_size": 16,
+  "r": 16,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "k_proj",
+    "q_proj",
+    "v_proj",
+    "o_proj"
+  ],
+  "target_parameters": null,
+  "task_type": "CAUSAL_LM",
+  "trainable_token_indices": null,
+  "use_dora": false,
+  "use_qalora": false,
+  "use_rslora": false
+}

starcoder2-7b/dp3/adapter/adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:853fac2b8cce7de6abb5b5d69b51e3a5f349fcbb8c2008b2dfeca74d3f065d9a
+size 2804312360

starcoder2-7b/dp3/audit_results.json

@@ -0,0 +1,137 @@
+{
+  "delta": 1e-05,
+  "num_canaries": 500,
+  "num_members": 250,
+  "paper_guess_fraction": 0.2,
+  "paper_guess_steps": 20,
+  "loss": {
+    "auc": 0.525808,
+    "empirical_epsilon": {
+      "0.05": 0.05073561053723097,
+      "0.01": 0.0
+    },
+    "empirical_epsilon_details": {
+      "0.05": {
+        "epsilon": 0.05073561053723097,
+        "num_guesses": 100,
+        "correct_guesses": 60,
+        "candidate_num_guesses": [
+          5,
+          10,
+          15,
+          20,
+          25,
+          30,
+          35,
+          40,
+          45,
+          50,
+          55,
+          60,
+          65,
+          70,
+          75,
+          80,
+          85,
+          90,
+          95,
+          100
+        ],
+        "direction": "higher"
+      },
+      "0.01": {
+        "epsilon": 0.0,
+        "num_guesses": 0,
+        "correct_guesses": 0,
+        "candidate_num_guesses": [
+          5,
+          10,
+          15,
+          20,
+          25,
+          30,
+          35,
+          40,
+          45,
+          50,
+          55,
+          60,
+          65,
+          70,
+          75,
+          80,
+          85,
+          90,
+          95,
+          100
+        ],
+        "direction": "lower"
+      }
+    }
+  },
+  "embedding": {
+    "auc": 0.520928,
+    "empirical_epsilon": {
+      "0.05": 0.0,
+      "0.01": 0.0
+    },
+    "empirical_epsilon_details": {
+      "0.05": {
+        "epsilon": 0.0,
+        "num_guesses": 0,
+        "correct_guesses": 0,
+        "candidate_num_guesses": [
+          5,
+          10,
+          15,
+          20,
+          25,
+          30,
+          35,
+          40,
+          45,
+          50,
+          55,
+          60,
+          65,
+          70,
+          75,
+          80,
+          85,
+          90,
+          95,
+          100
+        ],
+        "direction": "lower"
+      },
+      "0.01": {
+        "epsilon": 0.0,
+        "num_guesses": 0,
+        "correct_guesses": 0,
+        "candidate_num_guesses": [
+          5,
+          10,
+          15,
+          20,
+          25,
+          30,
+          35,
+          40,
+          45,
+          50,
+          55,
+          60,
+          65,
+          70,
+          75,
+          80,
+          85,
+          90,
+          95,
+          100
+        ],
+        "direction": "lower"
+      }
+    }
+  }
+}

starcoder2-7b/dp3/audit_scores.npz

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4fe821d6aec2e4b9779349695c9909f92bfbca3b25bd240d418bb85f362550a5
+size 12784

starcoder2-7b/dp3/codecarbon.csv

@@ -0,0 +1,2 @@
+timestamp,project_name,run_id,experiment_id,duration,emissions,emissions_rate,cpu_power,gpu_power,ram_power,cpu_energy,gpu_energy,ram_energy,energy_consumed,water_consumed,country_name,country_iso_code,region,cloud_provider,cloud_region,os,python_version,codecarbon_version,cpu_count,cpu_model,gpu_count,gpu_model,longitude,latitude,ram_total_size,tracking_mode,cpu_utilization_percent,gpu_utilization_percent,ram_utilization_percent,ram_used_gb,on_cloud,pue,wue
+2026-03-21T13:36:12,codedp-starcoder2-7b-cpt-dp3,457cc088-8edf-418e-8c6d-170a67e73d63,5b0fa12a-3dd7-45bb-9766-cc326314d9f1,3938.0607158355415,0.5702959606126596,0.00014481644691749236,179.92904019353898,2298.550082608512,70.0,0.18975838183590477,2.514705143984827,0.07382037755078118,2.778283903371513,0.0,United States,USA,california,,,Linux-5.15.0-157-generic-x86_64-with-glibc2.39,3.12.13,3.2.5,224,Intel(R) Xeon(R) Platinum 8480C,4,4 x NVIDIA H200,-121.9552,37.3541,2015.5625190734863,machine,2.4580389144905275,95.87682458386683,2.36410650281618,47.93522234837092,N,1.0,0.0

starcoder2-7b/dp3/epochs/epoch_001/adapter/README.md

@@ -0,0 +1,207 @@
+---
+base_model: bigcode/starcoder2-7b
+library_name: peft
+pipeline_tag: text-generation
+tags:
+- base_model:adapter:bigcode/starcoder2-7b
+- lora
+- transformers
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.18.1

starcoder2-7b/dp3/epochs/epoch_001/adapter/adapter_config.json

@@ -0,0 +1,46 @@
+{
+  "alora_invocation_tokens": null,
+  "alpha_pattern": {},
+  "arrow_config": null,
+  "auto_mapping": null,
+  "base_model_name_or_path": "bigcode/starcoder2-7b",
+  "bias": "none",
+  "corda_config": null,
+  "ensure_weight_tying": true,
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 32,
+  "lora_bias": false,
+  "lora_dropout": 0.05,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": [
+    "lm_head",
+    "embed_tokens"
+  ],
+  "peft_type": "LORA",
+  "peft_version": "0.18.1",
+  "qalora_group_size": 16,
+  "r": 16,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "k_proj",
+    "q_proj",
+    "v_proj",
+    "o_proj"
+  ],
+  "target_parameters": null,
+  "task_type": "CAUSAL_LM",
+  "trainable_token_indices": null,
+  "use_dora": false,
+  "use_qalora": false,
+  "use_rslora": false
+}

starcoder2-7b/dp3/epochs/epoch_001/adapter/adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1eb0abf981ab5eb12b4b55a3c95748d96446558dfe488188b7929dae1d80c5b4
+size 2804312360

starcoder2-7b/dp3/epochs/epoch_001/audit_results.json

@@ -0,0 +1,137 @@
+{
+  "delta": 1e-05,
+  "num_canaries": 500,
+  "num_members": 250,
+  "paper_guess_fraction": 0.2,
+  "paper_guess_steps": 20,
+  "loss": {
+    "auc": 0.527168,
+    "empirical_epsilon": {
+      "0.05": 0.09125839080661535,
+      "0.01": 0.0
+    },
+    "empirical_epsilon_details": {
+      "0.05": {
+        "epsilon": 0.09125839080661535,
+        "num_guesses": 100,
+        "correct_guesses": 61,
+        "candidate_num_guesses": [
+          5,
+          10,
+          15,
+          20,
+          25,
+          30,
+          35,
+          40,
+          45,
+          50,
+          55,
+          60,
+          65,
+          70,
+          75,
+          80,
+          85,
+          90,
+          95,
+          100
+        ],
+        "direction": "higher"
+      },
+      "0.01": {
+        "epsilon": 0.0,
+        "num_guesses": 0,
+        "correct_guesses": 0,
+        "candidate_num_guesses": [
+          5,
+          10,
+          15,
+          20,
+          25,
+          30,
+          35,
+          40,
+          45,
+          50,
+          55,
+          60,
+          65,
+          70,
+          75,
+          80,
+          85,
+          90,
+          95,
+          100
+        ],
+        "direction": "lower"
+      }
+    }
+  },
+  "embedding": {
+    "auc": 0.514792,
+    "empirical_epsilon": {
+      "0.05": 0.0,
+      "0.01": 0.0
+    },
+    "empirical_epsilon_details": {
+      "0.05": {
+        "epsilon": 0.0,
+        "num_guesses": 0,
+        "correct_guesses": 0,
+        "candidate_num_guesses": [
+          5,
+          10,
+          15,
+          20,
+          25,
+          30,
+          35,
+          40,
+          45,
+          50,
+          55,
+          60,
+          65,
+          70,
+          75,
+          80,
+          85,
+          90,
+          95,
+          100
+        ],
+        "direction": "lower"
+      },
+      "0.01": {
+        "epsilon": 0.0,
+        "num_guesses": 0,
+        "correct_guesses": 0,
+        "candidate_num_guesses": [
+          5,
+          10,
+          15,
+          20,
+          25,
+          30,
+          35,
+          40,
+          45,
+          50,
+          55,
+          60,
+          65,
+          70,
+          75,
+          80,
+          85,
+          90,
+          95,
+          100
+        ],
+        "direction": "lower"
+      }
+    }
+  }
+}

starcoder2-7b/dp3/epochs/epoch_001/audit_scores.npz

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e3dedd61e7d2e9b43bbdd58fe90dba89fa6bab75a49f7c8b244e1a9d3a2cb374
+size 12784

starcoder2-7b/dp3/epochs/epoch_002/adapter/README.md

@@ -0,0 +1,207 @@
+---
+base_model: bigcode/starcoder2-7b
+library_name: peft
+pipeline_tag: text-generation
+tags:
+- base_model:adapter:bigcode/starcoder2-7b
+- lora
+- transformers
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.18.1

starcoder2-7b/dp3/epochs/epoch_002/adapter/adapter_config.json

@@ -0,0 +1,46 @@
+{
+  "alora_invocation_tokens": null,
+  "alpha_pattern": {},
+  "arrow_config": null,
+  "auto_mapping": null,
+  "base_model_name_or_path": "bigcode/starcoder2-7b",
+  "bias": "none",
+  "corda_config": null,
+  "ensure_weight_tying": true,
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 32,
+  "lora_bias": false,
+  "lora_dropout": 0.05,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": [
+    "lm_head",
+    "embed_tokens"
+  ],
+  "peft_type": "LORA",
+  "peft_version": "0.18.1",
+  "qalora_group_size": 16,
+  "r": 16,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "k_proj",
+    "q_proj",
+    "v_proj",
+    "o_proj"
+  ],
+  "target_parameters": null,
+  "task_type": "CAUSAL_LM",
+  "trainable_token_indices": null,
+  "use_dora": false,
+  "use_qalora": false,
+  "use_rslora": false
+}

starcoder2-7b/dp3/epochs/epoch_002/adapter/adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:853fac2b8cce7de6abb5b5d69b51e3a5f349fcbb8c2008b2dfeca74d3f065d9a
+size 2804312360

starcoder2-7b/dp3/epochs/epoch_002/audit_results.json

@@ -0,0 +1,137 @@
+{
+  "delta": 1e-05,
+  "num_canaries": 500,
+  "num_members": 250,
+  "paper_guess_fraction": 0.2,
+  "paper_guess_steps": 20,
+  "loss": {
+    "auc": 0.525808,
+    "empirical_epsilon": {
+      "0.05": 0.05073561053723097,
+      "0.01": 0.0
+    },
+    "empirical_epsilon_details": {
+      "0.05": {
+        "epsilon": 0.05073561053723097,
+        "num_guesses": 100,
+        "correct_guesses": 60,
+        "candidate_num_guesses": [
+          5,
+          10,
+          15,
+          20,
+          25,
+          30,
+          35,
+          40,
+          45,
+          50,
+          55,
+          60,
+          65,
+          70,
+          75,
+          80,
+          85,
+          90,
+          95,
+          100
+        ],
+        "direction": "higher"
+      },
+      "0.01": {
+        "epsilon": 0.0,
+        "num_guesses": 0,
+        "correct_guesses": 0,
+        "candidate_num_guesses": [
+          5,
+          10,
+          15,
+          20,
+          25,
+          30,
+          35,
+          40,
+          45,
+          50,
+          55,
+          60,
+          65,
+          70,
+          75,
+          80,
+          85,
+          90,
+          95,
+          100
+        ],
+        "direction": "lower"
+      }
+    }
+  },
+  "embedding": {
+    "auc": 0.520928,
+    "empirical_epsilon": {
+      "0.05": 0.0,
+      "0.01": 0.0
+    },
+    "empirical_epsilon_details": {
+      "0.05": {
+        "epsilon": 0.0,
+        "num_guesses": 0,
+        "correct_guesses": 0,
+        "candidate_num_guesses": [
+          5,
+          10,
+          15,
+          20,
+          25,
+          30,
+          35,
+          40,
+          45,
+          50,
+          55,
+          60,
+          65,
+          70,
+          75,
+          80,
+          85,
+          90,
+          95,
+          100
+        ],
+        "direction": "lower"
+      },
+      "0.01": {
+        "epsilon": 0.0,
+        "num_guesses": 0,
+        "correct_guesses": 0,
+        "candidate_num_guesses": [
+          5,
+          10,
+          15,
+          20,
+          25,
+          30,
+          35,
+          40,
+          45,
+          50,
+          55,
+          60,
+          65,
+          70,
+          75,
+          80,
+          85,
+          90,
+          95,
+          100
+        ],
+        "direction": "lower"
+      }
+    }
+  }
+}

starcoder2-7b/dp3/epochs/epoch_002/audit_scores.npz

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4fe821d6aec2e4b9779349695c9909f92bfbca3b25bd240d418bb85f362550a5
+size 12784

starcoder2-7b/dp3/metrics.jsonl

@@ -0,0 +1,30 @@
+{"timestamp": 1774096500.8353336, "event": "train_step", "step": 10, "epoch": 1, "metrics": {"train/step_loss": 1.6323321043555417, "train/step_real_loss": 1.330917976796627, "train/lr": 0.00018181818181818183, "train/step_canary_loss": 8.062500158945719, "perf/step_duration_sec": 17.560116425156593, "perf/samples_per_sec": 7.573981674145647, "perf/tokens_per_sec": 6185.665138551686, "perf/logical_batch_size": 133.0, "perf/logical_token_count": 108621.0, "perf/physical_batches": 18.0, "privacy/epsilon": 0.7131647471248268, "system/cuda_memory_allocated_gb": 16.63217782974243, "system/cuda_max_memory_allocated_gb": 74.69534254074097}}
+{"timestamp": 1774096676.393492, "event": "train_step", "step": 20, "epoch": 1, "metrics": {"train/step_loss": 1.826857232699429, "train/step_real_loss": 1.3395854756236076, "train/lr": 0.00019897180218885507, "train/step_canary_loss": 8.756944444444445, "perf/step_duration_sec": 18.843991773203015, "perf/samples_per_sec": 7.217154498729827, "perf/tokens_per_sec": 5434.1989336686165, "perf/logical_batch_size": 136.0, "perf/logical_token_count": 102402.0, "perf/physical_batches": 18.0, "privacy/epsilon": 0.9542480237478311, "system/cuda_memory_allocated_gb": 16.916476249694824, "system/cuda_max_memory_allocated_gb": 74.69534397125244}}
+{"timestamp": 1774096852.7017646, "event": "train_step", "step": 30, "epoch": 1, "metrics": {"train/step_loss": 1.5258880311792546, "train/step_real_loss": 1.3621462360024452, "train/lr": 0.00019544467510209388, "train/step_canary_loss": 6.765625476837158, "perf/step_duration_sec": 18.006671600043774, "perf/samples_per_sec": 7.275081309289916, "perf/tokens_per_sec": 5874.1561099910805, "perf/logical_batch_size": 131.0, "perf/logical_token_count": 105774.0, "perf/physical_batches": 18.0, "privacy/epsilon": 1.145336977893831, "system/cuda_memory_allocated_gb": 16.442769050598145, "system/cuda_max_memory_allocated_gb": 74.69534397125244}}
+{"timestamp": 1774097029.0227332, "event": "train_step", "step": 40, "epoch": 1, "metrics": {"train/step_loss": 1.6870967794347693, "train/step_real_loss": 1.2007466033101082, "train/lr": 0.00018949541262593762, "train/step_canary_loss": 10.580357142857142, "perf/step_duration_sec": 17.532358843833208, "perf/samples_per_sec": 7.643010344106256, "perf/tokens_per_sec": 6436.156195815627, "perf/logical_batch_size": 134.0, "perf/logical_token_count": 112841.0, "perf/physical_batches": 18.0, "privacy/epsilon": 1.311193108872294, "system/cuda_memory_allocated_gb": 16.726882934570312, "system/cuda_max_memory_allocated_gb": 74.69534683227539}}
+{"timestamp": 1774097204.3534977, "event": "train_step", "step": 50, "epoch": 1, "metrics": {"train/step_loss": 1.5818433902881763, "train/step_real_loss": 1.2425691708922386, "train/lr": 0.00018127499143005268, "train/step_canary_loss": 7.785714830671038, "perf/step_duration_sec": 17.701514894142747, "perf/samples_per_sec": 7.569973575783576, "perf/tokens_per_sec": 6216.078152520672, "perf/logical_batch_size": 134.0, "perf/logical_token_count": 110034.0, "perf/physical_batches": 18.0, "privacy/epsilon": 1.4583641061524852, "system/cuda_memory_allocated_gb": 16.726882934570312, "system/cuda_max_memory_allocated_gb": 74.69534683227539}}
+{"timestamp": 1774097231.3920577, "event": "eval_step", "step": 50, "epoch": 1, "metrics": {"eval/loss": 0.8423196817266531, "eval/duration_sec": 27.033981669694185}}
+{"timestamp": 1774097408.5970793, "event": "train_step", "step": 60, "epoch": 1, "metrics": {"train/step_loss": 1.6164713336112804, "train/step_real_loss": 1.3143803551793098, "train/lr": 0.0001709920242324663, "train/step_canary_loss": 9.350000381469727, "perf/step_duration_sec": 17.616566620767117, "perf/samples_per_sec": 7.549711749349289, "perf/tokens_per_sec": 5881.736335493052, "perf/logical_batch_size": 133.0, "perf/logical_token_count": 103616.0, "perf/physical_batches": 17.0, "privacy/epsilon": 1.5929146459722179, "system/cuda_memory_allocated_gb": 16.63217782974243, "system/cuda_max_memory_allocated_gb": 74.69534969329834}}
+{"timestamp": 1774097584.0989442, "event": "train_step", "step": 70, "epoch": 1, "metrics": {"train/step_loss": 1.5537053346633911, "train/step_real_loss": 1.274133626371622, "train/lr": 0.00015890746575622231, "train/step_canary_loss": 10.5, "perf/step_duration_sec": 17.34652972780168, "perf/samples_per_sec": 7.609591201889827, "perf/tokens_per_sec": 6165.613622913036, "perf/logical_batch_size": 132.0, "perf/logical_token_count": 106952.0, "perf/physical_batches": 17.0, "privacy/epsilon": 1.7182200620091768, "system/cuda_memory_allocated_gb": 16.537474155426025, "system/cuda_max_memory_allocated_gb": 74.69534969329834}}
+{"timestamp": 1774097758.6071105, "event": "train_step", "step": 80, "epoch": 1, "metrics": {"train/step_loss": 1.6813924115403254, "train/step_real_loss": 1.3261733427643776, "train/lr": 0.00014532799038330385, "train/step_canary_loss": 10.77500057220459, "perf/step_duration_sec": 16.975525552406907, "perf/samples_per_sec": 7.834808977748694, "perf/tokens_per_sec": 6001.228043602786, "perf/logical_batch_size": 133.0, "perf/logical_token_count": 101874.0, "perf/physical_batches": 17.0, "privacy/epsilon": 1.8360751937320303, "system/cuda_memory_allocated_gb": 16.63217782974243, "system/cuda_max_memory_allocated_gb": 74.69534969329834}}
+{"timestamp": 1774097934.2494926, "event": "train_step", "step": 90, "epoch": 1, "metrics": {"train/step_loss": 1.8558794460157408, "train/step_real_loss": 1.2407655008137226, "train/lr": 0.00013059820956358998, "train/step_canary_loss": 10.604166666666666, "perf/step_duration_sec": 17.994612561538815, "perf/samples_per_sec": 7.557817626520207, "perf/tokens_per_sec": 6203.967971981345, "perf/logical_batch_size": 136.0, "perf/logical_token_count": 111638.0, "perf/physical_batches": 18.0, "privacy/epsilon": 1.9476934830264792, "system/cuda_memory_allocated_gb": 16.916476249694824, "system/cuda_max_memory_allocated_gb": 74.69534969329834}}
+{"timestamp": 1774098110.4647262, "event": "train_step", "step": 100, "epoch": 1, "metrics": {"train/step_loss": 1.86213939777319, "train/step_real_loss": 1.2256366163492203, "train/lr": 0.00011509192648058249, "train/step_canary_loss": 10.009375, "perf/step_duration_sec": 17.502219082787633, "perf/samples_per_sec": 7.884714466619527, "perf/tokens_per_sec": 6282.689039593839, "perf/logical_batch_size": 138.0, "perf/logical_token_count": 109961.0, "perf/physical_batches": 18.0, "privacy/epsilon": 2.054057754637441, "system/cuda_memory_allocated_gb": 16.34807252883911, "system/cuda_max_memory_allocated_gb": 74.69534969329834}}
+{"timestamp": 1774098137.4743512, "event": "eval_step", "step": 100, "epoch": 1, "metrics": {"eval/loss": 0.8425039929905157, "eval/duration_sec": 27.006801065057516}}
+{"timestamp": 1774098228.5487797, "event": "train_epoch", "step": 104, "epoch": 1, "metrics": {"train/epoch_loss": 1.7094427731604633, "train/epoch_real_loss": 1.285309421313279, "train/epoch_canary_loss": 9.719154772066274, "perf/epoch_duration_sec": 1878.0538540109992, "perf/epoch_samples_per_sec": 29.64824458099586, "perf/epoch_tokens_per_sec": 23429.532069074678, "perf/epoch_samples": 55681.0, "perf/epoch_tokens": 44001923.0, "system/cuda_epoch_peak_memory_gb": 74.69534969329834, "eval/loss": 0.8425441932404178, "eval/duration_sec": 26.999734753742814, "privacy/epsilon": 2.0952814257505974}}
+{"timestamp": 1774098238.4572744, "event": "audit_epoch", "step": 104, "epoch": 1, "metrics": {"audit/delta": 1e-05, "audit/num_canaries": 500.0, "audit/num_members": 250.0, "audit/paper_guess_fraction": 0.2, "audit/paper_guess_steps": 20.0, "audit/loss/auc": 0.527168, "audit/loss/empirical_epsilon/0.05": 0.09125839080661535, "audit/loss/empirical_epsilon/0.01": 0.0, "audit/loss/empirical_epsilon_details/0.05/epsilon": 0.09125839080661535, "audit/loss/empirical_epsilon_details/0.05/num_guesses": 100.0, "audit/loss/empirical_epsilon_details/0.05/correct_guesses": 61.0, "audit/loss/empirical_epsilon_details/0.01/epsilon": 0.0, "audit/loss/empirical_epsilon_details/0.01/num_guesses": 0.0, "audit/loss/empirical_epsilon_details/0.01/correct_guesses": 0.0, "audit/embedding/auc": 0.514792, "audit/embedding/empirical_epsilon/0.05": 0.0, "audit/embedding/empirical_epsilon/0.01": 0.0, "audit/embedding/empirical_epsilon_details/0.05/epsilon": 0.0, "audit/embedding/empirical_epsilon_details/0.05/num_guesses": 0.0, "audit/embedding/empirical_epsilon_details/0.05/correct_guesses": 0.0, "audit/embedding/empirical_epsilon_details/0.01/epsilon": 0.0, "audit/embedding/empirical_epsilon_details/0.01/num_guesses": 0.0, "audit/embedding/empirical_epsilon_details/0.01/correct_guesses": 0.0, "perf/audit_duration_sec": 5.7471471559256315}}
+{"timestamp": 1774098344.630848, "event": "train_step", "step": 110, "epoch": 2, "metrics": {"train/step_loss": 1.570878519940732, "train/step_real_loss": 1.2690251618623734, "train/lr": 9.920264990753837e-05, "train/step_canary_loss": 8.010416825612387, "perf/step_duration_sec": 17.30536743812263, "perf/samples_per_sec": 7.685476802243991, "perf/tokens_per_sec": 6186.173184867881, "perf/logical_batch_size": 133.0, "perf/logical_token_count": 107054.0, "perf/physical_batches": 18.0, "privacy/epsilon": 2.155895236118489, "system/cuda_memory_allocated_gb": 16.63217782974243, "system/cuda_max_memory_allocated_gb": 74.69534254074097}}
+{"timestamp": 1774098520.7483177, "event": "train_step", "step": 120, "epoch": 2, "metrics": {"train/step_loss": 1.733701615437974, "train/step_real_loss": 1.2730829790234566, "train/lr": 8.333360798744496e-05, "train/step_canary_loss": 8.284722222222221, "perf/step_duration_sec": 18.148367216810584, "perf/samples_per_sec": 7.493787092539381, "perf/tokens_per_sec": 5505.233545608104, "perf/logical_batch_size": 136.0, "perf/logical_token_count": 99911.0, "perf/physical_batches": 18.0, "privacy/epsilon": 2.2538737991591966, "system/cuda_memory_allocated_gb": 16.916476249694824, "system/cuda_max_memory_allocated_gb": 74.69534254074097}}
+{"timestamp": 1774098695.815589, "event": "train_step", "step": 130, "epoch": 2, "metrics": {"train/step_loss": 1.4695805564071194, "train/step_real_loss": 1.2108174487948418, "train/lr": 6.788751536089739e-05, "train/step_canary_loss": 9.75, "perf/step_duration_sec": 16.7934517133981, "perf/samples_per_sec": 7.860206600331495, "perf/tokens_per_sec": 6584.88807942771, "perf/logical_batch_size": 132.0, "perf/logical_token_count": 110583.0, "perf/physical_batches": 17.0, "privacy/epsilon": 2.34829265801524, "system/cuda_memory_allocated_gb": 16.537474155426025, "system/cuda_max_memory_allocated_gb": 74.69534254074097}}
+{"timestamp": 1774098871.6479974, "event": "train_step", "step": 140, "epoch": 2, "metrics": {"train/step_loss": 1.409767813354958, "train/step_real_loss": 1.280211590230465, "train/lr": 5.325635332531864e-05, "train/step_canary_loss": 6.9375, "perf/step_duration_sec": 17.296851823106408, "perf/samples_per_sec": 7.515818562215838, "perf/tokens_per_sec": 5869.218343212228, "perf/logical_batch_size": 130.0, "perf/logical_token_count": 101519.0, "perf/physical_batches": 18.0, "privacy/epsilon": 2.4397183333948855, "system/cuda_memory_allocated_gb": 16.348063945770264, "system/cuda_max_memory_allocated_gb": 74.69534254074097}}
+{"timestamp": 1774099047.329602, "event": "train_step", "step": 150, "epoch": 2, "metrics": {"train/step_loss": 1.470641034250041, "train/step_real_loss": 1.2892888560891151, "train/lr": 3.981142237826332e-05, "train/step_canary_loss": 9.208333969116211, "perf/step_duration_sec": 17.535365080460906, "perf/samples_per_sec": 7.4706172012334715, "perf/tokens_per_sec": 5992.347437184814, "perf/logical_batch_size": 131.0, "perf/logical_token_count": 105078.0, "perf/physical_batches": 17.0, "privacy/epsilon": 2.5283910517887054, "system/cuda_memory_allocated_gb": 16.442769050598145, "system/cuda_max_memory_allocated_gb": 74.69534254074097}}
+{"timestamp": 1774099074.3730319, "event": "eval_step", "step": 150, "epoch": 2, "metrics": {"eval/loss": 0.842570746081999, "eval/duration_sec": 27.040043698623776}}
+{"timestamp": 1774099249.8653634, "event": "train_step", "step": 160, "epoch": 2, "metrics": {"train/step_loss": 1.536541504646415, "train/step_real_loss": 1.2477270364761353, "train/lr": 2.789391958515183e-05, "train/step_canary_loss": 7.697916825612386, "perf/step_duration_sec": 17.38564824871719, "perf/samples_per_sec": 7.649987972683935, "perf/tokens_per_sec": 6056.662282222899, "perf/logical_batch_size": 133.0, "perf/logical_token_count": 105299.0, "perf/physical_batches": 18.0, "privacy/epsilon": 2.6145698431381854, "system/cuda_memory_allocated_gb": 16.63217782974243, "system/cuda_max_memory_allocated_gb": 74.69534540176392}}
+{"timestamp": 1774099424.502899, "event": "train_step", "step": 170, "epoch": 2, "metrics": {"train/step_loss": 1.4997789441174223, "train/step_real_loss": 1.2844417318701744, "train/lr": 1.7806279893114875e-05, "train/step_canary_loss": 10.6875, "perf/step_duration_sec": 17.294984631240368, "perf/samples_per_sec": 7.574450211616343, "perf/tokens_per_sec": 5730.794337970551, "perf/logical_batch_size": 131.0, "perf/logical_token_count": 99114.0, "perf/physical_batches": 17.0, "privacy/epsilon": 2.6985357679843074, "system/cuda_memory_allocated_gb": 16.442769050598145, "system/cuda_max_memory_allocated_gb": 74.69534540176392}}
+{"timestamp": 1774099602.7806635, "event": "train_step", "step": 180, "epoch": 2, "metrics": {"train/step_loss": 1.5966231734664351, "train/step_real_loss": 1.2403349727392197, "train/lr": 9.804501125681243e-06, "train/step_canary_loss": 8.111607415335518, "perf/step_duration_sec": 17.99831911176443, "perf/samples_per_sec": 7.445139691539982, "perf/tokens_per_sec": 5983.892125215344, "perf/logical_batch_size": 134.0, "perf/logical_token_count": 107700.0, "perf/physical_batches": 18.0, "privacy/epsilon": 2.780402267783889, "system/cuda_memory_allocated_gb": 16.726882934570312, "system/cuda_max_memory_allocated_gb": 74.69534540176392}}
+{"timestamp": 1774099786.8860722, "event": "train_step", "step": 190, "epoch": 2, "metrics": {"train/step_loss": 1.301622830904447, "train/step_real_loss": 1.246277093887329, "train/lr": 4.091647429802869e-06, "train/step_canary_loss": 4.84375, "perf/step_duration_sec": 17.514903953298926, "perf/samples_per_sec": 7.3651560033649455, "perf/tokens_per_sec": 6243.596898480441, "perf/logical_batch_size": 129.0, "perf/logical_token_count": 109356.0, "perf/physical_batches": 18.0, "privacy/epsilon": 2.860377969759561, "system/cuda_memory_allocated_gb": 16.253358840942383, "system/cuda_max_memory_allocated_gb": 74.69534540176392}}
+{"timestamp": 1774099963.4199688, "event": "train_step", "step": 200, "epoch": 2, "metrics": {"train/step_loss": 1.5310632604541201, "train/step_real_loss": 1.2371121123433113, "train/lr": 8.126960406835249e-07, "train/step_canary_loss": 10.9375, "perf/step_duration_sec": 16.793452875688672, "perf/samples_per_sec": 7.860206056319249, "perf/tokens_per_sec": 6343.96039865215, "perf/logical_batch_size": 132.0, "perf/logical_token_count": 106537.0, "perf/physical_batches": 17.0, "privacy/epsilon": 2.938565800812133, "system/cuda_memory_allocated_gb": 16.537474155426025, "system/cuda_max_memory_allocated_gb": 74.69534540176392}}
+{"timestamp": 1774099990.6645026, "event": "eval_step", "step": 200, "epoch": 2, "metrics": {"eval/loss": 0.8425621900750303, "eval/duration_sec": 27.24077794700861}}
+{"timestamp": 1774100151.6417675, "event": "train_epoch", "step": 208, "epoch": 2, "metrics": {"train/epoch_loss": 1.6631105757675884, "train/epoch_real_loss": 1.2492633356516678, "train/epoch_canary_loss": 9.74998009501868, "perf/epoch_duration_sec": 1886.0955602359027, "perf/epoch_samples_per_sec": 29.488961838732855, "perf/epoch_tokens_per_sec": 23327.959583605018, "perf/epoch_samples": 55619.0, "perf/epoch_tokens": 43998761.0, "system/cuda_epoch_peak_memory_gb": 74.69534540176392, "eval/loss": 0.8425623961221212, "eval/duration_sec": 27.067450830712914, "privacy/epsilon": 2.9999680995370417}}
+{"timestamp": 1774100161.8989484, "event": "audit_epoch", "step": 208, "epoch": 2, "metrics": {"audit/delta": 1e-05, "audit/num_canaries": 500.0, "audit/num_members": 250.0, "audit/paper_guess_fraction": 0.2, "audit/paper_guess_steps": 20.0, "audit/loss/auc": 0.525808, "audit/loss/empirical_epsilon/0.05": 0.05073561053723097, "audit/loss/empirical_epsilon/0.01": 0.0, "audit/loss/empirical_epsilon_details/0.05/epsilon": 0.05073561053723097, "audit/loss/empirical_epsilon_details/0.05/num_guesses": 100.0, "audit/loss/empirical_epsilon_details/0.05/correct_guesses": 60.0, "audit/loss/empirical_epsilon_details/0.01/epsilon": 0.0, "audit/loss/empirical_epsilon_details/0.01/num_guesses": 0.0, "audit/loss/empirical_epsilon_details/0.01/correct_guesses": 0.0, "audit/embedding/auc": 0.520928, "audit/embedding/empirical_epsilon/0.05": 0.0, "audit/embedding/empirical_epsilon/0.01": 0.0, "audit/embedding/empirical_epsilon_details/0.05/epsilon": 0.0, "audit/embedding/empirical_epsilon_details/0.05/num_guesses": 0.0, "audit/embedding/empirical_epsilon_details/0.05/correct_guesses": 0.0, "audit/embedding/empirical_epsilon_details/0.01/epsilon": 0.0, "audit/embedding/empirical_epsilon_details/0.01/num_guesses": 0.0, "audit/embedding/empirical_epsilon_details/0.01/correct_guesses": 0.0, "perf/audit_duration_sec": 5.841585950925946}}
+{"timestamp": 1774100172.4001799, "event": "audit_final", "step": 208, "epoch": 2, "metrics": {"audit/delta": 1e-05, "audit/num_canaries": 500.0, "audit/num_members": 250.0, "audit/paper_guess_fraction": 0.2, "audit/paper_guess_steps": 20.0, "audit/loss/auc": 0.525808, "audit/loss/empirical_epsilon/0.05": 0.05073561053723097, "audit/loss/empirical_epsilon/0.01": 0.0, "audit/loss/empirical_epsilon_details/0.05/epsilon": 0.05073561053723097, "audit/loss/empirical_epsilon_details/0.05/num_guesses": 100.0, "audit/loss/empirical_epsilon_details/0.05/correct_guesses": 60.0, "audit/loss/empirical_epsilon_details/0.01/epsilon": 0.0, "audit/loss/empirical_epsilon_details/0.01/num_guesses": 0.0, "audit/loss/empirical_epsilon_details/0.01/correct_guesses": 0.0, "audit/embedding/auc": 0.520928, "audit/embedding/empirical_epsilon/0.05": 0.0, "audit/embedding/empirical_epsilon/0.01": 0.0, "audit/embedding/empirical_epsilon_details/0.05/epsilon": 0.0, "audit/embedding/empirical_epsilon_details/0.05/num_guesses": 0.0, "audit/embedding/empirical_epsilon_details/0.05/correct_guesses": 0.0, "audit/embedding/empirical_epsilon_details/0.01/epsilon": 0.0, "audit/embedding/empirical_epsilon_details/0.01/num_guesses": 0.0, "audit/embedding/empirical_epsilon_details/0.01/correct_guesses": 0.0}}
+{"timestamp": 1774100172.924808, "event": "energy_final", "step": 208, "epoch": null, "metrics": {"energy/codecarbon/duration": 3938.0607158355415, "energy/codecarbon/emissions": 0.5702959606126596, "energy/codecarbon/emissions_rate": 0.00014481644691749236, "energy/codecarbon/cpu_power": 179.92904019353898, "energy/codecarbon/gpu_power": 2298.550082608512, "energy/codecarbon/ram_power": 70.0, "energy/codecarbon/cpu_energy": 0.18975838183590477, "energy/codecarbon/gpu_energy": 2.514705143984827, "energy/codecarbon/ram_energy": 0.07382037755078118, "energy/codecarbon/energy_consumed": 2.778283903371513, "energy/codecarbon/water_consumed": 0.0, "energy/codecarbon/cpu_count": 224.0, "energy/codecarbon/gpu_count": 4.0, "energy/codecarbon/longitude": -121.9552, "energy/codecarbon/latitude": 37.3541, "energy/codecarbon/ram_total_size": 2015.5625190734863, "energy/codecarbon/cpu_utilization_percent": 2.4580389144905275, "energy/codecarbon/gpu_utilization_percent": 95.87682458386683, "energy/codecarbon/ram_utilization_percent": 2.36410650281618, "energy/codecarbon/ram_used_gb": 47.93522234837092, "energy/codecarbon/pue": 1.0, "energy/codecarbon/wue": 0.0}}

starcoder2-7b/dp3/pretrain_lm_head.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c9a9b40bb4ecc60cfc3a7e2f7cd31be2121d8f32f92d1a4ecdac554172fb4e5b
+size 457594472

starcoder2-7b/dp3/resolved_config.yaml

@@ -0,0 +1,101 @@
+model:
+  name: bigcode/starcoder2-7b
+  tokenizer_name: bigcode/starcoder2-7b
+  max_length: 1024
+  dtype: bfloat16
+  trust_remote_code: true
+  use_fast_tokenizer: true
+  cache_dir: null
+  local_files_only: false
+  low_cpu_mem_usage: true
+  tie_word_embeddings: true
+  gradient_checkpointing: false
+  use_chat_template: false
+dataset:
+  name: melihcatal/codedp-cpt
+  split: train
+  mode: cpt
+  text_column: text
+  validation_ratio: 0.05
+  max_samples: -1
+lora:
+  enabled: true
+  r: 16
+  alpha: 32
+  dropout: 0.05
+  target_modules:
+  - q_proj
+  - k_proj
+  - v_proj
+  - o_proj
+  modules_to_save:
+  - lm_head
+  bias: none
+training:
+  seed: 42
+  epochs: 2
+  warmup_steps: null
+  warmup_ratio: 0.05
+  mixed_precision: false
+  mixed_precision_dtype: bfloat16
+  batch_size: 8
+  eval_batch_size: 8
+  eval_every_steps: 50
+  eval_every_epochs: 1
+  learning_rate: 0.0002
+  optimizer: adamw
+  lr_scheduler: cosine
+  adam_beta1: 0.9
+  adam_beta2: 0.999
+  adam_epsilon: 1.0e-08
+  sgd_momentum: 0.9
+  weight_decay: 0.01
+  max_grad_norm: 1.0
+  log_every: 10
+  gradient_accumulation_steps: 16
+  num_workers: 4
+  output_dir: runs/cpt/starcoder2-7b/dp3
+distributed:
+  strategy: dpddp
+  backend: nccl
+  devices: null
+dp:
+  module_validator: auto
+  target_delta: 1.0e-05
+  noise_multiplier: null
+  max_grad_norm: 1.0
+  grad_sample_mode: hooks
+  clipping: flat
+  secure_mode: false
+  enabled: true
+  target_epsilon: 3.0
+audit:
+  enabled: true
+  run_every_epoch: true
+  epoch_device: cuda
+  q_canary: auto
+  num_canaries: 500
+  prefix_length: 49
+  num_digits: 12
+  batch_size: 32
+  delta: 1.0e-05
+  p_values:
+  - 0.05
+  - 0.01
+  paper_guess_fraction: 0.2
+  paper_guess_steps: 20
+  enable_holdout_empirical_epsilon: false
+  holdout_seed: 42
+  tie_seed: 42
+tracking:
+  enabled: true
+  tensorboard: true
+  wandb: false
+  wandb_project: codedp-finetune-h200-audit
+  wandb_run_name: starcoder2-7b-cpt-dp3
+  wandb_mode: online
+  codecarbon: true
+  codecarbon_output_file: codecarbon.csv
+  codecarbon_measure_power_secs: 15
+  codecarbon_country_iso_code: null
+  codecarbon_project_name: codedp-starcoder2-7b-cpt-dp3

starcoder2-7b/dp3/scalars.csv

@@ -0,0 +1,386 @@
+timestamp,event,step,epoch,key,value
+1774096500.8353336,train_step,10,1,train/step_loss,1.6323321043555417
+1774096500.8353336,train_step,10,1,train/step_real_loss,1.330917976796627
+1774096500.8353336,train_step,10,1,train/lr,0.00018181818181818183
+1774096500.8353336,train_step,10,1,train/step_canary_loss,8.062500158945719
+1774096500.8353336,train_step,10,1,perf/step_duration_sec,17.560116425156593
+1774096500.8353336,train_step,10,1,perf/samples_per_sec,7.573981674145647
+1774096500.8353336,train_step,10,1,perf/tokens_per_sec,6185.665138551686
+1774096500.8353336,train_step,10,1,perf/logical_batch_size,133.0
+1774096500.8353336,train_step,10,1,perf/logical_token_count,108621.0
+1774096500.8353336,train_step,10,1,perf/physical_batches,18.0
+1774096500.8353336,train_step,10,1,privacy/epsilon,0.7131647471248268
+1774096500.8353336,train_step,10,1,system/cuda_memory_allocated_gb,16.63217782974243
+1774096500.8353336,train_step,10,1,system/cuda_max_memory_allocated_gb,74.69534254074097
+1774096676.393492,train_step,20,1,train/step_loss,1.826857232699429
+1774096676.393492,train_step,20,1,train/step_real_loss,1.3395854756236076
+1774096676.393492,train_step,20,1,train/lr,0.00019897180218885507
+1774096676.393492,train_step,20,1,train/step_canary_loss,8.756944444444445
+1774096676.393492,train_step,20,1,perf/step_duration_sec,18.843991773203015
+1774096676.393492,train_step,20,1,perf/samples_per_sec,7.217154498729827
+1774096676.393492,train_step,20,1,perf/tokens_per_sec,5434.1989336686165
+1774096676.393492,train_step,20,1,perf/logical_batch_size,136.0
+1774096676.393492,train_step,20,1,perf/logical_token_count,102402.0
+1774096676.393492,train_step,20,1,perf/physical_batches,18.0
+1774096676.393492,train_step,20,1,privacy/epsilon,0.9542480237478311
+1774096676.393492,train_step,20,1,system/cuda_memory_allocated_gb,16.916476249694824
+1774096676.393492,train_step,20,1,system/cuda_max_memory_allocated_gb,74.69534397125244
+1774096852.7017646,train_step,30,1,train/step_loss,1.5258880311792546
+1774096852.7017646,train_step,30,1,train/step_real_loss,1.3621462360024452
+1774096852.7017646,train_step,30,1,train/lr,0.00019544467510209388
+1774096852.7017646,train_step,30,1,train/step_canary_loss,6.765625476837158
+1774096852.7017646,train_step,30,1,perf/step_duration_sec,18.006671600043774
+1774096852.7017646,train_step,30,1,perf/samples_per_sec,7.275081309289916
+1774096852.7017646,train_step,30,1,perf/tokens_per_sec,5874.1561099910805
+1774096852.7017646,train_step,30,1,perf/logical_batch_size,131.0
+1774096852.7017646,train_step,30,1,perf/logical_token_count,105774.0
+1774096852.7017646,train_step,30,1,perf/physical_batches,18.0
+1774096852.7017646,train_step,30,1,privacy/epsilon,1.145336977893831
+1774096852.7017646,train_step,30,1,system/cuda_memory_allocated_gb,16.442769050598145
+1774096852.7017646,train_step,30,1,system/cuda_max_memory_allocated_gb,74.69534397125244
+1774097029.0227332,train_step,40,1,train/step_loss,1.6870967794347693
+1774097029.0227332,train_step,40,1,train/step_real_loss,1.2007466033101082
+1774097029.0227332,train_step,40,1,train/lr,0.00018949541262593762
+1774097029.0227332,train_step,40,1,train/step_canary_loss,10.580357142857142
+1774097029.0227332,train_step,40,1,perf/step_duration_sec,17.532358843833208
+1774097029.0227332,train_step,40,1,perf/samples_per_sec,7.643010344106256
+1774097029.0227332,train_step,40,1,perf/tokens_per_sec,6436.156195815627
+1774097029.0227332,train_step,40,1,perf/logical_batch_size,134.0
+1774097029.0227332,train_step,40,1,perf/logical_token_count,112841.0
+1774097029.0227332,train_step,40,1,perf/physical_batches,18.0
+1774097029.0227332,train_step,40,1,privacy/epsilon,1.311193108872294
+1774097029.0227332,train_step,40,1,system/cuda_memory_allocated_gb,16.726882934570312
+1774097029.0227332,train_step,40,1,system/cuda_max_memory_allocated_gb,74.69534683227539
+1774097204.3534977,train_step,50,1,train/step_loss,1.5818433902881763
+1774097204.3534977,train_step,50,1,train/step_real_loss,1.2425691708922386
+1774097204.3534977,train_step,50,1,train/lr,0.00018127499143005268
+1774097204.3534977,train_step,50,1,train/step_canary_loss,7.785714830671038
+1774097204.3534977,train_step,50,1,perf/step_duration_sec,17.701514894142747
+1774097204.3534977,train_step,50,1,perf/samples_per_sec,7.569973575783576
+1774097204.3534977,train_step,50,1,perf/tokens_per_sec,6216.078152520672
+1774097204.3534977,train_step,50,1,perf/logical_batch_size,134.0
+1774097204.3534977,train_step,50,1,perf/logical_token_count,110034.0
+1774097204.3534977,train_step,50,1,perf/physical_batches,18.0
+1774097204.3534977,train_step,50,1,privacy/epsilon,1.4583641061524852
+1774097204.3534977,train_step,50,1,system/cuda_memory_allocated_gb,16.726882934570312
+1774097204.3534977,train_step,50,1,system/cuda_max_memory_allocated_gb,74.69534683227539
+1774097231.3920577,eval_step,50,1,eval/loss,0.8423196817266531
+1774097231.3920577,eval_step,50,1,eval/duration_sec,27.033981669694185
+1774097408.5970793,train_step,60,1,train/step_loss,1.6164713336112804
+1774097408.5970793,train_step,60,1,train/step_real_loss,1.3143803551793098
+1774097408.5970793,train_step,60,1,train/lr,0.0001709920242324663
+1774097408.5970793,train_step,60,1,train/step_canary_loss,9.350000381469727
+1774097408.5970793,train_step,60,1,perf/step_duration_sec,17.616566620767117
+1774097408.5970793,train_step,60,1,perf/samples_per_sec,7.549711749349289
+1774097408.5970793,train_step,60,1,perf/tokens_per_sec,5881.736335493052
+1774097408.5970793,train_step,60,1,perf/logical_batch_size,133.0
+1774097408.5970793,train_step,60,1,perf/logical_token_count,103616.0
+1774097408.5970793,train_step,60,1,perf/physical_batches,17.0
+1774097408.5970793,train_step,60,1,privacy/epsilon,1.5929146459722179
+1774097408.5970793,train_step,60,1,system/cuda_memory_allocated_gb,16.63217782974243
+1774097408.5970793,train_step,60,1,system/cuda_max_memory_allocated_gb,74.69534969329834
+1774097584.0989442,train_step,70,1,train/step_loss,1.5537053346633911
+1774097584.0989442,train_step,70,1,train/step_real_loss,1.274133626371622
+1774097584.0989442,train_step,70,1,train/lr,0.00015890746575622231
+1774097584.0989442,train_step,70,1,train/step_canary_loss,10.5
+1774097584.0989442,train_step,70,1,perf/step_duration_sec,17.34652972780168
+1774097584.0989442,train_step,70,1,perf/samples_per_sec,7.609591201889827
+1774097584.0989442,train_step,70,1,perf/tokens_per_sec,6165.613622913036
+1774097584.0989442,train_step,70,1,perf/logical_batch_size,132.0
+1774097584.0989442,train_step,70,1,perf/logical_token_count,106952.0
+1774097584.0989442,train_step,70,1,perf/physical_batches,17.0
+1774097584.0989442,train_step,70,1,privacy/epsilon,1.7182200620091768
+1774097584.0989442,train_step,70,1,system/cuda_memory_allocated_gb,16.537474155426025
+1774097584.0989442,train_step,70,1,system/cuda_max_memory_allocated_gb,74.69534969329834
+1774097758.6071105,train_step,80,1,train/step_loss,1.6813924115403254
+1774097758.6071105,train_step,80,1,train/step_real_loss,1.3261733427643776
+1774097758.6071105,train_step,80,1,train/lr,0.00014532799038330385
+1774097758.6071105,train_step,80,1,train/step_canary_loss,10.77500057220459
+1774097758.6071105,train_step,80,1,perf/step_duration_sec,16.975525552406907
+1774097758.6071105,train_step,80,1,perf/samples_per_sec,7.834808977748694
+1774097758.6071105,train_step,80,1,perf/tokens_per_sec,6001.228043602786
+1774097758.6071105,train_step,80,1,perf/logical_batch_size,133.0
+1774097758.6071105,train_step,80,1,perf/logical_token_count,101874.0
+1774097758.6071105,train_step,80,1,perf/physical_batches,17.0
+1774097758.6071105,train_step,80,1,privacy/epsilon,1.8360751937320303
+1774097758.6071105,train_step,80,1,system/cuda_memory_allocated_gb,16.63217782974243
+1774097758.6071105,train_step,80,1,system/cuda_max_memory_allocated_gb,74.69534969329834
+1774097934.2494926,train_step,90,1,train/step_loss,1.8558794460157408
+1774097934.2494926,train_step,90,1,train/step_real_loss,1.2407655008137226
+1774097934.2494926,train_step,90,1,train/lr,0.00013059820956358998
+1774097934.2494926,train_step,90,1,train/step_canary_loss,10.604166666666666
+1774097934.2494926,train_step,90,1,perf/step_duration_sec,17.994612561538815
+1774097934.2494926,train_step,90,1,perf/samples_per_sec,7.557817626520207
+1774097934.2494926,train_step,90,1,perf/tokens_per_sec,6203.967971981345
+1774097934.2494926,train_step,90,1,perf/logical_batch_size,136.0
+1774097934.2494926,train_step,90,1,perf/logical_token_count,111638.0
+1774097934.2494926,train_step,90,1,perf/physical_batches,18.0
+1774097934.2494926,train_step,90,1,privacy/epsilon,1.9476934830264792
+1774097934.2494926,train_step,90,1,system/cuda_memory_allocated_gb,16.916476249694824
+1774097934.2494926,train_step,90,1,system/cuda_max_memory_allocated_gb,74.69534969329834
+1774098110.4647262,train_step,100,1,train/step_loss,1.86213939777319
+1774098110.4647262,train_step,100,1,train/step_real_loss,1.2256366163492203
+1774098110.4647262,train_step,100,1,train/lr,0.00011509192648058249
+1774098110.4647262,train_step,100,1,train/step_canary_loss,10.009375
+1774098110.4647262,train_step,100,1,perf/step_duration_sec,17.502219082787633
+1774098110.4647262,train_step,100,1,perf/samples_per_sec,7.884714466619527
+1774098110.4647262,train_step,100,1,perf/tokens_per_sec,6282.689039593839
+1774098110.4647262,train_step,100,1,perf/logical_batch_size,138.0
+1774098110.4647262,train_step,100,1,perf/logical_token_count,109961.0
+1774098110.4647262,train_step,100,1,perf/physical_batches,18.0
+1774098110.4647262,train_step,100,1,privacy/epsilon,2.054057754637441
+1774098110.4647262,train_step,100,1,system/cuda_memory_allocated_gb,16.34807252883911
+1774098110.4647262,train_step,100,1,system/cuda_max_memory_allocated_gb,74.69534969329834
+1774098137.4743512,eval_step,100,1,eval/loss,0.8425039929905157
+1774098137.4743512,eval_step,100,1,eval/duration_sec,27.006801065057516
+1774098228.5487797,train_epoch,104,1,train/epoch_loss,1.7094427731604633
+1774098228.5487797,train_epoch,104,1,train/epoch_real_loss,1.285309421313279
+1774098228.5487797,train_epoch,104,1,train/epoch_canary_loss,9.719154772066274
+1774098228.5487797,train_epoch,104,1,perf/epoch_duration_sec,1878.0538540109992
+1774098228.5487797,train_epoch,104,1,perf/epoch_samples_per_sec,29.64824458099586
+1774098228.5487797,train_epoch,104,1,perf/epoch_tokens_per_sec,23429.532069074678
+1774098228.5487797,train_epoch,104,1,perf/epoch_samples,55681.0
+1774098228.5487797,train_epoch,104,1,perf/epoch_tokens,44001923.0
+1774098228.5487797,train_epoch,104,1,system/cuda_epoch_peak_memory_gb,74.69534969329834
+1774098228.5487797,train_epoch,104,1,eval/loss,0.8425441932404178
+1774098228.5487797,train_epoch,104,1,eval/duration_sec,26.999734753742814
+1774098228.5487797,train_epoch,104,1,privacy/epsilon,2.0952814257505974
+1774098238.4572744,audit_epoch,104,1,audit/delta,1e-05
+1774098238.4572744,audit_epoch,104,1,audit/num_canaries,500.0
+1774098238.4572744,audit_epoch,104,1,audit/num_members,250.0
+1774098238.4572744,audit_epoch,104,1,audit/paper_guess_fraction,0.2
+1774098238.4572744,audit_epoch,104,1,audit/paper_guess_steps,20.0
+1774098238.4572744,audit_epoch,104,1,audit/loss/auc,0.527168
+1774098238.4572744,audit_epoch,104,1,audit/loss/empirical_epsilon/0.05,0.09125839080661535
+1774098238.4572744,audit_epoch,104,1,audit/loss/empirical_epsilon/0.01,0.0
+1774098238.4572744,audit_epoch,104,1,audit/loss/empirical_epsilon_details/0.05/epsilon,0.09125839080661535
+1774098238.4572744,audit_epoch,104,1,audit/loss/empirical_epsilon_details/0.05/num_guesses,100.0
+1774098238.4572744,audit_epoch,104,1,audit/loss/empirical_epsilon_details/0.05/correct_guesses,61.0
+1774098238.4572744,audit_epoch,104,1,audit/loss/empirical_epsilon_details/0.01/epsilon,0.0
+1774098238.4572744,audit_epoch,104,1,audit/loss/empirical_epsilon_details/0.01/num_guesses,0.0
+1774098238.4572744,audit_epoch,104,1,audit/loss/empirical_epsilon_details/0.01/correct_guesses,0.0
+1774098238.4572744,audit_epoch,104,1,audit/embedding/auc,0.514792
+1774098238.4572744,audit_epoch,104,1,audit/embedding/empirical_epsilon/0.05,0.0
+1774098238.4572744,audit_epoch,104,1,audit/embedding/empirical_epsilon/0.01,0.0
+1774098238.4572744,audit_epoch,104,1,audit/embedding/empirical_epsilon_details/0.05/epsilon,0.0
+1774098238.4572744,audit_epoch,104,1,audit/embedding/empirical_epsilon_details/0.05/num_guesses,0.0
+1774098238.4572744,audit_epoch,104,1,audit/embedding/empirical_epsilon_details/0.05/correct_guesses,0.0
+1774098238.4572744,audit_epoch,104,1,audit/embedding/empirical_epsilon_details/0.01/epsilon,0.0
+1774098238.4572744,audit_epoch,104,1,audit/embedding/empirical_epsilon_details/0.01/num_guesses,0.0
+1774098238.4572744,audit_epoch,104,1,audit/embedding/empirical_epsilon_details/0.01/correct_guesses,0.0
+1774098238.4572744,audit_epoch,104,1,perf/audit_duration_sec,5.7471471559256315
+1774098344.630848,train_step,110,2,train/step_loss,1.570878519940732
+1774098344.630848,train_step,110,2,train/step_real_loss,1.2690251618623734
+1774098344.630848,train_step,110,2,train/lr,9.920264990753837e-05
+1774098344.630848,train_step,110,2,train/step_canary_loss,8.010416825612387
+1774098344.630848,train_step,110,2,perf/step_duration_sec,17.30536743812263
+1774098344.630848,train_step,110,2,perf/samples_per_sec,7.685476802243991
+1774098344.630848,train_step,110,2,perf/tokens_per_sec,6186.173184867881
+1774098344.630848,train_step,110,2,perf/logical_batch_size,133.0
+1774098344.630848,train_step,110,2,perf/logical_token_count,107054.0
+1774098344.630848,train_step,110,2,perf/physical_batches,18.0
+1774098344.630848,train_step,110,2,privacy/epsilon,2.155895236118489
+1774098344.630848,train_step,110,2,system/cuda_memory_allocated_gb,16.63217782974243
+1774098344.630848,train_step,110,2,system/cuda_max_memory_allocated_gb,74.69534254074097
+1774098520.7483177,train_step,120,2,train/step_loss,1.733701615437974
+1774098520.7483177,train_step,120,2,train/step_real_loss,1.2730829790234566
+1774098520.7483177,train_step,120,2,train/lr,8.333360798744496e-05
+1774098520.7483177,train_step,120,2,train/step_canary_loss,8.284722222222221
+1774098520.7483177,train_step,120,2,perf/step_duration_sec,18.148367216810584
+1774098520.7483177,train_step,120,2,perf/samples_per_sec,7.493787092539381
+1774098520.7483177,train_step,120,2,perf/tokens_per_sec,5505.233545608104
+1774098520.7483177,train_step,120,2,perf/logical_batch_size,136.0
+1774098520.7483177,train_step,120,2,perf/logical_token_count,99911.0
+1774098520.7483177,train_step,120,2,perf/physical_batches,18.0
+1774098520.7483177,train_step,120,2,privacy/epsilon,2.2538737991591966
+1774098520.7483177,train_step,120,2,system/cuda_memory_allocated_gb,16.916476249694824
+1774098520.7483177,train_step,120,2,system/cuda_max_memory_allocated_gb,74.69534254074097
+1774098695.815589,train_step,130,2,train/step_loss,1.4695805564071194
+1774098695.815589,train_step,130,2,train/step_real_loss,1.2108174487948418
+1774098695.815589,train_step,130,2,train/lr,6.788751536089739e-05
+1774098695.815589,train_step,130,2,train/step_canary_loss,9.75
+1774098695.815589,train_step,130,2,perf/step_duration_sec,16.7934517133981
+1774098695.815589,train_step,130,2,perf/samples_per_sec,7.860206600331495
+1774098695.815589,train_step,130,2,perf/tokens_per_sec,6584.88807942771
+1774098695.815589,train_step,130,2,perf/logical_batch_size,132.0
+1774098695.815589,train_step,130,2,perf/logical_token_count,110583.0
+1774098695.815589,train_step,130,2,perf/physical_batches,17.0
+1774098695.815589,train_step,130,2,privacy/epsilon,2.34829265801524
+1774098695.815589,train_step,130,2,system/cuda_memory_allocated_gb,16.537474155426025
+1774098695.815589,train_step,130,2,system/cuda_max_memory_allocated_gb,74.69534254074097
+1774098871.6479974,train_step,140,2,train/step_loss,1.409767813354958
+1774098871.6479974,train_step,140,2,train/step_real_loss,1.280211590230465
+1774098871.6479974,train_step,140,2,train/lr,5.325635332531864e-05
+1774098871.6479974,train_step,140,2,train/step_canary_loss,6.9375
+1774098871.6479974,train_step,140,2,perf/step_duration_sec,17.296851823106408
+1774098871.6479974,train_step,140,2,perf/samples_per_sec,7.515818562215838
+1774098871.6479974,train_step,140,2,perf/tokens_per_sec,5869.218343212228
+1774098871.6479974,train_step,140,2,perf/logical_batch_size,130.0
+1774098871.6479974,train_step,140,2,perf/logical_token_count,101519.0
+1774098871.6479974,train_step,140,2,perf/physical_batches,18.0
+1774098871.6479974,train_step,140,2,privacy/epsilon,2.4397183333948855
+1774098871.6479974,train_step,140,2,system/cuda_memory_allocated_gb,16.348063945770264
+1774098871.6479974,train_step,140,2,system/cuda_max_memory_allocated_gb,74.69534254074097
+1774099047.329602,train_step,150,2,train/step_loss,1.470641034250041
+1774099047.329602,train_step,150,2,train/step_real_loss,1.2892888560891151
+1774099047.329602,train_step,150,2,train/lr,3.981142237826332e-05
+1774099047.329602,train_step,150,2,train/step_canary_loss,9.208333969116211
+1774099047.329602,train_step,150,2,perf/step_duration_sec,17.535365080460906
+1774099047.329602,train_step,150,2,perf/samples_per_sec,7.4706172012334715
+1774099047.329602,train_step,150,2,perf/tokens_per_sec,5992.347437184814
+1774099047.329602,train_step,150,2,perf/logical_batch_size,131.0
+1774099047.329602,train_step,150,2,perf/logical_token_count,105078.0
+1774099047.329602,train_step,150,2,perf/physical_batches,17.0
+1774099047.329602,train_step,150,2,privacy/epsilon,2.5283910517887054
+1774099047.329602,train_step,150,2,system/cuda_memory_allocated_gb,16.442769050598145
+1774099047.329602,train_step,150,2,system/cuda_max_memory_allocated_gb,74.69534254074097
+1774099074.3730319,eval_step,150,2,eval/loss,0.842570746081999
+1774099074.3730319,eval_step,150,2,eval/duration_sec,27.040043698623776
+1774099249.8653634,train_step,160,2,train/step_loss,1.536541504646415
+1774099249.8653634,train_step,160,2,train/step_real_loss,1.2477270364761353
+1774099249.8653634,train_step,160,2,train/lr,2.789391958515183e-05
+1774099249.8653634,train_step,160,2,train/step_canary_loss,7.697916825612386
+1774099249.8653634,train_step,160,2,perf/step_duration_sec,17.38564824871719
+1774099249.8653634,train_step,160,2,perf/samples_per_sec,7.649987972683935
+1774099249.8653634,train_step,160,2,perf/tokens_per_sec,6056.662282222899
+1774099249.8653634,train_step,160,2,perf/logical_batch_size,133.0
+1774099249.8653634,train_step,160,2,perf/logical_token_count,105299.0
+1774099249.8653634,train_step,160,2,perf/physical_batches,18.0
+1774099249.8653634,train_step,160,2,privacy/epsilon,2.6145698431381854
+1774099249.8653634,train_step,160,2,system/cuda_memory_allocated_gb,16.63217782974243
+1774099249.8653634,train_step,160,2,system/cuda_max_memory_allocated_gb,74.69534540176392
+1774099424.502899,train_step,170,2,train/step_loss,1.4997789441174223
+1774099424.502899,train_step,170,2,train/step_real_loss,1.2844417318701744
+1774099424.502899,train_step,170,2,train/lr,1.7806279893114875e-05
+1774099424.502899,train_step,170,2,train/step_canary_loss,10.6875
+1774099424.502899,train_step,170,2,perf/step_duration_sec,17.294984631240368
+1774099424.502899,train_step,170,2,perf/samples_per_sec,7.574450211616343
+1774099424.502899,train_step,170,2,perf/tokens_per_sec,5730.794337970551
+1774099424.502899,train_step,170,2,perf/logical_batch_size,131.0
+1774099424.502899,train_step,170,2,perf/logical_token_count,99114.0
+1774099424.502899,train_step,170,2,perf/physical_batches,17.0
+1774099424.502899,train_step,170,2,privacy/epsilon,2.6985357679843074
+1774099424.502899,train_step,170,2,system/cuda_memory_allocated_gb,16.442769050598145
+1774099424.502899,train_step,170,2,system/cuda_max_memory_allocated_gb,74.69534540176392
+1774099602.7806635,train_step,180,2,train/step_loss,1.5966231734664351
+1774099602.7806635,train_step,180,2,train/step_real_loss,1.2403349727392197
+1774099602.7806635,train_step,180,2,train/lr,9.804501125681243e-06
+1774099602.7806635,train_step,180,2,train/step_canary_loss,8.111607415335518
+1774099602.7806635,train_step,180,2,perf/step_duration_sec,17.99831911176443
+1774099602.7806635,train_step,180,2,perf/samples_per_sec,7.445139691539982
+1774099602.7806635,train_step,180,2,perf/tokens_per_sec,5983.892125215344
+1774099602.7806635,train_step,180,2,perf/logical_batch_size,134.0
+1774099602.7806635,train_step,180,2,perf/logical_token_count,107700.0
+1774099602.7806635,train_step,180,2,perf/physical_batches,18.0
+1774099602.7806635,train_step,180,2,privacy/epsilon,2.780402267783889
+1774099602.7806635,train_step,180,2,system/cuda_memory_allocated_gb,16.726882934570312
+1774099602.7806635,train_step,180,2,system/cuda_max_memory_allocated_gb,74.69534540176392
+1774099786.8860722,train_step,190,2,train/step_loss,1.301622830904447
+1774099786.8860722,train_step,190,2,train/step_real_loss,1.246277093887329
+1774099786.8860722,train_step,190,2,train/lr,4.091647429802869e-06
+1774099786.8860722,train_step,190,2,train/step_canary_loss,4.84375
+1774099786.8860722,train_step,190,2,perf/step_duration_sec,17.514903953298926
+1774099786.8860722,train_step,190,2,perf/samples_per_sec,7.3651560033649455
+1774099786.8860722,train_step,190,2,perf/tokens_per_sec,6243.596898480441
+1774099786.8860722,train_step,190,2,perf/logical_batch_size,129.0
+1774099786.8860722,train_step,190,2,perf/logical_token_count,109356.0
+1774099786.8860722,train_step,190,2,perf/physical_batches,18.0
+1774099786.8860722,train_step,190,2,privacy/epsilon,2.860377969759561
+1774099786.8860722,train_step,190,2,system/cuda_memory_allocated_gb,16.253358840942383
+1774099786.8860722,train_step,190,2,system/cuda_max_memory_allocated_gb,74.69534540176392
+1774099963.4199688,train_step,200,2,train/step_loss,1.5310632604541201
+1774099963.4199688,train_step,200,2,train/step_real_loss,1.2371121123433113
+1774099963.4199688,train_step,200,2,train/lr,8.126960406835249e-07
+1774099963.4199688,train_step,200,2,train/step_canary_loss,10.9375
+1774099963.4199688,train_step,200,2,perf/step_duration_sec,16.793452875688672
+1774099963.4199688,train_step,200,2,perf/samples_per_sec,7.860206056319249
+1774099963.4199688,train_step,200,2,perf/tokens_per_sec,6343.96039865215
+1774099963.4199688,train_step,200,2,perf/logical_batch_size,132.0
+1774099963.4199688,train_step,200,2,perf/logical_token_count,106537.0
+1774099963.4199688,train_step,200,2,perf/physical_batches,17.0
+1774099963.4199688,train_step,200,2,privacy/epsilon,2.938565800812133
+1774099963.4199688,train_step,200,2,system/cuda_memory_allocated_gb,16.537474155426025
+1774099963.4199688,train_step,200,2,system/cuda_max_memory_allocated_gb,74.69534540176392
+1774099990.6645026,eval_step,200,2,eval/loss,0.8425621900750303
+1774099990.6645026,eval_step,200,2,eval/duration_sec,27.24077794700861
+1774100151.6417675,train_epoch,208,2,train/epoch_loss,1.6631105757675884
+1774100151.6417675,train_epoch,208,2,train/epoch_real_loss,1.2492633356516678
+1774100151.6417675,train_epoch,208,2,train/epoch_canary_loss,9.74998009501868
+1774100151.6417675,train_epoch,208,2,perf/epoch_duration_sec,1886.0955602359027
+1774100151.6417675,train_epoch,208,2,perf/epoch_samples_per_sec,29.488961838732855
+1774100151.6417675,train_epoch,208,2,perf/epoch_tokens_per_sec,23327.959583605018
+1774100151.6417675,train_epoch,208,2,perf/epoch_samples,55619.0
+1774100151.6417675,train_epoch,208,2,perf/epoch_tokens,43998761.0
+1774100151.6417675,train_epoch,208,2,system/cuda_epoch_peak_memory_gb,74.69534540176392
+1774100151.6417675,train_epoch,208,2,eval/loss,0.8425623961221212
+1774100151.6417675,train_epoch,208,2,eval/duration_sec,27.067450830712914
+1774100151.6417675,train_epoch,208,2,privacy/epsilon,2.9999680995370417
+1774100161.8989484,audit_epoch,208,2,audit/delta,1e-05
+1774100161.8989484,audit_epoch,208,2,audit/num_canaries,500.0
+1774100161.8989484,audit_epoch,208,2,audit/num_members,250.0
+1774100161.8989484,audit_epoch,208,2,audit/paper_guess_fraction,0.2
+1774100161.8989484,audit_epoch,208,2,audit/paper_guess_steps,20.0
+1774100161.8989484,audit_epoch,208,2,audit/loss/auc,0.525808
+1774100161.8989484,audit_epoch,208,2,audit/loss/empirical_epsilon/0.05,0.05073561053723097
+1774100161.8989484,audit_epoch,208,2,audit/loss/empirical_epsilon/0.01,0.0
+1774100161.8989484,audit_epoch,208,2,audit/loss/empirical_epsilon_details/0.05/epsilon,0.05073561053723097
+1774100161.8989484,audit_epoch,208,2,audit/loss/empirical_epsilon_details/0.05/num_guesses,100.0
+1774100161.8989484,audit_epoch,208,2,audit/loss/empirical_epsilon_details/0.05/correct_guesses,60.0
+1774100161.8989484,audit_epoch,208,2,audit/loss/empirical_epsilon_details/0.01/epsilon,0.0
+1774100161.8989484,audit_epoch,208,2,audit/loss/empirical_epsilon_details/0.01/num_guesses,0.0
+1774100161.8989484,audit_epoch,208,2,audit/loss/empirical_epsilon_details/0.01/correct_guesses,0.0
+1774100161.8989484,audit_epoch,208,2,audit/embedding/auc,0.520928
+1774100161.8989484,audit_epoch,208,2,audit/embedding/empirical_epsilon/0.05,0.0
+1774100161.8989484,audit_epoch,208,2,audit/embedding/empirical_epsilon/0.01,0.0
+1774100161.8989484,audit_epoch,208,2,audit/embedding/empirical_epsilon_details/0.05/epsilon,0.0
+1774100161.8989484,audit_epoch,208,2,audit/embedding/empirical_epsilon_details/0.05/num_guesses,0.0
+1774100161.8989484,audit_epoch,208,2,audit/embedding/empirical_epsilon_details/0.05/correct_guesses,0.0
+1774100161.8989484,audit_epoch,208,2,audit/embedding/empirical_epsilon_details/0.01/epsilon,0.0
+1774100161.8989484,audit_epoch,208,2,audit/embedding/empirical_epsilon_details/0.01/num_guesses,0.0
+1774100161.8989484,audit_epoch,208,2,audit/embedding/empirical_epsilon_details/0.01/correct_guesses,0.0
+1774100161.8989484,audit_epoch,208,2,perf/audit_duration_sec,5.841585950925946
+1774100172.4001799,audit_final,208,2,audit/delta,1e-05
+1774100172.4001799,audit_final,208,2,audit/num_canaries,500.0
+1774100172.4001799,audit_final,208,2,audit/num_members,250.0
+1774100172.4001799,audit_final,208,2,audit/paper_guess_fraction,0.2
+1774100172.4001799,audit_final,208,2,audit/paper_guess_steps,20.0
+1774100172.4001799,audit_final,208,2,audit/loss/auc,0.525808
+1774100172.4001799,audit_final,208,2,audit/loss/empirical_epsilon/0.05,0.05073561053723097
+1774100172.4001799,audit_final,208,2,audit/loss/empirical_epsilon/0.01,0.0
+1774100172.4001799,audit_final,208,2,audit/loss/empirical_epsilon_details/0.05/epsilon,0.05073561053723097
+1774100172.4001799,audit_final,208,2,audit/loss/empirical_epsilon_details/0.05/num_guesses,100.0
+1774100172.4001799,audit_final,208,2,audit/loss/empirical_epsilon_details/0.05/correct_guesses,60.0
+1774100172.4001799,audit_final,208,2,audit/loss/empirical_epsilon_details/0.01/epsilon,0.0
+1774100172.4001799,audit_final,208,2,audit/loss/empirical_epsilon_details/0.01/num_guesses,0.0
+1774100172.4001799,audit_final,208,2,audit/loss/empirical_epsilon_details/0.01/correct_guesses,0.0
+1774100172.4001799,audit_final,208,2,audit/embedding/auc,0.520928
+1774100172.4001799,audit_final,208,2,audit/embedding/empirical_epsilon/0.05,0.0
+1774100172.4001799,audit_final,208,2,audit/embedding/empirical_epsilon/0.01,0.0
+1774100172.4001799,audit_final,208,2,audit/embedding/empirical_epsilon_details/0.05/epsilon,0.0
+1774100172.4001799,audit_final,208,2,audit/embedding/empirical_epsilon_details/0.05/num_guesses,0.0
+1774100172.4001799,audit_final,208,2,audit/embedding/empirical_epsilon_details/0.05/correct_guesses,0.0
+1774100172.4001799,audit_final,208,2,audit/embedding/empirical_epsilon_details/0.01/epsilon,0.0
+1774100172.4001799,audit_final,208,2,audit/embedding/empirical_epsilon_details/0.01/num_guesses,0.0
+1774100172.4001799,audit_final,208,2,audit/embedding/empirical_epsilon_details/0.01/correct_guesses,0.0
+1774100172.924808,energy_final,208,,energy/codecarbon/duration,3938.0607158355415
+1774100172.924808,energy_final,208,,energy/codecarbon/emissions,0.5702959606126596
+1774100172.924808,energy_final,208,,energy/codecarbon/emissions_rate,0.00014481644691749236
+1774100172.924808,energy_final,208,,energy/codecarbon/cpu_power,179.92904019353898
+1774100172.924808,energy_final,208,,energy/codecarbon/gpu_power,2298.550082608512
+1774100172.924808,energy_final,208,,energy/codecarbon/ram_power,70.0
+1774100172.924808,energy_final,208,,energy/codecarbon/cpu_energy,0.18975838183590477
+1774100172.924808,energy_final,208,,energy/codecarbon/gpu_energy,2.514705143984827
+1774100172.924808,energy_final,208,,energy/codecarbon/ram_energy,0.07382037755078118
+1774100172.924808,energy_final,208,,energy/codecarbon/energy_consumed,2.778283903371513
+1774100172.924808,energy_final,208,,energy/codecarbon/water_consumed,0.0
+1774100172.924808,energy_final,208,,energy/codecarbon/cpu_count,224.0
+1774100172.924808,energy_final,208,,energy/codecarbon/gpu_count,4.0
+1774100172.924808,energy_final,208,,energy/codecarbon/longitude,-121.9552
+1774100172.924808,energy_final,208,,energy/codecarbon/latitude,37.3541
+1774100172.924808,energy_final,208,,energy/codecarbon/ram_total_size,2015.5625190734863
+1774100172.924808,energy_final,208,,energy/codecarbon/cpu_utilization_percent,2.4580389144905275
+1774100172.924808,energy_final,208,,energy/codecarbon/gpu_utilization_percent,95.87682458386683
+1774100172.924808,energy_final,208,,energy/codecarbon/ram_utilization_percent,2.36410650281618
+1774100172.924808,energy_final,208,,energy/codecarbon/ram_used_gb,47.93522234837092
+1774100172.924808,energy_final,208,,energy/codecarbon/pue,1.0
+1774100172.924808,energy_final,208,,energy/codecarbon/wue,0.0

starcoder2-7b/dp3/summary.json

@@ -0,0 +1,72 @@
+{
+  "audit/delta": 1e-05,
+  "audit/embedding/auc": 0.520928,
+  "audit/embedding/empirical_epsilon/0.01": 0.0,
+  "audit/embedding/empirical_epsilon/0.05": 0.0,
+  "audit/embedding/empirical_epsilon_details/0.01/correct_guesses": 0.0,
+  "audit/embedding/empirical_epsilon_details/0.01/epsilon": 0.0,
+  "audit/embedding/empirical_epsilon_details/0.01/num_guesses": 0.0,
+  "audit/embedding/empirical_epsilon_details/0.05/correct_guesses": 0.0,
+  "audit/embedding/empirical_epsilon_details/0.05/epsilon": 0.0,
+  "audit/embedding/empirical_epsilon_details/0.05/num_guesses": 0.0,
+  "audit/loss/auc": 0.525808,
+  "audit/loss/empirical_epsilon/0.01": 0.0,
+  "audit/loss/empirical_epsilon/0.05": 0.05073561053723097,
+  "audit/loss/empirical_epsilon_details/0.01/correct_guesses": 0.0,
+  "audit/loss/empirical_epsilon_details/0.01/epsilon": 0.0,
+  "audit/loss/empirical_epsilon_details/0.01/num_guesses": 0.0,
+  "audit/loss/empirical_epsilon_details/0.05/correct_guesses": 60.0,
+  "audit/loss/empirical_epsilon_details/0.05/epsilon": 0.05073561053723097,
+  "audit/loss/empirical_epsilon_details/0.05/num_guesses": 100.0,
+  "audit/num_canaries": 500.0,
+  "audit/num_members": 250.0,
+  "audit/paper_guess_fraction": 0.2,
+  "audit/paper_guess_steps": 20.0,
+  "energy/codecarbon/cpu_count": 224.0,
+  "energy/codecarbon/cpu_energy": 0.18975838183590477,
+  "energy/codecarbon/cpu_power": 179.92904019353898,
+  "energy/codecarbon/cpu_utilization_percent": 2.4580389144905275,
+  "energy/codecarbon/duration": 3938.0607158355415,
+  "energy/codecarbon/emissions": 0.5702959606126596,
+  "energy/codecarbon/emissions_rate": 0.00014481644691749236,
+  "energy/codecarbon/energy_consumed": 2.778283903371513,
+  "energy/codecarbon/gpu_count": 4.0,
+  "energy/codecarbon/gpu_energy": 2.514705143984827,
+  "energy/codecarbon/gpu_power": 2298.550082608512,
+  "energy/codecarbon/gpu_utilization_percent": 95.87682458386683,
+  "energy/codecarbon/latitude": 37.3541,
+  "energy/codecarbon/longitude": -121.9552,
+  "energy/codecarbon/pue": 1.0,
+  "energy/codecarbon/ram_energy": 0.07382037755078118,
+  "energy/codecarbon/ram_power": 70.0,
+  "energy/codecarbon/ram_total_size": 2015.5625190734863,
+  "energy/codecarbon/ram_used_gb": 47.93522234837092,
+  "energy/codecarbon/ram_utilization_percent": 2.36410650281618,
+  "energy/codecarbon/water_consumed": 0.0,
+  "energy/codecarbon/wue": 0.0,
+  "eval/duration_sec": 27.067450830712914,
+  "eval/loss": 0.8425623961221212,
+  "perf/audit_duration_sec": 5.841585950925946,
+  "perf/epoch_duration_sec": 1886.0955602359027,
+  "perf/epoch_samples": 55619.0,
+  "perf/epoch_samples_per_sec": 29.488961838732855,
+  "perf/epoch_tokens": 43998761.0,
+  "perf/epoch_tokens_per_sec": 23327.959583605018,
+  "perf/logical_batch_size": 132.0,
+  "perf/logical_token_count": 106537.0,
+  "perf/physical_batches": 17.0,
+  "perf/samples_per_sec": 7.860206056319249,
+  "perf/step_duration_sec": 16.793452875688672,
+  "perf/tokens_per_sec": 6343.96039865215,
+  "privacy/epsilon": 2.9999680995370417,
+  "system/cuda_epoch_peak_memory_gb": 74.69534540176392,
+  "system/cuda_max_memory_allocated_gb": 74.69534540176392,
+  "system/cuda_memory_allocated_gb": 16.537474155426025,
+  "train/epoch_canary_loss": 9.74998009501868,
+  "train/epoch_loss": 1.6631105757675884,
+  "train/epoch_real_loss": 1.2492633356516678,
+  "train/lr": 8.126960406835249e-07,
+  "train/step_canary_loss": 10.9375,
+  "train/step_loss": 1.5310632604541201,
+  "train/step_real_loss": 1.2371121123433113
+}

starcoder2-7b/dp3/tensorboard/events.out.tfevents.1774096221.364c4f8de9dd.12837.0

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3b07c9c3ed310b74cc126d10e56bdacca0e2e7e9e9015a0a823e61040e77f436
+size 26671