Upload folder using huggingface_hub (#1)

- Upload folder using huggingface_hub (4863888d082c99eacaca6bf296804bde846feb5d)

README.md

@@ -1,3 +1,202 @@
 ---
-license: apache-2.0
+library_name: peft
+base_model: google/gemma-7b-it
 ---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.9.0

adapter_config.json

@@ -0,0 +1,33 @@
+{
+  "alpha_pattern": {},
+  "auto_mapping": null,
+  "base_model_name_or_path": "google/gemma-7b-it",
+  "bias": "none",
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 16,
+  "lora_dropout": 0.1,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "r": 64,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "gate_proj",
+    "k_proj",
+    "q_proj",
+    "o_proj",
+    "v_proj",
+    "down_proj",
+    "up_proj"
+  ],
+  "task_type": "CAUSAL_LM",
+  "use_dora": false,
+  "use_rslora": false
+}

adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3b740481846ec93f2e74214108562ccc6f94bf9fd773910066fe9fbf258cf680
+size 400085000

optimizer.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c0a40a444d17ad336d019c95a6dc03e052d4de11ea7020022fd36e1f710f728d
+size 401282868

rng_state.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:436b96bd2a58ceec2a3046c15b099a844f299af2f50e93943ee7ad0d81f8e8b0
+size 14244

scheduler.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:25ae61e837be3ada9de530496f3f4fe1da31f4699408459fe2a0b05c3e210f0d
+size 1064

trainer_state.json

@@ -0,0 +1,1421 @@
+{
+  "best_metric": null,
+  "best_model_checkpoint": null,
+  "epoch": 1.0,
+  "eval_steps": 500,
+  "global_step": 5000,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "epoch": 0.01,
+      "grad_norm": 2.296875,
+      "learning_rate": 8.333333333333334e-06,
+      "loss": 6.02,
+      "step": 25
+    },
+    {
+      "epoch": 0.01,
+      "grad_norm": 0.63671875,
+      "learning_rate": 1.6666666666666667e-05,
+      "loss": 5.3902,
+      "step": 50
+    },
+    {
+      "epoch": 0.01,
+      "grad_norm": 1.2734375,
+      "learning_rate": 2.5e-05,
+      "loss": 4.6611,
+      "step": 75
+    },
+    {
+      "epoch": 0.02,
+      "grad_norm": 7.71875,
+      "learning_rate": 3.3333333333333335e-05,
+      "loss": 3.8627,
+      "step": 100
+    },
+    {
+      "epoch": 0.03,
+      "grad_norm": 0.5625,
+      "learning_rate": 4.166666666666667e-05,
+      "loss": 2.3529,
+      "step": 125
+    },
+    {
+      "epoch": 0.03,
+      "grad_norm": 0.53515625,
+      "learning_rate": 5e-05,
+      "loss": 1.4613,
+      "step": 150
+    },
+    {
+      "epoch": 0.04,
+      "grad_norm": 0.4453125,
+      "learning_rate": 4.999672209164081e-05,
+      "loss": 1.3771,
+      "step": 175
+    },
+    {
+      "epoch": 0.04,
+      "grad_norm": 0.5546875,
+      "learning_rate": 4.998688922613788e-05,
+      "loss": 1.3205,
+      "step": 200
+    },
+    {
+      "epoch": 0.04,
+      "grad_norm": 0.73046875,
+      "learning_rate": 4.997050398198977e-05,
+      "loss": 1.2926,
+      "step": 225
+    },
+    {
+      "epoch": 0.05,
+      "grad_norm": 0.50390625,
+      "learning_rate": 4.9947570655942796e-05,
+      "loss": 1.2837,
+      "step": 250
+    },
+    {
+      "epoch": 0.06,
+      "grad_norm": 0.5703125,
+      "learning_rate": 4.991809526186424e-05,
+      "loss": 1.3392,
+      "step": 275
+    },
+    {
+      "epoch": 0.06,
+      "grad_norm": 0.46484375,
+      "learning_rate": 4.988208552916535e-05,
+      "loss": 1.187,
+      "step": 300
+    },
+    {
+      "epoch": 0.07,
+      "grad_norm": 0.447265625,
+      "learning_rate": 4.983955090077444e-05,
+      "loss": 1.2047,
+      "step": 325
+    },
+    {
+      "epoch": 0.07,
+      "grad_norm": 0.494140625,
+      "learning_rate": 4.9790502530660635e-05,
+      "loss": 1.2062,
+      "step": 350
+    },
+    {
+      "epoch": 0.07,
+      "grad_norm": 0.55859375,
+      "learning_rate": 4.9734953280908904e-05,
+      "loss": 1.248,
+      "step": 375
+    },
+    {
+      "epoch": 0.08,
+      "grad_norm": 0.82421875,
+      "learning_rate": 4.967291771834727e-05,
+      "loss": 1.225,
+      "step": 400
+    },
+    {
+      "epoch": 0.09,
+      "grad_norm": 0.484375,
+      "learning_rate": 4.960441211072686e-05,
+      "loss": 1.2077,
+      "step": 425
+    },
+    {
+      "epoch": 0.09,
+      "grad_norm": 0.5078125,
+      "learning_rate": 4.9529454422455976e-05,
+      "loss": 1.2046,
+      "step": 450
+    },
+    {
+      "epoch": 0.1,
+      "grad_norm": 0.404296875,
+      "learning_rate": 4.944806430988927e-05,
+      "loss": 1.2055,
+      "step": 475
+    },
+    {
+      "epoch": 0.1,
+      "grad_norm": 0.546875,
+      "learning_rate": 4.936026311617316e-05,
+      "loss": 1.2438,
+      "step": 500
+    },
+    {
+      "epoch": 0.1,
+      "grad_norm": 0.462890625,
+      "learning_rate": 4.926607386564898e-05,
+      "loss": 1.2093,
+      "step": 525
+    },
+    {
+      "epoch": 0.11,
+      "grad_norm": 0.50390625,
+      "learning_rate": 4.916552125781528e-05,
+      "loss": 1.1535,
+      "step": 550
+    },
+    {
+      "epoch": 0.12,
+      "grad_norm": 0.53125,
+      "learning_rate": 4.9058631660850765e-05,
+      "loss": 1.2239,
+      "step": 575
+    },
+    {
+      "epoch": 0.12,
+      "grad_norm": 0.50390625,
+      "learning_rate": 4.894543310469968e-05,
+      "loss": 1.1454,
+      "step": 600
+    },
+    {
+      "epoch": 0.12,
+      "grad_norm": 0.61328125,
+      "learning_rate": 4.882595527372152e-05,
+      "loss": 1.1234,
+      "step": 625
+    },
+    {
+      "epoch": 0.13,
+      "grad_norm": 0.486328125,
+      "learning_rate": 4.870022949890676e-05,
+      "loss": 1.1888,
+      "step": 650
+    },
+    {
+      "epoch": 0.14,
+      "grad_norm": 0.57421875,
+      "learning_rate": 4.856828874966086e-05,
+      "loss": 1.1476,
+      "step": 675
+    },
+    {
+      "epoch": 0.14,
+      "grad_norm": 0.435546875,
+      "learning_rate": 4.8430167625158595e-05,
+      "loss": 1.2208,
+      "step": 700
+    },
+    {
+      "epoch": 0.14,
+      "grad_norm": 0.49609375,
+      "learning_rate": 4.828590234527106e-05,
+      "loss": 1.16,
+      "step": 725
+    },
+    {
+      "epoch": 0.15,
+      "grad_norm": 0.40625,
+      "learning_rate": 4.813553074106761e-05,
+      "loss": 1.1312,
+      "step": 750
+    },
+    {
+      "epoch": 0.15,
+      "grad_norm": 0.53125,
+      "learning_rate": 4.7979092244895305e-05,
+      "loss": 1.1732,
+      "step": 775
+    },
+    {
+      "epoch": 0.16,
+      "grad_norm": 0.498046875,
+      "learning_rate": 4.781662788003851e-05,
+      "loss": 1.1404,
+      "step": 800
+    },
+    {
+      "epoch": 0.17,
+      "grad_norm": 0.484375,
+      "learning_rate": 4.764818024996117e-05,
+      "loss": 1.1498,
+      "step": 825
+    },
+    {
+      "epoch": 0.17,
+      "grad_norm": 0.5859375,
+      "learning_rate": 4.747379352713489e-05,
+      "loss": 1.148,
+      "step": 850
+    },
+    {
+      "epoch": 0.17,
+      "grad_norm": 0.5234375,
+      "learning_rate": 4.7293513441455364e-05,
+      "loss": 1.1245,
+      "step": 875
+    },
+    {
+      "epoch": 0.18,
+      "grad_norm": 0.49609375,
+      "learning_rate": 4.710738726825059e-05,
+      "loss": 1.0975,
+      "step": 900
+    },
+    {
+      "epoch": 0.18,
+      "grad_norm": 0.5390625,
+      "learning_rate": 4.69154638158837e-05,
+      "loss": 1.1204,
+      "step": 925
+    },
+    {
+      "epoch": 0.19,
+      "grad_norm": 0.55078125,
+      "learning_rate": 4.671779341295378e-05,
+      "loss": 1.1291,
+      "step": 950
+    },
+    {
+      "epoch": 0.2,
+      "grad_norm": 0.46484375,
+      "learning_rate": 4.6514427895098134e-05,
+      "loss": 1.1382,
+      "step": 975
+    },
+    {
+      "epoch": 0.2,
+      "grad_norm": 0.65625,
+      "learning_rate": 4.630542059139924e-05,
+      "loss": 1.1861,
+      "step": 1000
+    },
+    {
+      "epoch": 0.2,
+      "grad_norm": 0.73046875,
+      "learning_rate": 4.6090826310400116e-05,
+      "loss": 1.1358,
+      "step": 1025
+    },
+    {
+      "epoch": 0.21,
+      "grad_norm": 0.57421875,
+      "learning_rate": 4.587070132573178e-05,
+      "loss": 1.115,
+      "step": 1050
+    },
+    {
+      "epoch": 0.21,
+      "grad_norm": 0.69921875,
+      "learning_rate": 4.5645103361356415e-05,
+      "loss": 1.0639,
+      "step": 1075
+    },
+    {
+      "epoch": 0.22,
+      "grad_norm": 0.51953125,
+      "learning_rate": 4.541409157643027e-05,
+      "loss": 1.1185,
+      "step": 1100
+    },
+    {
+      "epoch": 0.23,
+      "grad_norm": 0.65234375,
+      "learning_rate": 4.517772654979023e-05,
+      "loss": 1.1038,
+      "step": 1125
+    },
+    {
+      "epoch": 0.23,
+      "grad_norm": 0.498046875,
+      "learning_rate": 4.493607026406802e-05,
+      "loss": 1.0626,
+      "step": 1150
+    },
+    {
+      "epoch": 0.23,
+      "grad_norm": 0.46484375,
+      "learning_rate": 4.4689186089436366e-05,
+      "loss": 1.0829,
+      "step": 1175
+    },
+    {
+      "epoch": 0.24,
+      "grad_norm": 0.53515625,
+      "learning_rate": 4.443713876699124e-05,
+      "loss": 1.1554,
+      "step": 1200
+    },
+    {
+      "epoch": 0.24,
+      "grad_norm": 0.486328125,
+      "learning_rate": 4.417999439177466e-05,
+      "loss": 1.0713,
+      "step": 1225
+    },
+    {
+      "epoch": 0.25,
+      "grad_norm": 0.5078125,
+      "learning_rate": 4.391782039544238e-05,
+      "loss": 1.0986,
+      "step": 1250
+    },
+    {
+      "epoch": 0.26,
+      "grad_norm": 0.474609375,
+      "learning_rate": 4.365068552858115e-05,
+      "loss": 1.0185,
+      "step": 1275
+    },
+    {
+      "epoch": 0.26,
+      "grad_norm": 0.53515625,
+      "learning_rate": 4.337865984268001e-05,
+      "loss": 1.0955,
+      "step": 1300
+    },
+    {
+      "epoch": 0.27,
+      "grad_norm": 0.58984375,
+      "learning_rate": 4.3101814671760546e-05,
+      "loss": 1.0747,
+      "step": 1325
+    },
+    {
+      "epoch": 0.27,
+      "grad_norm": 0.53515625,
+      "learning_rate": 4.2820222613670736e-05,
+      "loss": 1.0395,
+      "step": 1350
+    },
+    {
+      "epoch": 0.28,
+      "grad_norm": 0.63671875,
+      "learning_rate": 4.253395751104748e-05,
+      "loss": 1.0798,
+      "step": 1375
+    },
+    {
+      "epoch": 0.28,
+      "grad_norm": 0.828125,
+      "learning_rate": 4.224309443195261e-05,
+      "loss": 1.154,
+      "step": 1400
+    },
+    {
+      "epoch": 0.28,
+      "grad_norm": 0.6015625,
+      "learning_rate": 4.194770965018758e-05,
+      "loss": 1.0445,
+      "step": 1425
+    },
+    {
+      "epoch": 0.29,
+      "grad_norm": 0.5859375,
+      "learning_rate": 4.164788062529203e-05,
+      "loss": 1.0651,
+      "step": 1450
+    },
+    {
+      "epoch": 0.29,
+      "grad_norm": 0.5,
+      "learning_rate": 4.134368598223132e-05,
+      "loss": 1.1505,
+      "step": 1475
+    },
+    {
+      "epoch": 0.3,
+      "grad_norm": 0.52734375,
+      "learning_rate": 4.10352054907785e-05,
+      "loss": 1.085,
+      "step": 1500
+    },
+    {
+      "epoch": 0.3,
+      "grad_norm": 0.53515625,
+      "learning_rate": 4.072252004459611e-05,
+      "loss": 1.095,
+      "step": 1525
+    },
+    {
+      "epoch": 0.31,
+      "grad_norm": 0.8046875,
+      "learning_rate": 4.0405711640023186e-05,
+      "loss": 1.0373,
+      "step": 1550
+    },
+    {
+      "epoch": 0.32,
+      "grad_norm": 0.6015625,
+      "learning_rate": 4.008486335457312e-05,
+      "loss": 1.0507,
+      "step": 1575
+    },
+    {
+      "epoch": 0.32,
+      "grad_norm": 0.578125,
+      "learning_rate": 3.976005932514807e-05,
+      "loss": 1.0545,
+      "step": 1600
+    },
+    {
+      "epoch": 0.33,
+      "grad_norm": 0.65625,
+      "learning_rate": 3.943138472597549e-05,
+      "loss": 1.0342,
+      "step": 1625
+    },
+    {
+      "epoch": 0.33,
+      "grad_norm": 0.609375,
+      "learning_rate": 3.909892574627266e-05,
+      "loss": 1.0948,
+      "step": 1650
+    },
+    {
+      "epoch": 0.34,
+      "grad_norm": 0.59765625,
+      "learning_rate": 3.876276956764509e-05,
+      "loss": 0.9903,
+      "step": 1675
+    },
+    {
+      "epoch": 0.34,
+      "grad_norm": 0.5625,
+      "learning_rate": 3.84230043412246e-05,
+      "loss": 1.058,
+      "step": 1700
+    },
+    {
+      "epoch": 0.34,
+      "grad_norm": 0.515625,
+      "learning_rate": 3.807971916455325e-05,
+      "loss": 0.9873,
+      "step": 1725
+    },
+    {
+      "epoch": 0.35,
+      "grad_norm": 0.55859375,
+      "learning_rate": 3.773300405821908e-05,
+      "loss": 1.0034,
+      "step": 1750
+    },
+    {
+      "epoch": 0.35,
+      "grad_norm": 0.60546875,
+      "learning_rate": 3.7382949942249694e-05,
+      "loss": 1.0152,
+      "step": 1775
+    },
+    {
+      "epoch": 0.36,
+      "grad_norm": 0.578125,
+      "learning_rate": 3.702964861227013e-05,
+      "loss": 1.0289,
+      "step": 1800
+    },
+    {
+      "epoch": 0.36,
+      "grad_norm": 0.65625,
+      "learning_rate": 3.6673192715431015e-05,
+      "loss": 0.9901,
+      "step": 1825
+    },
+    {
+      "epoch": 0.37,
+      "grad_norm": 0.63671875,
+      "learning_rate": 3.631367572611348e-05,
+      "loss": 0.9345,
+      "step": 1850
+    },
+    {
+      "epoch": 0.38,
+      "grad_norm": 0.671875,
+      "learning_rate": 3.595119192141706e-05,
+      "loss": 1.035,
+      "step": 1875
+    },
+    {
+      "epoch": 0.38,
+      "grad_norm": 0.6328125,
+      "learning_rate": 3.5585836356437264e-05,
+      "loss": 1.0689,
+      "step": 1900
+    },
+    {
+      "epoch": 0.39,
+      "grad_norm": 0.6015625,
+      "learning_rate": 3.521770483933891e-05,
+      "loss": 0.9883,
+      "step": 1925
+    },
+    {
+      "epoch": 0.39,
+      "grad_norm": 0.51953125,
+      "learning_rate": 3.484689390623218e-05,
+      "loss": 1.0279,
+      "step": 1950
+    },
+    {
+      "epoch": 0.4,
+      "grad_norm": 0.6484375,
+      "learning_rate": 3.447350079585767e-05,
+      "loss": 1.0602,
+      "step": 1975
+    },
+    {
+      "epoch": 0.4,
+      "grad_norm": 0.59375,
+      "learning_rate": 3.409762342408719e-05,
+      "loss": 1.0178,
+      "step": 2000
+    },
+    {
+      "epoch": 0.41,
+      "grad_norm": 0.6484375,
+      "learning_rate": 3.3719360358247054e-05,
+      "loss": 1.0243,
+      "step": 2025
+    },
+    {
+      "epoch": 0.41,
+      "grad_norm": 0.640625,
+      "learning_rate": 3.333881079127052e-05,
+      "loss": 1.0175,
+      "step": 2050
+    },
+    {
+      "epoch": 0.41,
+      "grad_norm": 0.69140625,
+      "learning_rate": 3.29560745156861e-05,
+      "loss": 0.921,
+      "step": 2075
+    },
+    {
+      "epoch": 0.42,
+      "grad_norm": 0.65234375,
+      "learning_rate": 3.2571251897448765e-05,
+      "loss": 0.9597,
+      "step": 2100
+    },
+    {
+      "epoch": 0.42,
+      "grad_norm": 0.51171875,
+      "learning_rate": 3.218444384962071e-05,
+      "loss": 0.9761,
+      "step": 2125
+    },
+    {
+      "epoch": 0.43,
+      "grad_norm": 0.640625,
+      "learning_rate": 3.1795751805908573e-05,
+      "loss": 0.9706,
+      "step": 2150
+    },
+    {
+      "epoch": 0.43,
+      "grad_norm": 0.6640625,
+      "learning_rate": 3.1405277694064305e-05,
+      "loss": 0.932,
+      "step": 2175
+    },
+    {
+      "epoch": 0.44,
+      "grad_norm": 0.6796875,
+      "learning_rate": 3.101312390915634e-05,
+      "loss": 0.9486,
+      "step": 2200
+    },
+    {
+      "epoch": 0.45,
+      "grad_norm": 0.69140625,
+      "learning_rate": 3.061939328671824e-05,
+      "loss": 1.011,
+      "step": 2225
+    },
+    {
+      "epoch": 0.45,
+      "grad_norm": 0.60546875,
+      "learning_rate": 3.0224189075781884e-05,
+      "loss": 0.9463,
+      "step": 2250
+    },
+    {
+      "epoch": 0.46,
+      "grad_norm": 0.6796875,
+      "learning_rate": 2.9827614911802203e-05,
+      "loss": 0.9989,
+      "step": 2275
+    },
+    {
+      "epoch": 0.46,
+      "grad_norm": 0.7109375,
+      "learning_rate": 2.9429774789480575e-05,
+      "loss": 0.9677,
+      "step": 2300
+    },
+    {
+      "epoch": 0.47,
+      "grad_norm": 0.62890625,
+      "learning_rate": 2.9030773035493997e-05,
+      "loss": 0.9739,
+      "step": 2325
+    },
+    {
+      "epoch": 0.47,
+      "grad_norm": 0.69140625,
+      "learning_rate": 2.863071428113726e-05,
+      "loss": 1.0254,
+      "step": 2350
+    },
+    {
+      "epoch": 0.47,
+      "grad_norm": 0.8203125,
+      "learning_rate": 2.8229703434885163e-05,
+      "loss": 0.9969,
+      "step": 2375
+    },
+    {
+      "epoch": 0.48,
+      "grad_norm": 0.56640625,
+      "learning_rate": 2.782784565488211e-05,
+      "loss": 0.9918,
+      "step": 2400
+    },
+    {
+      "epoch": 0.48,
+      "grad_norm": 0.58203125,
+      "learning_rate": 2.7425246321366203e-05,
+      "loss": 1.0169,
+      "step": 2425
+    },
+    {
+      "epoch": 0.49,
+      "grad_norm": 0.75,
+      "learning_rate": 2.7022011009035107e-05,
+      "loss": 0.9844,
+      "step": 2450
+    },
+    {
+      "epoch": 0.49,
+      "grad_norm": 0.7265625,
+      "learning_rate": 2.6618245459360897e-05,
+      "loss": 1.026,
+      "step": 2475
+    },
+    {
+      "epoch": 0.5,
+      "grad_norm": 0.671875,
+      "learning_rate": 2.621405555286121e-05,
+      "loss": 0.9333,
+      "step": 2500
+    },
+    {
+      "epoch": 0.51,
+      "grad_norm": 0.6875,
+      "learning_rate": 2.5809547281333902e-05,
+      "loss": 1.0336,
+      "step": 2525
+    },
+    {
+      "epoch": 0.51,
+      "grad_norm": 0.63671875,
+      "learning_rate": 2.540482672006254e-05,
+      "loss": 0.9375,
+      "step": 2550
+    },
+    {
+      "epoch": 0.52,
+      "grad_norm": 0.87109375,
+      "learning_rate": 2.5e-05,
+      "loss": 0.9441,
+      "step": 2575
+    },
+    {
+      "epoch": 0.52,
+      "grad_norm": 0.7265625,
+      "learning_rate": 2.4595173279937464e-05,
+      "loss": 0.9643,
+      "step": 2600
+    },
+    {
+      "epoch": 0.53,
+      "grad_norm": 0.67578125,
+      "learning_rate": 2.419045271866611e-05,
+      "loss": 0.9172,
+      "step": 2625
+    },
+    {
+      "epoch": 0.53,
+      "grad_norm": 0.87109375,
+      "learning_rate": 2.3785944447138802e-05,
+      "loss": 0.8813,
+      "step": 2650
+    },
+    {
+      "epoch": 0.54,
+      "grad_norm": 0.8046875,
+      "learning_rate": 2.338175454063911e-05,
+      "loss": 0.9638,
+      "step": 2675
+    },
+    {
+      "epoch": 0.54,
+      "grad_norm": 0.859375,
+      "learning_rate": 2.29779889909649e-05,
+      "loss": 0.9476,
+      "step": 2700
+    },
+    {
+      "epoch": 0.55,
+      "grad_norm": 0.6796875,
+      "learning_rate": 2.25747536786338e-05,
+      "loss": 0.9536,
+      "step": 2725
+    },
+    {
+      "epoch": 0.55,
+      "grad_norm": 0.77734375,
+      "learning_rate": 2.2172154345117894e-05,
+      "loss": 0.9308,
+      "step": 2750
+    },
+    {
+      "epoch": 0.56,
+      "grad_norm": 0.734375,
+      "learning_rate": 2.177029656511485e-05,
+      "loss": 0.9432,
+      "step": 2775
+    },
+    {
+      "epoch": 0.56,
+      "grad_norm": 0.66015625,
+      "learning_rate": 2.136928571886275e-05,
+      "loss": 0.9253,
+      "step": 2800
+    },
+    {
+      "epoch": 0.56,
+      "grad_norm": 0.69921875,
+      "learning_rate": 2.0969226964506006e-05,
+      "loss": 0.9301,
+      "step": 2825
+    },
+    {
+      "epoch": 0.57,
+      "grad_norm": 0.7578125,
+      "learning_rate": 2.0570225210519434e-05,
+      "loss": 0.9509,
+      "step": 2850
+    },
+    {
+      "epoch": 0.57,
+      "grad_norm": 0.7421875,
+      "learning_rate": 2.0172385088197803e-05,
+      "loss": 0.8995,
+      "step": 2875
+    },
+    {
+      "epoch": 0.58,
+      "grad_norm": 0.8125,
+      "learning_rate": 1.9775810924218125e-05,
+      "loss": 0.8682,
+      "step": 2900
+    },
+    {
+      "epoch": 0.58,
+      "grad_norm": 0.8203125,
+      "learning_rate": 1.9380606713281775e-05,
+      "loss": 0.9384,
+      "step": 2925
+    },
+    {
+      "epoch": 0.59,
+      "grad_norm": 0.76171875,
+      "learning_rate": 1.8986876090843667e-05,
+      "loss": 0.9545,
+      "step": 2950
+    },
+    {
+      "epoch": 0.59,
+      "grad_norm": 0.74609375,
+      "learning_rate": 1.859472230593569e-05,
+      "loss": 0.8595,
+      "step": 2975
+    },
+    {
+      "epoch": 0.6,
+      "grad_norm": 0.8125,
+      "learning_rate": 1.820424819409143e-05,
+      "loss": 0.9636,
+      "step": 3000
+    },
+    {
+      "epoch": 0.6,
+      "grad_norm": 0.75390625,
+      "learning_rate": 1.7815556150379298e-05,
+      "loss": 0.973,
+      "step": 3025
+    },
+    {
+      "epoch": 0.61,
+      "grad_norm": 0.73828125,
+      "learning_rate": 1.7428748102551237e-05,
+      "loss": 0.9182,
+      "step": 3050
+    },
+    {
+      "epoch": 0.61,
+      "grad_norm": 0.85546875,
+      "learning_rate": 1.704392548431391e-05,
+      "loss": 0.9419,
+      "step": 3075
+    },
+    {
+      "epoch": 0.62,
+      "grad_norm": 0.71484375,
+      "learning_rate": 1.666118920872949e-05,
+      "loss": 0.8826,
+      "step": 3100
+    },
+    {
+      "epoch": 0.62,
+      "grad_norm": 0.83203125,
+      "learning_rate": 1.6280639641752942e-05,
+      "loss": 0.8989,
+      "step": 3125
+    },
+    {
+      "epoch": 0.63,
+      "grad_norm": 0.8125,
+      "learning_rate": 1.5902376575912815e-05,
+      "loss": 0.8946,
+      "step": 3150
+    },
+    {
+      "epoch": 0.64,
+      "grad_norm": 0.7734375,
+      "learning_rate": 1.552649920414233e-05,
+      "loss": 0.8636,
+      "step": 3175
+    },
+    {
+      "epoch": 0.64,
+      "grad_norm": 0.91015625,
+      "learning_rate": 1.5153106093767827e-05,
+      "loss": 0.9318,
+      "step": 3200
+    },
+    {
+      "epoch": 0.65,
+      "grad_norm": 0.80078125,
+      "learning_rate": 1.4782295160661103e-05,
+      "loss": 0.9496,
+      "step": 3225
+    },
+    {
+      "epoch": 0.65,
+      "grad_norm": 0.734375,
+      "learning_rate": 1.4414163643562755e-05,
+      "loss": 0.8607,
+      "step": 3250
+    },
+    {
+      "epoch": 0.66,
+      "grad_norm": 0.7734375,
+      "learning_rate": 1.4048808078582942e-05,
+      "loss": 0.8367,
+      "step": 3275
+    },
+    {
+      "epoch": 0.66,
+      "grad_norm": 0.83203125,
+      "learning_rate": 1.368632427388653e-05,
+      "loss": 0.9584,
+      "step": 3300
+    },
+    {
+      "epoch": 0.67,
+      "grad_norm": 0.80859375,
+      "learning_rate": 1.3326807284568984e-05,
+      "loss": 0.8844,
+      "step": 3325
+    },
+    {
+      "epoch": 0.67,
+      "grad_norm": 0.87890625,
+      "learning_rate": 1.2970351387729873e-05,
+      "loss": 0.9125,
+      "step": 3350
+    },
+    {
+      "epoch": 0.68,
+      "grad_norm": 0.74609375,
+      "learning_rate": 1.2617050057750322e-05,
+      "loss": 0.9229,
+      "step": 3375
+    },
+    {
+      "epoch": 0.68,
+      "grad_norm": 0.80078125,
+      "learning_rate": 1.2266995941780934e-05,
+      "loss": 0.9541,
+      "step": 3400
+    },
+    {
+      "epoch": 0.69,
+      "grad_norm": 0.8359375,
+      "learning_rate": 1.1920280835446748e-05,
+      "loss": 0.8816,
+      "step": 3425
+    },
+    {
+      "epoch": 0.69,
+      "grad_norm": 0.82421875,
+      "learning_rate": 1.1576995658775405e-05,
+      "loss": 0.862,
+      "step": 3450
+    },
+    {
+      "epoch": 0.69,
+      "grad_norm": 0.80078125,
+      "learning_rate": 1.1237230432354912e-05,
+      "loss": 0.8541,
+      "step": 3475
+    },
+    {
+      "epoch": 0.7,
+      "grad_norm": 0.8203125,
+      "learning_rate": 1.0901074253727336e-05,
+      "loss": 0.8586,
+      "step": 3500
+    },
+    {
+      "epoch": 0.7,
+      "grad_norm": 0.7890625,
+      "learning_rate": 1.0568615274024522e-05,
+      "loss": 0.9033,
+      "step": 3525
+    },
+    {
+      "epoch": 0.71,
+      "grad_norm": 0.7890625,
+      "learning_rate": 1.0239940674851941e-05,
+      "loss": 0.8456,
+      "step": 3550
+    },
+    {
+      "epoch": 0.71,
+      "grad_norm": 0.91796875,
+      "learning_rate": 9.915136645426884e-06,
+      "loss": 0.9165,
+      "step": 3575
+    },
+    {
+      "epoch": 0.72,
+      "grad_norm": 0.80859375,
+      "learning_rate": 9.594288359976817e-06,
+      "loss": 0.8225,
+      "step": 3600
+    },
+    {
+      "epoch": 0.72,
+      "grad_norm": 0.85546875,
+      "learning_rate": 9.277479955403887e-06,
+      "loss": 0.815,
+      "step": 3625
+    },
+    {
+      "epoch": 0.73,
+      "grad_norm": 0.6640625,
+      "learning_rate": 8.964794509221508e-06,
+      "loss": 0.8517,
+      "step": 3650
+    },
+    {
+      "epoch": 0.73,
+      "grad_norm": 0.8125,
+      "learning_rate": 8.656314017768693e-06,
+      "loss": 0.9108,
+      "step": 3675
+    },
+    {
+      "epoch": 0.74,
+      "grad_norm": 0.8671875,
+      "learning_rate": 8.352119374707978e-06,
+      "loss": 0.8665,
+      "step": 3700
+    },
+    {
+      "epoch": 0.74,
+      "grad_norm": 0.8359375,
+      "learning_rate": 8.052290349812419e-06,
+      "loss": 0.875,
+      "step": 3725
+    },
+    {
+      "epoch": 0.75,
+      "grad_norm": 0.8359375,
+      "learning_rate": 7.756905568047393e-06,
+      "loss": 0.9181,
+      "step": 3750
+    },
+    {
+      "epoch": 0.76,
+      "grad_norm": 0.91015625,
+      "learning_rate": 7.466042488952521e-06,
+      "loss": 0.8715,
+      "step": 3775
+    },
+    {
+      "epoch": 0.76,
+      "grad_norm": 0.74609375,
+      "learning_rate": 7.179777386329276e-06,
+      "loss": 0.9051,
+      "step": 3800
+    },
+    {
+      "epoch": 0.77,
+      "grad_norm": 0.796875,
+      "learning_rate": 6.898185328239468e-06,
+      "loss": 0.871,
+      "step": 3825
+    },
+    {
+      "epoch": 0.77,
+      "grad_norm": 0.71875,
+      "learning_rate": 6.621340157319997e-06,
+      "loss": 0.8633,
+      "step": 3850
+    },
+    {
+      "epoch": 0.78,
+      "grad_norm": 0.98046875,
+      "learning_rate": 6.349314471418849e-06,
+      "loss": 0.8513,
+      "step": 3875
+    },
+    {
+      "epoch": 0.78,
+      "grad_norm": 0.81640625,
+      "learning_rate": 6.082179604557617e-06,
+      "loss": 0.8063,
+      "step": 3900
+    },
+    {
+      "epoch": 0.79,
+      "grad_norm": 0.80078125,
+      "learning_rate": 5.820005608225346e-06,
+      "loss": 0.8652,
+      "step": 3925
+    },
+    {
+      "epoch": 0.79,
+      "grad_norm": 0.81640625,
+      "learning_rate": 5.562861233008774e-06,
+      "loss": 0.872,
+      "step": 3950
+    },
+    {
+      "epoch": 0.8,
+      "grad_norm": 0.74609375,
+      "learning_rate": 5.310813910563644e-06,
+      "loss": 0.849,
+      "step": 3975
+    },
+    {
+      "epoch": 0.8,
+      "grad_norm": 0.79296875,
+      "learning_rate": 5.063929735931985e-06,
+      "loss": 0.8926,
+      "step": 4000
+    },
+    {
+      "epoch": 0.81,
+      "grad_norm": 0.77734375,
+      "learning_rate": 4.8222734502097665e-06,
+      "loss": 0.8093,
+      "step": 4025
+    },
+    {
+      "epoch": 0.81,
+      "grad_norm": 0.8515625,
+      "learning_rate": 4.585908423569724e-06,
+      "loss": 0.8563,
+      "step": 4050
+    },
+    {
+      "epoch": 0.81,
+      "grad_norm": 0.97265625,
+      "learning_rate": 4.35489663864359e-06,
+      "loss": 0.8393,
+      "step": 4075
+    },
+    {
+      "epoch": 0.82,
+      "grad_norm": 0.8984375,
+      "learning_rate": 4.129298674268225e-06,
+      "loss": 0.9031,
+      "step": 4100
+    },
+    {
+      "epoch": 0.82,
+      "grad_norm": 0.9296875,
+      "learning_rate": 3.90917368959989e-06,
+      "loss": 0.8981,
+      "step": 4125
+    },
+    {
+      "epoch": 0.83,
+      "grad_norm": 0.8828125,
+      "learning_rate": 3.694579408600771e-06,
+      "loss": 0.8628,
+      "step": 4150
+    },
+    {
+      "epoch": 0.83,
+      "grad_norm": 0.796875,
+      "learning_rate": 3.4855721049018688e-06,
+      "loss": 0.8707,
+      "step": 4175
+    },
+    {
+      "epoch": 0.84,
+      "grad_norm": 0.88671875,
+      "learning_rate": 3.2822065870462217e-06,
+      "loss": 0.8423,
+      "step": 4200
+    },
+    {
+      "epoch": 0.84,
+      "grad_norm": 0.7578125,
+      "learning_rate": 3.08453618411631e-06,
+      "loss": 0.8448,
+      "step": 4225
+    },
+    {
+      "epoch": 0.85,
+      "grad_norm": 0.8984375,
+      "learning_rate": 2.892612731749414e-06,
+      "loss": 0.8743,
+      "step": 4250
+    },
+    {
+      "epoch": 0.85,
+      "grad_norm": 0.8125,
+      "learning_rate": 2.7064865585446434e-06,
+      "loss": 0.8389,
+      "step": 4275
+    },
+    {
+      "epoch": 0.86,
+      "grad_norm": 0.8046875,
+      "learning_rate": 2.52620647286512e-06,
+      "loss": 0.8437,
+      "step": 4300
+    },
+    {
+      "epoch": 0.86,
+      "grad_norm": 0.92578125,
+      "learning_rate": 2.351819750038828e-06,
+      "loss": 0.7811,
+      "step": 4325
+    },
+    {
+      "epoch": 0.87,
+      "grad_norm": 0.8359375,
+      "learning_rate": 2.183372119961499e-06,
+      "loss": 0.8266,
+      "step": 4350
+    },
+    {
+      "epoch": 0.88,
+      "grad_norm": 0.9296875,
+      "learning_rate": 2.020907755104698e-06,
+      "loss": 0.8374,
+      "step": 4375
+    },
+    {
+      "epoch": 0.88,
+      "grad_norm": 0.80859375,
+      "learning_rate": 1.864469258932397e-06,
+      "loss": 0.9058,
+      "step": 4400
+    },
+    {
+      "epoch": 0.89,
+      "grad_norm": 0.78125,
+      "learning_rate": 1.7140976547289438e-06,
+      "loss": 0.8627,
+      "step": 4425
+    },
+    {
+      "epoch": 0.89,
+      "grad_norm": 0.8359375,
+      "learning_rate": 1.5698323748414124e-06,
+      "loss": 0.8706,
+      "step": 4450
+    },
+    {
+      "epoch": 0.9,
+      "grad_norm": 0.95703125,
+      "learning_rate": 1.4317112503391432e-06,
+      "loss": 0.8315,
+      "step": 4475
+    },
+    {
+      "epoch": 0.9,
+      "grad_norm": 0.80859375,
+      "learning_rate": 1.2997705010932393e-06,
+      "loss": 0.791,
+      "step": 4500
+    },
+    {
+      "epoch": 0.91,
+      "grad_norm": 0.859375,
+      "learning_rate": 1.1740447262784781e-06,
+      "loss": 0.8368,
+      "step": 4525
+    },
+    {
+      "epoch": 0.91,
+      "grad_norm": 0.9375,
+      "learning_rate": 1.0545668953003241e-06,
+      "loss": 0.8122,
+      "step": 4550
+    },
+    {
+      "epoch": 0.92,
+      "grad_norm": 0.84375,
+      "learning_rate": 9.413683391492456e-07,
+      "loss": 0.875,
+      "step": 4575
+    },
+    {
+      "epoch": 0.92,
+      "grad_norm": 0.7265625,
+      "learning_rate": 8.344787421847217e-07,
+      "loss": 0.8642,
+      "step": 4600
+    },
+    {
+      "epoch": 0.93,
+      "grad_norm": 0.90625,
+      "learning_rate": 7.339261343510206e-07,
+      "loss": 0.8466,
+      "step": 4625
+    },
+    {
+      "epoch": 0.93,
+      "grad_norm": 0.78125,
+      "learning_rate": 6.397368838268497e-07,
+      "loss": 0.8382,
+      "step": 4650
+    },
+    {
+      "epoch": 0.94,
+      "grad_norm": 0.83203125,
+      "learning_rate": 5.519356901107358e-07,
+      "loss": 0.8325,
+      "step": 4675
+    },
+    {
+      "epoch": 0.94,
+      "grad_norm": 0.89453125,
+      "learning_rate": 4.7054557754402373e-07,
+      "loss": 0.8688,
+      "step": 4700
+    },
+    {
+      "epoch": 0.94,
+      "grad_norm": 0.859375,
+      "learning_rate": 3.9558788927314407e-07,
+      "loss": 0.8879,
+      "step": 4725
+    },
+    {
+      "epoch": 0.95,
+      "grad_norm": 0.8203125,
+      "learning_rate": 3.270822816527325e-07,
+      "loss": 0.7926,
+      "step": 4750
+    },
+    {
+      "epoch": 0.95,
+      "grad_norm": 0.859375,
+      "learning_rate": 2.650467190910999e-07,
+      "loss": 0.8789,
+      "step": 4775
+    },
+    {
+      "epoch": 0.96,
+      "grad_norm": 0.8203125,
+      "learning_rate": 2.094974693393731e-07,
+      "loss": 0.7915,
+      "step": 4800
+    },
+    {
+      "epoch": 0.96,
+      "grad_norm": 0.8125,
+      "learning_rate": 1.6044909922555974e-07,
+      "loss": 0.8365,
+      "step": 4825
+    },
+    {
+      "epoch": 0.97,
+      "grad_norm": 0.859375,
+      "learning_rate": 1.1791447083465134e-07,
+      "loss": 0.8521,
+      "step": 4850
+    },
+    {
+      "epoch": 0.97,
+      "grad_norm": 0.83203125,
+      "learning_rate": 8.190473813576572e-08,
+      "loss": 0.8351,
+      "step": 4875
+    },
+    {
+      "epoch": 0.98,
+      "grad_norm": 0.87890625,
+      "learning_rate": 5.242934405720879e-08,
+      "loss": 0.8468,
+      "step": 4900
+    },
+    {
+      "epoch": 0.98,
+      "grad_norm": 0.92578125,
+      "learning_rate": 2.9496018010233274e-08,
+      "loss": 0.8121,
+      "step": 4925
+    },
+    {
+      "epoch": 0.99,
+      "grad_norm": 0.9296875,
+      "learning_rate": 1.3110773862126669e-08,
+      "loss": 0.8535,
+      "step": 4950
+    },
+    {
+      "epoch": 0.99,
+      "grad_norm": 0.87109375,
+      "learning_rate": 3.2779083591949478e-09,
+      "loss": 0.9272,
+      "step": 4975
+    },
+    {
+      "epoch": 1.0,
+      "grad_norm": 0.8359375,
+      "learning_rate": 0.0,
+      "loss": 0.8806,
+      "step": 5000
+    }
+  ],
+  "logging_steps": 25,
+  "max_steps": 5000,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 9223372036854775807,
+  "save_steps": 500,
+  "total_flos": 9.7705141469184e+17,
+  "train_batch_size": 1,
+  "trial_name": null,
+  "trial_params": null
+}

training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3460479bf9efa1f6bc1b563fc6a790fb29cfb70a59fda8cbb0bab8e475316c7e
+size 4856