Update README.md

README.md

@@ -113,3 +113,53 @@ EleutherAI/pythia-160m is a causal language model with approximately 162 million
 - **Hardware:** Tesla T4 GPU
 - **Software:** PyTorch 2.4.1+cu121, Transformers 4.44.2
 - **Date of Evaluation:** October 18, 2024
+
+### Command
+
+```
+lm_eval --model hf \
+    --model_args pretrained=EleutherAI/pythia-160m,revision=step100000,dtype="float" \
+    --tasks hellaswag \
+    --device cuda \
+    --batch_size auto:4 \
+    --output_path hellaswag_test \
+    --log_samples
+```
+
+#### Command output
+```
+Passed argument batch_size = auto:4.0. Detecting largest batch size
+Determined largest batch size: 64
+Passed argument batch_size = auto:4.0. Detecting largest batch size
+Determined largest batch size: 64
+hf (pretrained=EleutherAI/pythia-160m,revision=step100000,dtype=float), gen_kwargs: (None), limit: None, num_fewshot: None, batch_size: auto:4 (64,64,64,64,64)
+|  Tasks  |Version|Filter|n-shot| Metric |   |Value |   |Stderr|
+|---------|------:|------|-----:|--------|---|-----:|---|-----:|
+|hellaswag|      1|none  |     0|acc     |↑  |0.2872|±  |0.0045|
+|         |       |none  |     0|acc_norm|↑  |0.3082|±  |0.0046|
+
+2024-10-18 12:25:25.770584: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:485] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
+2024-10-18 12:25:25.847675: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:8454] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
+2024-10-18 12:25:25.887843: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1452] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
+2024-10-18 12:25:25.961158: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
+To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
+2024-10-18 12:25:27.647707: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
+2024-10-18:12:25:29,450 INFO     [__main__.py:279] Verbosity set to INFO
+2024-10-18:12:25:42,060 INFO     [__main__.py:376] Selected Tasks: ['hellaswag']
+2024-10-18:12:25:42,062 INFO     [evaluator.py:164] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
+2024-10-18:12:25:42,062 INFO     [evaluator.py:201] Initializing hf model, with arguments: {'pretrained': 'EleutherAI/pythia-160m', 'revision': 'step100000', 'dtype': 'float'}
+2024-10-18:12:25:42,128 INFO     [huggingface.py:129] Using device 'cuda'
+2024-10-18:12:25:42,395 INFO     [huggingface.py:481] Using model type 'default'
+/usr/local/lib/python3.10/dist-packages/transformers/tokenization_utils_base.py:1601: FutureWarning: `clean_up_tokenization_spaces` was not set. It will be set to `True` by default. This behavior will be depracted in transformers v4.45, and will be then set to `False` by default. For more details check this issue: https://github.com/huggingface/transformers/issues/31884
+  warnings.warn(
+2024-10-18:12:25:42,769 INFO     [huggingface.py:365] Model parallel was set to False, max memory was not set, and device map was set to {'': 'cuda'}
+2024-10-18:12:25:56,709 WARNING  [model.py:422] model.chat_template was called with the chat_template set to False or None. Therefore no chat template will be applied. Make sure this is an intended behavior.
+2024-10-18:12:25:56,710 INFO     [task.py:415] Building contexts for hellaswag on rank 0...
+100%|██████████| 10042/10042 [00:05<00:00, 1695.72it/s]
+2024-10-18:12:26:04,007 INFO     [evaluator.py:489] Running loglikelihood requests
+Running loglikelihood requests: 100%|██████████| 40168/40168 [03:53<00:00, 171.85it/s]
+fatal: not a git repository (or any of the parent directories): .git
+2024-10-18:12:30:36,510 INFO     [evaluation_tracker.py:206] Saving results aggregated
+2024-10-18:12:30:36,524 INFO     [evaluation_tracker.py:287] Saving per-sample results for: hellaswag
+```
+