SGEcon commited on
Commit
459f367
1 Parent(s): 96dfd64

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +2 -2
README.md CHANGED
@@ -84,11 +84,11 @@ If you wish to use the original data, please contact the original author directl
84
 
85
  ## Training Details
86
 
87
- We use QLora to train the base model.
88
  Quantized Low Rank Adapters (QLoRA) is an efficient technique that uses 4-bit quantized pre-trained language models to fine-tune 65 billion parameter models on a 48 GB GPU while significantly reducing memory usage.
89
  The method uses NormalFloat 4-bit (NF4), a new data type that is theoretically optimal for normally distributed weights; Double Quantization, which further quantizes quantization constants to reduce average memory usage; and Paged Optimizers, which manage memory spikes during mini-batch processing, to increase memory efficiency without sacrificing performance.
90
 
91
- Also, we performed instruction tuning using the data that we collected and the kyujinpy/KOR-OpenOrca-Platypus-v3 dataset on the hugging face.
92
  Instruction tuning is learning in a supervised learning format that uses instructions and input data together as input and output data as a pair.
93
 
94
 
 
84
 
85
  ## Training Details
86
 
87
+ - We use QLora to train the base model.
88
  Quantized Low Rank Adapters (QLoRA) is an efficient technique that uses 4-bit quantized pre-trained language models to fine-tune 65 billion parameter models on a 48 GB GPU while significantly reducing memory usage.
89
  The method uses NormalFloat 4-bit (NF4), a new data type that is theoretically optimal for normally distributed weights; Double Quantization, which further quantizes quantization constants to reduce average memory usage; and Paged Optimizers, which manage memory spikes during mini-batch processing, to increase memory efficiency without sacrificing performance.
90
 
91
+ - Also, we performed instruction tuning using the data that we collected and the kyujinpy/KOR-OpenOrca-Platypus-v3 dataset on the hugging face.
92
  Instruction tuning is learning in a supervised learning format that uses instructions and input data together as input and output data as a pair.
93
 
94