regisss HF staff commited on
Commit
1c4e7ab
1 Parent(s): 3590e64

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +45 -0
README.md CHANGED
@@ -1,3 +1,48 @@
1
  ---
2
  license: apache-2.0
3
  ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  ---
2
  license: apache-2.0
3
  ---
4
+
5
+ [Optimum Habana](https://github.com/huggingface/optimum-habana) is the interface between the Transformers library and Habana's Gaudi processor (HPU). It provides a set of tools enabling easy and fast model loading and fine-tuning on single- and multi-HPU settings for different downstream tasks.
6
+ Learn more about how to take advantage of the power of Habana HPUs to train Transformers models at [hf.co/Habana](https://huggingface.co/Habana).
7
+
8
+ ## BERT Large model HPU configuration
9
+
10
+ This model contains just the `GaudiConfig` file for running the [bert-large-uncased-whole-word-masking](https://huggingface.co/bert-large-uncased-whole-word-masking) model on Habana's Gaudi processors (HPU).
11
+
12
+ **This model contains no model weights, only a GaudiConfig.**
13
+
14
+ This enables to specify:
15
+ - `use_habana_mixed_precision`: whether to use Habana Mixed Precision (HMP)
16
+ - `hmp_opt_level`: optimization level for HMP, see [here](https://docs.habana.ai/en/latest/PyTorch/PyTorch_User_Guide/PT_Mixed_Precision.html#configuration-options) for a detailed explanation
17
+ - `hmp_bf16_ops`: list of operators that should run in bf16
18
+ - `hmp_fp32_ops`: list of operators that should run in fp32
19
+ - `hmp_is_verbose`: verbosity
20
+ - `use_fused_adam`: whether to use Habana's custom AdamW implementation
21
+ - `use_fused_clip_norm`: whether to use Habana's fused gradient norm clipping operator
22
+
23
+ ## Usage
24
+
25
+ The model is instantiated the same way as in the Transformers library.
26
+ The only difference is that the Gaudi configuration has to be loaded and provided to the trainer:
27
+
28
+ ```
29
+ from optimum.habana import GaudiConfig, GaudiTrainer, GaudiTrainingArguments
30
+ from transformers import BertTokenizer, BertModel
31
+
32
+ tokenizer = BertTokenizer.from_pretrained("bert-large-uncased-whole-word-masking")
33
+ model = BertModel.from_pretrained("bert-large-uncased-whole-word-masking")
34
+ gaudi_config = GaudiConfig.from_pretrained("Habana/bert-large-uncased-whole-word-masking")
35
+ args = GaudiTrainingArguments(
36
+ output_dir="/tmp/output_dir",
37
+ use_habana=True,
38
+ use_lazy_mode=True,
39
+ )
40
+
41
+ trainer = GaudiTrainer(
42
+ model=model,
43
+ gaudi_config=gaudi_config,
44
+ args=args,
45
+ tokenizer=tokenizer,
46
+ )
47
+ trainer.train()
48
+ ```