michaelfeil
commited on
Commit
·
eea668b
1
Parent(s):
a19a70f
Upload Salesforce/codet5p-770m ctranslate2 weights
Browse files
README.md
CHANGED
@@ -14,26 +14,6 @@ quantized version of [Salesforce/codet5p-770m](https://huggingface.co/Salesforce
|
|
14 |
pip install ctranslate2
|
15 |
```
|
16 |
|
17 |
-
```python
|
18 |
-
# from transformers import AutoTokenizer
|
19 |
-
model_name = "OpenNMT/codet5p-770m-ct2-int8"
|
20 |
-
|
21 |
-
|
22 |
-
from hf_hub_ctranslate2 import TranslatorCT2fromHfHub
|
23 |
-
model = TranslatorCT2fromHfHub(
|
24 |
-
# load in int8 on CUDA
|
25 |
-
model_name_or_path=model_name,
|
26 |
-
device="cuda",
|
27 |
-
compute_type="int8_float16",
|
28 |
-
# tokenizer=AutoTokenizer.from_pretrained("{ORG}/{NAME}")
|
29 |
-
)
|
30 |
-
outputs = model.generate(
|
31 |
-
text=["def fibonnaci(", "User: How are you doing? Bot:"],
|
32 |
-
max_length=64,
|
33 |
-
)
|
34 |
-
print(outputs)
|
35 |
-
```
|
36 |
-
|
37 |
Checkpoint compatible to [ctranslate2>=3.22.0](https://github.com/OpenNMT/CTranslate2)
|
38 |
- `compute_type=int8_float16` for `device="cuda"`
|
39 |
- `compute_type=int8` for `device="cpu"`
|
|
|
14 |
pip install ctranslate2
|
15 |
```
|
16 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
17 |
Checkpoint compatible to [ctranslate2>=3.22.0](https://github.com/OpenNMT/CTranslate2)
|
18 |
- `compute_type=int8_float16` for `device="cuda"`
|
19 |
- `compute_type=int8` for `device="cpu"`
|