Hugging Face's logo

TMElyralab
/
lyraChatGLM

English
LLM
ChatGLM6B
Model card Files Community
49
lyraChatGLM
File size: 2,178 Bytes 
d1850c3
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
 
---
license: creativeml-openrail-m
language:
- en
tags:
- LLM
- tensorRT
- chatGLM
---
## Model Card for lyraChatGLM

lyraChatGLM is currently the **fastest chatGLM-6B** available, as far as we know, it is also the **fisrt accelerated version of chatGLM-6B**.

The inference speed of lyraChatGLM is **10x** faster than the original version, and we're still working to improve the performance.

Among its main features are:

- weights: original ChatGLM-6B weights released by THUDM.
- device: lyraChatGLM is mainly based on FasterTransformer compiled for SM=80 (A100, for example).

## Speed

### test environment

- device: Nvidia A100 40G
- img size: 512x512
- percision:fp16
- steps: 30
- solver: LMSD

### text2img




## Model Sources

- **Repository:** [https://huggingface.co/THUDM/chatglm-6b]

## Uses

```python
from faster_chat_glm import GLM6B, FasterChatGLM

# kernel for chat model.
kernel = GLM6B(plan_path=plan_path,
               batch_size=BATCH_SIZE,
               num_beams=1,
               use_cache=USE_CACHE,
               num_heads=32,
               emb_size_per_heads=128,
               decoder_layers=28,
               vocab_size=150528,
               max_seq_len=MAX_OUT_LEN)

chat = FasterChatGLM(model_dir=chatglm6b_dir, kernel=kernel).half().cuda()

# generate
sample_output = chat.generate(inputs=input_ids, max_length=MAX_OUT_LEN)

```
## Demo output

### text2img
![text2img_demo](./output/text2img_demo.jpg)

### img2img

![text2img_demo](./output/img2img_input.jpg)

![text2img_demo](./output/img2img_demo.jpg)



## Environment

- hardware: Nvidia Ampere architecture (A100) or compatable
- docker image avaible: https://hub.docker.com/r/bigmoyan/lyra_aigc/tags
```
docker pull bigmoyan/lyra_aigc:v0.1
```

## Citation
``` bibtex
@Misc{lyraChatGLM2023,
  author =       {Kangjian Wu, Zhengtao Wang, Bin Wu},
  title =        {lyaraChatGLM: Accelerating chatGLM by 10x+},
  howpublished = {\url{https://huggingface.co/TMElyralab/lyraChatGLM}},
  year =         {2023}
}
```

## Report bug
- start a discussion to report any bugs!--> https://huggingface.co/TMElyralab/lyraChatGLM/discussions
- report bug with a `[bug]` mark in the title.