metadata
license: creativeml-openrail-m
language:
- en
tags:
- LLM
- tensorRT
- chatGLM
Model Card for lyraChatGLM
lyraChatGLM is currently the fastest chatGLM-6B available, as far as we know, it is also the fisrt accelerated version of chatGLM-6B.
The inference speed of lyraChatGLM is 10x faster than the original version, and we're still working to improve the performance.
Among its main features are:
- weights: original ChatGLM-6B weights released by THUDM.
- device: lyraChatGLM is mainly based on FasterTransformer compiled for SM=80 (A100, for example).
Speed
test environment
- device: Nvidia A100 40G
- img size: 512x512
- percision:fp16
- steps: 30
- solver: LMSD
text2img
Model Sources
- Repository: [https://huggingface.co/THUDM/chatglm-6b]
Uses
from faster_chat_glm import GLM6B, FasterChatGLM
# kernel for chat model.
kernel = GLM6B(plan_path=plan_path,
batch_size=BATCH_SIZE,
num_beams=1,
use_cache=USE_CACHE,
num_heads=32,
emb_size_per_heads=128,
decoder_layers=28,
vocab_size=150528,
max_seq_len=MAX_OUT_LEN)
chat = FasterChatGLM(model_dir=chatglm6b_dir, kernel=kernel).half().cuda()
# generate
sample_output = chat.generate(inputs=input_ids, max_length=MAX_OUT_LEN)
Demo output
text2img
img2img
Environment
- hardware: Nvidia Ampere architecture (A100) or compatable
- docker image avaible: https://hub.docker.com/r/bigmoyan/lyra_aigc/tags
docker pull bigmoyan/lyra_aigc:v0.1
Citation
@Misc{lyraChatGLM2023,
author = {Kangjian Wu, Zhengtao Wang, Bin Wu},
title = {lyaraChatGLM: Accelerating chatGLM by 10x+},
howpublished = {\url{https://huggingface.co/TMElyralab/lyraChatGLM}},
year = {2023}
}
Report bug
- start a discussion to report any bugs!--> https://huggingface.co/TMElyralab/lyraChatGLM/discussions
- report bug with a
[bug]mark in the title.