MInference/bfd
Viewer
•
Updated
•
9.31k
•
65
MInference
AI & ML interests
None defined yet.
Recent Activity
View all activity
MInference's activity
liyuchengÂ
updated
a
dataset
11 days ago
liyuchengÂ
updated
a
model
12 days ago
liyuchengÂ
published
a
model
12 days ago
liyuchengÂ
published
a
dataset
17 days ago
liyuchengÂ
updated
2
models
about 2 months ago
liyuchengÂ
published
a
model
about 2 months ago
liyuchengÂ
updated
a
dataset
about 2 months ago
liyuchengÂ
published
a
dataset
about 2 months ago
liyuchengÂ
published
a
model
about 2 months ago
liyuchengÂ
updated
a
dataset
about 2 months ago
liyuchengÂ
authored
a
paper
3 months ago
iofu728Â
authored
a
paper
3 months ago
iofu728Â
updated
a
dataset
3 months ago
liyuchengÂ
updated
a
dataset
3 months ago
liyuchengÂ
updated
a
dataset
4 months ago
iofu728Â
authored
a
paper
6 months ago
liyuchengÂ
authored
a
paper
7 months ago
Post
1118
Weclome to use MInference, which leverages the dynamic sparse nature of LLMs' attention, which exhibits some static patterns, to speed up the pre-filling for million tokens LLMs. It first determines offline which sparse pattern each head belongs to, then approximates the sparse index online and dynamically computes attention with the optimal custom kernels. This approach achieves up to a 10x speedup for pre-filling on an A100 while maintaining accuracy with 1M tokens.
For more detail please check,
project page: https://aka.ms/MInference
code: https://github.com/microsoft/MInference
paper: MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attention (2407.02490)
hf demo: microsoft/MInference
For more detail please check,
project page: https://aka.ms/MInference
code: https://github.com/microsoft/MInference
paper: MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attention (2407.02490)
hf demo: microsoft/MInference
liyuchengÂ
authored
a
paper
8 months ago