Topic 23: What is LLM Inference, it's challenges and solutions for it
•
2
Ksenia Se
Kseniase
AI & ML interests
None yet
Recent Activity
posted
an
update
about 14 hours ago
Today, we spoke with Snowflake’s AI Research Team Leads, Yuxiong He and Samyam Rajbhandari (@samyam) (he is also one the researchers behind https://huggingface.co/papers/2401.08671 and other DeepSpeed papers)
Collaborating with their co-authors to reduce inference costs for enterprise-specific tasks, they observed that inputs are often significantly larger than outputs. This is because it’s in the nature of enterprises to analyze enormous amounts of information trying to extract valuable insights, which are much shorter. To address this, they developed SwiftKV https://huggingface.co/papers/2410.03960, an optimization that reduces LLM inference costs by up to 75% for Meta Llama LLMs, enhancing efficiency and performance in enterprise AI tasks.
Today they are open-sourcing SwiftKV (https://huggingface.co/Snowflake/Llama-3.1-SwiftKV-8B-Instruct) and ArcticTrainging Platform.
In our new episode "15 minutes with a Researcher" they explain how SwiftKV works, its applicability to other architectures, its limitations, and additional methods to further reduce computation costs in inference.
Watch the full 15 min interview here (https://youtu.be/9x1k7eXe-6Q?si=4_HQOyi1CPHgvlrx)
published
an
article
about 15 hours ago
Topic 23: What is LLM Inference, it's challenges and solutions for it
upvoted
an
article
4 days ago
🌁#83: GAN is back
Articles
Organizations
Kseniase's activity
liked
a
Space
10 days ago
liked
a
Space
about 1 month ago