In Search of Needles in a 10M Haystack

Recurrent Memory Finds What LLMs Miss

paper addresses the challenge of processing long documents using generative transformer models. To evaluate different approaches, we introduce BABILong, a new benchmark designed to assess model capabilities in extracting and processing distributed facts within extensive texts. Our evaluation, which includes benchmarks for GPT-4 and RAG, reveals that common methods are effective only for sequences up to 10^4 elements. In contrast, fine-tuning GPT-2 with recurrent memory augmentations enables it to handle tasks involving up to 10^7 elements. This achievement marks a substantial leap, as it is by far the longest input processed by any open neural network model to date, demonstrating a significant improvement in the processing capabilities for long sequences

liked a model about 1 year ago

01-ai/Yi-VL-34B

Image-Text-to-Text • Updated Jun 26, 2024 • 90 • 262

updated a dataset over 1 year ago

jclian91/open_domain_triple_extraction

Updated Sep 29, 2023 • 34

liked a model over 1 year ago

meta-llama/Llama-2-7b

Text Generation • Updated Apr 17, 2024 • 4.23k

updated a dataset over 1 year ago

jclian91/people_relation_classification

Viewer • Updated Jul 30, 2023 • 3.88k • 51