

TITLE
Adding kNN language modeling and Machine Translation

COMMENTS
6

REACTIONS
+1: 0
-1: 0
laugh: 0
hooray: 0
heart: 6
rocket: 0
eyes: 0

BODY
# 🌟 Adding the kNN-LM and kNN-MT models

## Model description

The kNN [Language Model](https://arxiv.org/pdf/1911.00172.pdf) and [Machine Translation](https://arxiv.org/pdf/2010.00710.pdf) leverage additional data at test time by looking up examples in the data store that are similar to the test example and using them to inform the prediction. This leads to decreased perplexity for language modeling and increased BLEU for MT.

Ideally, we'd have a general KNNDecoder class which can use any model in the library along with a [🤗datasets](https://github.com/huggingface/datasets) data store, similarly to what RAG does currently.

## Open source status

* [x] the model implementation is available: for kNN-LM, an implementation using Fairseq and FAISS can be found [here](https://github.com/urvashik/knnlm)
* [x] the model weights are available: the methods use pre-trained models which are already in the library, [GPT-2](https://huggingface.co/gpt2) and the [Facebook WMT'19 models](https://huggingface.co/facebook/wmt19-de-en)
* [x] who are the authors: first author @urvashik
