text: 
  
  TITLE
  RAG - reproducing RAG-Sequence QA score 
  
  COMMENTS
  4
  
  REACTIONS
  +1: 0
  -1: 0
  laugh: 0
  hooray: 0
  heart: 0
  rocket: 0
  eyes: 0
  
  BODY
  I'm trying to reproduce RAG-Sequence NQ score of 44.5 presented in Table 1 of the paper at https://arxiv.org/abs/2005.11401.
  
  I used the command in the examples/rag readme
  ```bash
  python examples/rag/eval_rag.py \
      --model_name_or_path facebook/rag-sequence-nq \
      --model_type rag_sequence \
      --evaluation_set path/to/test.source \
      --gold_data_path path/to/gold_data \
      --predictions_path path/to/e2e_preds.txt \
      --eval_mode e2e \
      --gold_data_mode qa \
      --n_docs 5 \
      --print_predictions \
      --recalculate \
  ```
  
  For gold_data_path I used data.retriever.qas.nq-test from DPR repo, consisting of 3610 questions and answers: https://github.com/facebookresearch/DPR/blob/master/data/download_data.py#L91-L97
  
  For evaluation_set, my understanding it should be the questions, so I extracted just the questions from the qas.nq-test csv file.
  
  I tried the above command with n_docs 5 and 10, with the following results:
  
  n_docs 5
  INFO:__main__:F1: 49.67
  INFO:__main__:EM: 42.58
  
  n_docs 10
  INFO:__main__:F1: 50.62
  INFO:__main__:EM: 43.49
  
  With n_docs 10 it's still 1 point below the score in paper. What would be the proper setup to reproduce the number, is the pretrained model loaded different, higher n_docs, or different test data?
  
  Thanks in advance!