text: 
  
  TITLE
  RuntimeError: found torch.cuda.HalfTensor expected torch.cuda.FloatTensor while fine-tuning RAGSequence-base with custom data
  
  COMMENTS
  5
  
  REACTIONS
  +1: 0
  -1: 0
  laugh: 0
  hooray: 0
  heart: 0
  rocket: 0
  eyes: 0
  
  BODY
  ## Environment info
       
  - `transformers` version: 4.0.0-rc-1 (master)
  - Platform: Linux-3.10.0-862.el7.x86_64-x86_64-with-centos-7.5.1804-Core
  - Python version: 3.6.5
  - PyTorch version (GPU?): 1.5.0 (True)
  - Tensorflow version (GPU?): not installed (NA)
  - Using GPU in script?: yes
  - Using distributed or parallel set-up in script?: yes
  
  ### Who can help
   RAG: @patrickvonplaten, @lhoestq
  
  ## Information
  I am fine-tuning RAGSequence-base according to the fine-tuning examples, along with a custom knowledge dataset (<10 lines)
  
  The problem arises when using:
  * [x] the official example scripts: (give details below)
  * [ ] my own modified scripts: (give details below)
  
  The tasks I am working on is:
  * [ ] an official GLUE/SQUaD task: (give the name)
  * [x] my own task or dataset: (give details below)
  
  ## To reproduce
  
  Steps to reproduce the behavior:
  
  1. Generated the faiss index and the custom embeddings.
  2. Executed `finetune_rag.py` with the same parameters as the given shell script. (except reducing epoch to 1 for test)
  3. After going through the specified epoch, it abruptly ends with the following runtime error.
  
  ```python
  Traceback (most recent call last):
    File "transformers/examples/rag/finetune_rag.py", line 512, in <module>
      main()
    File "transformers/examples/rag/finetune_rag.py", line 507, in main
      trainer.test()
    File "/lib/python3.6/site-packages/pytorch_lightning/trainer/trainer.py", line 721, in test
      results = self.__test_using_best_weights(ckpt_path, test_dataloaders)
    File "/lib/python3.6/site-packages/pytorch_lightning/trainer/trainer.py", line 763, in __test_using_best_weights
      results = self.fit(model)
    File "/python3.6/site-packages/pytorch_lightning/trainer/trainer.py", line 445, in fit
      results = self.accelerator_backend.train()
    File "/python3.6/site-packages/pytorch_lightning/accelerators/ddp_accelerator.py", line 148, in train
      results = self.ddp_train(process_idx=self.task_idx, model=model)
    File "/lib/python3.6/site-packages/pytorch_lightning/accelerators/ddp_accelerator.py", line 269, in ddp_train
      model = self.trainer.precision_connector.connect(model)
    File "/lib/python3.6/site-packages/pytorch_lightning/trainer/connectors/precision_connector.py", line 78, in connect
      model, optimizers = self.backend.connect(model, self.trainer.optimizers)
    File "/lib/python3.6/site-packages/pytorch_lightning/plugins/apex.py", line 37, in connect
      model, optimizers = self.configure_apex(amp, model, optimizers, self.trainer.amp_level)
    File "/lib/python3.6/site-packages/pytorch_lightning/plugins/apex.py", line 102, in configure_apex
      model, optimizers = amp.initialize(model, optimizers, opt_level=amp_level)
    File "/lib/python3.6/site-packages/apex/amp/frontend.py", line 358, in initialize
      return _initialize(models, optimizers, _amp_state.opt_properties, num_losses, cast_model_outputs)
    File "/lib/python3.6/site-packages/apex/amp/_initialize.py", line 171, in _initialize
      check_params_fp32(models)
    File "/lib/python3.6/site-packages/apex/amp/_initialize.py", line 87, in check_params_fp32
      name, param.type()))
    File "/lib/python3.6/site-packages/apex/amp/_amp_state.py", line 32, in warn_or_err
      raise RuntimeError(msg)
  RuntimeError: Found param model.rag.question_encoder.question_encoder.bert_model.embeddings.word_embeddings.weight with type torch.cuda.HalfTensor, expected torch.cuda.FloatTensor.
  When using amp.initialize, you do not need to call .half() on your model
  before passing it, no matter what optimization level you choose.
  ```
  To add, I am using Nvidia's Apex library for --fp16 training.
  
  ## Expected behavior
  fine-tuning to complete, with generated models in the output directory.
  
  Have spent a couple hours tinkering with no clue how to proceed, any help would be appreciated. Thank you!