text: 
  
  TITLE
  @ehsan-soe I fixed the problem by truncating incomplete batches. So if there are 2001 examples and my batch size = 2, then I truncate the last example and train on the first 2000. This has fixed it for me both with and without distributed. My load_and_cache function now looks like this
  
  COMMENTS
  2
  
  REACTIONS
  +1: 0
  -1: 0
  laugh: 0
  hooray: 0
  heart: 0
  rocket: 0
  eyes: 0
  
  BODY
  @ehsan-soe I fixed the problem by truncating incomplete batches. So if there are 2001 examples and my batch size = 2, then I truncate the last example and train on the first 2000. This has fixed it for me both with and without distributed. My load_and_cache function now looks like this
  ```
  def load_and_cache_examples(args, tokenizer, evaluate=False, fpath=None):
      if fpath:
          dataset = TextDataset(tokenizer, args, fpath)
      else:
          dataset = TextDataset(tokenizer, args, args.eval_data_path if evaluate else args.train_data_path)
  
      # Ignore incomplete batches
      # If you don't do this, you'll get an error at the end of training
      n = len(dataset) % args.per_gpu_train_batch_size
      if n != 0:
          dataset.examples = dataset.examples[:-n]
      return dataset
  ```
  
  _Originally posted by @isabelcachola in https://github.com/huggingface/transformers/issues/1220#issuecomment-557237248_