text: 
  
  TITLE
  Turning the SQuAD dataset class into an iterator to save ram and redistribute time
  
  COMMENTS
  1
  
  REACTIONS
  +1: 0
  -1: 0
  laugh: 0
  hooray: 0
  heart: 0
  rocket: 0
  eyes: 0
  
  BODY
  # 🚀 Feature request
  When I'm using the SquadDataset class I sometimes run out of RAM in my Colab session. It also takes a lot of time to process the data, so if we could just preprocess the data when using the __getitem__. I think we should turn the examples into an iterator rather than a list. And when using the __getitem__ we should then convert the example to a feature
  
  ## Motivation
  As I mentioned, this can save a lot of ram and redistribute the time, so we do the preprocessing simultaneously.
  
  When I run this snippet in Google Colab it runs out of memory even though I've 12 GB of ram
  ```python
  train_dataset = SquadDataset(
      args=data_args,
      tokenizer=tokenizer,
      cache_dir=model_args.cache_dir)
  ```
  
  <!-- Please outline the motivation for the proposal. Is your feature request
       related to a problem? e.g., I'm always frustrated when [...]. If this is related
       to another GitHub issue, please link here too. -->
  
  
  
  ## Your contribution
  I can pretty much code all of it we you believe it would be a necessary feature
  <!-- Is there any way that you could help, e.g. by submitting a PR?
       Make sure to read the CONTRIBUTING.MD readme:
       https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md -->
  