text: 
  
  TITLE
  Resource exhausted when training in loop
  
  COMMENTS
  1
  
  REACTIONS
  +1: 0
  -1: 0
  laugh: 0
  hooray: 0
  heart: 0
  rocket: 0
  eyes: 0
  
  BODY
  ## Environment info     
  - `transformers` version: 3.4.0
  - Platform: Linux-4.15.0-109-generic-x86_64-with-Ubuntu-18.04-bionic
  - Python version: 3.7.5
  - PyTorch version (GPU?): 1.4.0 (True)
  - Tensorflow version (GPU?): 2.3.1 (True)
  - Using GPU in script?: Yes
  - Using distributed or parallel set-up in script?: No
  
  ### Who can help
  @LysandreJik , @thomwolf or @jplu maybe?
  
  ## Information
  
  Model I am using Bert:
  
  The problem arises when using:
  * [ ] the official example scripts: (give details below)
  * [X] my own modified scripts: (give details below)
  
  The tasks I am working on is:
  * [ ] an official GLUE/SQUaD task: (give the name)
  * [X] my own task or dataset: (give details below)
  
  ## To reproduce
  To reproduce this error, one can use the script below. I have taken it from a blog post I made and altered it a bit - so it is a bit long. The main thing however, seems to be that if I load and fit a model several times it will at some point make my GPU run out of memory. Here I use kerastuner for hyperparam optimization and after 7 trials my GPU runs out of memory and thoughs the error below:
  `  (0) Resource exhausted:  OOM when allocating tensor with shape[3200,3072] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc         [[node BERT_MultiLabel_MultiClass/tf_bert_model/bert/encoder/layer_._11/intermediate/dense/Tensordot/MatMul (defined at myLib/python3.7/site-packages/transformers/modeling_tf_bert.py:327) ]]
  `
  
  ### Reproducible script
  ```
  #######################################
  ### -------- Load libraries ------- ###
  
  # Load Huggingface transformers
  from transformers import TFBertModel, BertConfig, BertTokenizerFast
  
  # Then what you need from tensorflow.keras
  from tensorflow.keras.layers import Input, Dropout, Dense
  from tensorflow.keras.models import Model
  from tensorflow.keras.optimizers import Adam
  from tensorflow.keras.callbacks import EarlyStopping
  from tensorflow.keras.initializers import TruncatedNormal
  from tensorflow.keras.losses import CategoricalCrossentropy
  from tensorflow.keras.metrics import CategoricalAccuracy
  from tensorflow.keras.utils import to_categorical
  
  # And pandas for data import + sklearn because you allways need sklearn
  import pandas as pd
  from sklearn.model_selection import train_test_split
  
  
  #######################################
  ### --------- Import data --------- ###
  
  # Import data from csv
  data = pd.read_csv('dev/Fun with BERT/complaints.csv')
  
  # Select required columns
  data = data[['Consumer complaint narrative', 'Product', 'Issue']]
  
  # Remove a row if any of the three remaining columns are missing
  data = data.dropna()
  data = data.sample(n = 5000)
  
  # Remove rows, where the label is present only ones (can't be split)
  data = data.groupby('Issue').filter(lambda x : len(x) > 1)
  data = data.groupby('Product').filter(lambda x : len(x) > 1)
  
  # Set your model output as categorical and save in new label col
  data['Issue_label'] = pd.Categorical(data['Issue'])
  data['Product_label'] = pd.Categorical(data['Product'])
  
  # Transform your output to numeric
  data['Issue'] = data['Issue_label'].cat.codes
  data['Product'] = data['Product_label'].cat.codes
  
  # Split into train and test - stratify over Issue
  data, data_test = train_test_split(data, test_size = 0.2, stratify = data[['Issue']])
  
  
  #######################################
  ### --------- Setup BERT ---------- ###
  
  # Name of the BERT model to use
  model_name = 'bert-base-uncased'
  
  # Max length of tokens
  max_length = 100
  
  # Load transformers config and set output_hidden_states to False
  config = BertConfig.from_pretrained(model_name)
  config.output_hidden_states = False
  
  # Load BERT tokenizer
  tokenizer = BertTokenizerFast.from_pretrained(pretrained_model_name_or_path = model_name, config = config)
  
  
  
  #######################################
  ### ------- Build the model ------- ###
  
  def model_build(hp):
      # TF Keras documentation: https://www.tensorflow.org/api_docs/python/tf/keras/Model
  
      # Load the Transformers BERT model
      transformer_model = TFBertModel.from_pretrained(model_name, config = config)
  
      # Build your model input
      input_ids = Input(shape=(max_length,), name='input_ids', dtype='int32')
      # attention_mask = Input(shape=(max_length,), name='attention_mask', dtype='int32') 
      # inputs = {'input_ids': input_ids, 'attention_mask': attention_mask}
      inputs = {'input_ids': input_ids}
  
      # Load the Transformers BERT model as a layer in a Keras model
      bert_model = transformer_model(inputs)[1]
      dropout = Dropout(hp.Float('hp_dropout', min_value=0, max_value=0.99, default = 0.2), name='pooled_output')
      pooled_output = dropout(bert_model, training=False)
  
      # Then build your model output
      issue = Dense(units=len(data.Issue_label.value_counts()), kernel_initializer=TruncatedNormal(stddev=hp.Float('hp_stddev_issue', min_value=0, max_value=1, default = 0.02)), name='issue')(pooled_output)
      product = Dense(units=len(data.Product_label.value_counts()), kernel_initializer=TruncatedNormal(stddev=hp.Float('hp_stddev_product', min_value=0, max_value=1, default = 0.02)), name='product')(pooled_output)
      outputs = {'issue': issue, 'product': product}
  
      # And combine it all in a model object
      model = Model(inputs=inputs, outputs=outputs, name='BERT_MultiLabel_MultiClass')
  
      # Set an optimizer
      optimizer = Adam(
          learning_rate=5e-05,
          epsilon=1e-08,
          decay=0.01,
          clipnorm=1.0)
  
      # Set loss and metrics
      loss = {'issue': CategoricalCrossentropy(from_logits = True), 'product': CategoricalCrossentropy(from_logits = True)}
      metric = {'issue': CategoricalAccuracy('accuracy'), 'product': CategoricalAccuracy('accuracy')}
  
      # Compile the model
      model.compile(
          optimizer = optimizer,
          loss = loss, 
          metrics = metric)
      
      return model
  
  
  from kerastuner import HyperModel
  class ClsModel(HyperModel):
  
      def build(self, fitting_param = None):
          
          model = model_build()
  
          return model
  
  
  
  #######################################
  ### ------- Train the model ------- ###
  
  
  # Ready output data for the model
  y_issue = to_categorical(data['Issue'])
  y_product = to_categorical(data['Product'])
  
  # Tokenize the input (takes some time)
  x = tokenizer(
      text=data['Consumer complaint narrative'].to_list(),
      add_special_tokens=True,
      max_length=max_length,
      truncation=True,
      padding=True, 
      return_tensors='tf',
      return_token_type_ids = False,
      return_attention_mask = True,
      verbose = True)
  
  from kerastuner.tuners import BayesianOptimization
  tuner = BayesianOptimization(
      model_build,
      objective='loss',
      max_trials=50,
      executions_per_trial=5,
      project_name='modeltest')
  
  tuner.search(
      x={'input_ids': x['input_ids']},
      y={'issue': y_issue, 'product': y_product},
      epochs=5)
  ```
  
  ### Full error log
  Or allmost - removed some 100 lines of `2020-11-04 16:53:37.6569xx: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size xxxxxxxx totalling xxxMiB `
  
  ```
  2020-11-04 16:53:37.656906: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 2405376 totalling 2.29MiB  
  2020-11-04 16:53:37.656913: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 2457600 totalling 2.34MiB  
  2020-11-04 16:53:37.656919: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 2950912 totalling 2.81MiB  
  2020-11-04 16:53:37.656926: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 2967552 totalling 2.83MiB  
  2020-11-04 16:53:37.656933: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 3091200 totalling 2.95MiB  
  2020-11-04 16:53:37.656939: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 3104512 totalling 2.96MiB    
  2020-11-04 16:53:37.656946: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 3161088 totalling 3.01MiB  
  2020-11-04 16:53:37.656953: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 3555840 totalling 3.39MiB  
  2020-11-04 16:53:37.656960: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 3575808 totalling 3.41MiB  
  2020-11-04 16:53:37.656966: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 4012032 totalling 3.83MiB  
  2020-11-04 16:53:37.656973: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 18 Chunks of size 4110336 totalling 70.56MiB
  2020-11-04 16:53:37.656980: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 4111104 totalling 3.92MiB  
  2020-11-04 16:53:37.656987: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 4207872 totalling 4.01MiB  
  2020-11-04 16:53:37.656994: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 4236544 totalling 4.04MiB  
  2020-11-04 16:53:37.657001: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 4245504 totalling 4.05MiB  
  2020-11-04 16:53:37.657008: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 4354816 totalling 4.15MiB  
  2020-11-04 16:53:37.657015: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 809 Chunks of size 9437184 totalling 7.11GiB 
  2020-11-04 16:53:37.657023: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 143 Chunks of size 9830400 totalling 1.31GiB 
  2020-11-04 16:53:37.657030: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 10106624 totalling 9.64MiB   
  2020-11-04 16:53:37.657037: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 10291200 totalling 9.81MiB   
  2020-11-04 16:53:37.657044: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 10386432 totalling 9.91MiB   
  2020-11-04 16:53:37.657051: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 10658560 totalling 10.16MiB  
  2020-11-04 16:53:37.657059: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 10850048 totalling 10.35MiB  
  2020-11-04 16:53:37.657066: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 10895872 totalling 10.39MiB  
  2020-11-04 16:53:37.657073: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 10896384 totalling 10.39MiB
  2020-11-04 16:53:37.657080: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 11045120 totalling 10.53MiB                                                                         
  2020-11-04 16:53:37.657087: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 11188224 totalling 10.67MiB
  2020-11-04 16:53:37.657093: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 11581440 totalling 11.04MiB
  2020-11-04 16:53:37.657100: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 19 Chunks of size 11796480 totalling 213.75MiB
  2020-11-04 16:53:37.657108: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 12137472 totalling 11.58MiB
  2020-11-04 16:53:37.657115: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 12404736 totalling 11.83MiB
  2020-11-04 16:53:37.657122: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 12939264 totalling 12.34MiB
  2020-11-04 16:53:37.657129: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 2 Chunks of size 13547520 totalling 25.84MiB
  2020-11-04 16:53:37.657136: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 13621248 totalling 12.99MiB
  2020-11-04 16:53:37.657144: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 13785856 totalling 13.15MiB
  2020-11-04 16:53:37.657151: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 13865984 totalling 13.22MiB
  2020-11-04 16:53:37.657160: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 3 Chunks of size 14155776 totalling 40.50MiB
  2020-11-04 16:53:37.657164: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 14304768 totalling 13.64MiB                                                       
  2020-11-04 16:53:37.657168: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 14649600 totalling 13.97MiB                                                                         
  2020-11-04 16:53:37.657172: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 35 Chunks of size 15360000 totalling 512.70MiB
  2020-11-04 16:53:37.657176: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 3 Chunks of size 15906816 totalling 45.51MiB
  2020-11-04 16:53:37.657181: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 17 Chunks of size 16515072 totalling 267.75MiB
  2020-11-04 16:53:37.657186: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 16575488 totalling 15.81MiB
  2020-11-04 16:53:37.657194: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 17123328 totalling 16.33MiB
  2020-11-04 16:53:37.657198: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 17658112 totalling 16.84MiB
  2020-11-04 16:53:37.657202: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 18265088 totalling 17.42MiB
  2020-11-04 16:53:37.657206: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 18265856 totalling 17.42MiB
  2020-11-04 16:53:37.657211: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 31 Chunks of size 18266112 totalling 540.02MiB
  2020-11-04 16:53:37.657215: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 2 Chunks of size 18266368 totalling 34.84MiB
  2020-11-04 16:53:37.657219: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 18315776 totalling 17.47MiB
  2020-11-04 16:53:37.657223: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 18501888 totalling 17.64MiB
  2020-11-04 16:53:37.657229: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 18659328 totalling 17.79MiB
  2020-11-04 16:53:37.657234: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 18659584 totalling 17.79MiB
  2020-11-04 16:53:37.657240: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 24899584 totalling 23.75MiB
  2020-11-04 16:53:37.657245: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 44 Chunks of size 39321600 totalling 1.61GiB
  2020-11-04 16:53:37.657248: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 28 Chunks of size 93763584 totalling 2.44GiB
  2020-11-04 16:53:37.657252: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 95564544 totalling 91.14MiB
  2020-11-04 16:53:37.657256: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 99225600 totalling 94.63MiB
  2020-11-04 16:53:37.657260: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 102592512 totalling 97.84MiB
  2020-11-04 16:53:37.657264: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 103200768 totalling 98.42MiB
  2020-11-04 16:53:37.657268: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 122075136 totalling 116.42MiB
  2020-11-04 16:53:37.657273: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 143917056 totalling 137.25MiB
  2020-11-04 16:53:37.657278: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 173402880 totalling 165.37MiB
  2020-11-04 16:53:37.657283: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 174149120 totalling 166.08MiB
  2020-11-04 16:53:37.657287: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 184383744 totalling 175.84MiB
  2020-11-04 16:53:37.657291: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 184633344 totalling 176.08MiB
  2020-11-04 16:53:37.657295: I tensorflow/core/common_runtime/bfc_allocator.cc:1038] Sum Total of in-use chunks: 19.95GiB
  2020-11-04 16:53:37.657299: I tensorflow/core/common_runtime/bfc_allocator.cc:1040] total_region_allocated_bytes_: 21468487680 memory_limit_: 21468487808 available bytes: 128 curr_region_allocation_byt
  es_: 42936975872
  2020-11-04 16:53:37.657313: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] Stats:
  Limit:                     21468487808
  InUse:                     21422771968
  MaxInUse:                  21432602368
  NumAllocs:                    46783782MaxAllocSize:                187526912Reserved:                            0
  PeakReserved:                        0
  LargestFreeBlock:                    0
  
  2020-11-04 16:53:37.657683: W tensorflow/core/common_runtime/bfc_allocator.cc:439] ****************************************************************************************************
  2020-11-04 16:53:37.657727: W tensorflow/core/framework/op_kernel.cc:1767] OP_REQUIRES failed at matmul_op.cc:481 : Resource exhausted: OOM when allocating tensor with shape[3200,3072] and type float o
  n /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
  Traceback (most recent call last):
    File "<stdin>", line 4, in <module>
    File "myLib/python3.7/site-packages/kerastuner/engine/base_tuner.py", line 131, in search
      self.run_trial(trial, *fit_args, **fit_kwargs)
    File "myLib/python3.7/site-packages/kerastuner/engine/multi_execution_tuner.py", line 98, in run_trial
      history = model.fit(*fit_args, **copied_fit_kwargs)
    File "myLib/python3.7/site-packages/tensorflow/python/keras/engine/training.py", line 108, in _method_wrapper
      return method(self, *args, **kwargs)
    File "myLib/python3.7/site-packages/tensorflow/python/keras/engine/training.py", line 1098, in fit
      tmp_logs = train_function(iterator)
    File "myLib/python3.7/site-packages/tensorflow/python/eager/def_function.py", line 780, in __call__
      result = self._call(*args, **kwds)
    File "myLib/python3.7/site-packages/tensorflow/python/eager/def_function.py", line 840, in _call
      return self._stateless_fn(*args, **kwds)
    File "myLib/python3.7/site-packages/tensorflow/python/eager/function.py", line 2829, in __call__
      return graph_function._filtered_call(args, kwargs)  # pylint: disable=protected-access
    File "myLib/python3.7/site-packages/tensorflow/python/eager/function.py", line 1848, in _filtered_call
      cancellation_manager=cancellation_manager)
    File "myLib/python3.7/site-packages/tensorflow/python/eager/function.py", line 1924, in _call_flat
      ctx, args, cancellation_manager=cancellation_manager))
    File "myLib/python3.7/site-packages/tensorflow/python/eager/function.py", line 550, in call
      ctx=ctx)
    File "myLib/python3.7/site-packages/tensorflow/python/eager/execute.py", line 60, in quick_execute
      inputs, attrs, num_outputs)
  tensorflow.python.framework.errors_impl.ResourceExhaustedError: 2 root error(s) found.
    (0) Resource exhausted:  OOM when allocating tensor with shape[3200,3072] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc         [[node BERT_MultiLabel_MultiClass/tf_bert_model/bert/encoder/layer_._11/intermediate/dense/Tensordot/MatMul (defined at myLib/python3.7/site-packages/transformers/modeling_tf_bert.py:327) ]]
  Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
  
           [[clip_by_norm_1/truediv/_546]]
  Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
  
    (1) Resource exhausted:  OOM when allocating tensor with shape[3200,3072] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc         [[node BERT_MultiLabel_MultiClass/tf_bert_model/bert/encoder/layer_._11/intermediate/dense/Tensordot/MatMul (defined at myLib/python3.7/site-packages/transformers/modeling_tf_bert.py:327) ]]
  Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
  
  0 successful operations.
  0 derived errors ignored. [Op:__inference_train_function_1394299]
  
  Function call stack:
  train_function -> train_function
  ```