codellama/CodeLlama-7b-hf · Model trying to allocate 200.00 GiB

I was trying to get the hidden states from an input batch of 100, each with 128 tokens. My code worked with other models. Could someone give me a hint on why this is happening?
OutOfMemoryError: CUDA out of memory. Tried to allocate 200.00 GiB. GPU 0 has a total capacty of 39.56 GiB of which 30.36 GiB is free. Process 22149 has 9.20 GiB memory in use. Of the allocated memory 8.72 GiB is allocated by PyTorch, and 3.84 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF