text: 
  
  TITLE
  Add support for bitsandbytes
  
  COMMENTS
  10
  
  REACTIONS
  +1: 0
  -1: 0
  laugh: 0
  hooray: 0
  heart: 3
  rocket: 0
  eyes: 0
  
  BODY
  # What does this PR do?
  
  Fixes #14819
  
  
  ## Before submitting
  - [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
  - [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
        Pull Request section?
  - [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
        to it if that's the case.
  - [ ] Did you make sure to update the documentation with your changes? Here are the
        [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
        [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
  - [x] Did you write any new necessary tests?
  
  
  ## Who can review?
  
  Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
  members/contributors who may be interested in your PR.
  
  @stas00 @sgugger @TimDettmers
  
  ## Status
  
  * [ ] Need to instrument CI to install bnb (binary package so a bit trickier than normal dependency)
  * [x] I have implemented a CLI parameter to support `bitsandbytes`
  * [ ] I did not write any documentation yet
  * [ ] I followed @TimDettmers 's [suggestion](https://github.com/huggingface/transformers/issues/14819#issuecomment-1016017746) to override the embedding layers. However, I am unsure about a couple of things:
      * Does the override need to happen before the model is loaded onto the GPU as the official documentation [describes for other overrides](https://github.com/facebookresearch/bitsandbytes/blob/main/howto_config_override.md)?
      * Are there any pitfalls to my current approach to identifying `Embedding` layers? It seems to work fine for `RoBERTa` and for `GPT-2`.
  * [x] So far, I've used `run_mlm.py` and `run_clm.py` from the examples directory to check that the code runs. Using RTX A6000 GPUs, I see
  
  | Model           | visible devices | optimizer         | per device batch size | GPU memory     |
  | :---                |    :----             |  :----               | :----:         |           ---: |
  | gpt2-large    | 0                       | adamw_torch   | 2                |  48638MiB / 49140MiB |
  | gpt2-large    | 0                       | adamw_bnb     | 2                |  42412MiB / 49140MiB |
  | gpt2-large    | 0,1                     | adamw_torch   | 1                |  30724MiB / 49140MiB <br> 21040MiB / 49140MiB |
  | gpt2-large    | 0,1                     | adamw_torch   | 2                |  OOM |
  | gpt2-large    | 0,1                     | adamw_bnb     | 1                | 26820MiB / 49140MiB <br> 21042MiB / 49140MiB   |
  | gpt2-large    | 0,1                     | adamw_bnb     | 2                | 44458MiB / 49140MiB <br> 36906MiB / 49140MiB  |
  
  