

TITLE
Simplify GPT-Neo local attention implementation

COMMENTS
10

REACTIONS
+1: 0
-1: 0
laugh: 0
hooray: 0
heart: 0
rocket: 0
eyes: 1

BODY
# What does this PR do?

The current local self attention implementation for gpt-neo follows the implementation in mtf, which makes it hard to understand. This pull request implements local self attention by changing the bias mask in global attention. Local self attention is a sliding window where each token can only attend to the previous window_size tokens. This implementation clearly reflects this.

[Measured performance](https://gist.github.com/finetuneanon/5b2186c3555b652f387c86160cd89b55) (apply just 330686a3c0520c9727fe5ebed385e708d0178d77 and 269c497be1691556c830f61fa8f90001c692722f on #11320 patched to use) shows no noticable difference between implementations with respect to speed or VRAM usage. The results of both implementations are also identical.

Fixes #11320

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

Since this PR mostly removes code, no additional tests or documentation were written.

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

gpt-neo: @patil-suraj
