metadata
language:
- en
license: mit
datasets:
- Salesforce/wikitext
This is a custom implementation of gpt2, where we replace attention with our implementation. Currently, we don't replace softmax, but in future submits we would like to replace the softmax function in attention with other softmax variations.
We directly use the huggingface gpt2 model: https://huggingface.co/openai-community/gpt2
This model was finetuned on the wikitext dataset: https://paperswithcode.com/dataset/wikitext-2
base model: huggingface gpt2