

TITLE
New Model: Charformer: Fast Character Transformers via Gradient-based Subword Tokenization 

COMMENTS
3

REACTIONS
+1: 11
-1: 0
laugh: 0
hooray: 0
heart: 0
rocket: 0
eyes: 0

BODY
# 🌟 New model addition

## Model description

arXiv = https://arxiv.org/pdf/2106.12672.pdf (pre-print; under review)

In this paper, they introduce a soft gradient-based subword tokenization module (GBST) that automatically learns latent subword representations from characters in a data-driven fashion. More importantly, is the introduction of Charformer, a deep Transformer model that integrates GBST and operates on the byte level. 

> Via extensive experiments on English GLUE, multilingual, and noisy text datasets, we show that Charformer outperforms a series of competitive byte-level baselines while generally performing on par and sometimes outperforming subword-based models. Additionally, Charformer is fast, improving the speed of both vanilla byte-level and subword-level Transformers by 28%-100% while maintaining competitive quality. We believe this work paves the way for highly performant token-free models that are trained completely end-to-end.

## Open source status

* [ ] the model implementation is available: ----] [Implementation and weights](https://github.com/google-research/google-research/charformer)
* [ ] the model weights are available: ------------] [to be released soon here](https://github.com/google-research/google-research/charformer)
* [x] who are the authors: Yi Tay, Vinh Q. Tran, Sebastian Ruder*, Jai Gupta, Hyung Won Chung, Dara BahriZhen Qin, Simon Baumgartner, Cong Yu, Donald Metzler (Google and DeepMind-->`*`)
