initial commit

README.md

@@ -1,3 +1,92 @@
 ---
 license: mit
+language: en
 ---
+
+# T5(v1.1)-SLED (SLiding-Encoder and Decoder, base-sized model) 
+
+SLED models use pretrained, short-range encoder-decoder models, and apply them over 
+long-text inputs by splitting the input into multiple overlapping chunks, encoding each independently and perform fusion-in-decoder
+
+## Model description
+
+This SLED model is based on the T5(V1.1) model, which is described in its [model card](https://huggingface.co/google/t5-v1_1-large).
+
+The developers write in a [blog post](https://ai.googleblog.com/2020/02/exploring-transfer-learning-with-t5.html) that the T5 model:
+> Our text-to-text framework allows us to use the same model, loss function, and hyperparameters on any NLP task, including machine translation, document summarization, question answering, and classification tasks (e.g., sentiment analysis). We can even apply T5 to regression tasks by training it to predict the string representation of a number instead of the number itself.
+T5 v1.1 includes several improvments on top of the original checkpoint. see its card for details
+
+## Intended uses & limitations
+
+You can use the raw model for text infilling. However, the model is mostly meant to be fine-tuned on a supervised dataset. 
+
+### How to use
+To use the model, you first need to install `py-sled` in your environment (or clone the code from the [official repository](https://github.com/Mivg/SLED/blob/main/README.md))
+```
+pip install py-sled
+```
+For more installation instructions, see [here](https://github.com/Mivg/SLED#Installation).
+
+Once installed, SLED is fully compatible with HuggingFace's AutoClasses (AutoTokenizer, AutoConfig, AutoModel 
+and AutoModelForCausalLM) and can be loaded using the from_pretrained methods
+```python
+import sled   # *** required so that SledModels will be registered for the AutoClasses ***
+model = AutoModel.from_pretrained('tau/t5-v1_1-large-sled')
+```
+
+Here is how to use this model in PyTorch:
+
+```python
+from sled import SledTokenizer, SledModel
+tokenizer = SledTokenizer.from_pretrained('tau/t5-v1_1-large-sled')
+model = SledModel.from_pretrained('tau/t5-v1_1-large-sled')
+inputs = tokenizer("Hello, my dog is cute", return_tensors="pt")
+outputs = model(**inputs)
+last_hidden_states = outputs.last_hidden_state
+```
+You can also replace SledModel by SledModelForConditionalGeneration for Seq2Seq generation 
+```python
+model = SledModelForConditionalGeneration.from_pretrained('tau/t5-v1_1-large-sled')
+```
+
+In case you wish to apply SLED on a task containing a prefix (e.g. question) which should be given as a context to 
+every chunk, you can pass the `prefix_length` tensor input as well (A LongTensor in the length of the batch size).
+```python
+import torch
+import sled   # *** required so that SledModels will be registered for the AutoClasses ***
+tokenizer = AutoTokenizer.from_pretrained('tau/t5-v1_1-large-sled')
+model = AutoModel.from_pretrained('tau/t5-v1_1-large-sled')
+document_input_ids = tokenizer("Dogs are great for you.", return_tensors="pt").input_ids
+prefix_input_ids = tokenizer("Are dogs good for you?", return_tensors="pt").input_ids
+input_ids = torch.cat((prefix_input_ids, document_input_ids), dim=-1)
+attention_mask = torch.ones_like(input_ids)
+prefix_length = torch.LongTensor([[prefix_input_ids.size(1)]])
+
+outputs = model(input_ids=input_ids, attention_mask=attention_mask, prefix_length=prefix_length)
+last_hidden_states = outputs.last_hidden_state
+```
+
+### BibTeX entry and citation info
+
+Please cite both the SLED [paper](https://arxiv.org/abs/2208.00748.pdf) and the T5 [paper](https://arxiv.org/pdf/1910.10683.pdf) by Raffel et al
+
+```bibtex
+@inproceedings{Ivgi2022EfficientLU,
+  title={Efficient Long-Text Understanding with Short-Text Models},
+  author={Maor Ivgi and Uri Shaham and Jonathan Berant},
+  year={2022}
+}
+```
+
+```bibtex
+@article{2020t5,
+  author  = {Colin Raffel and Noam Shazeer and Adam Roberts and Katherine Lee and Sharan Narang and Michael Matena and Yanqi Zhou and Wei Li and Peter J. Liu},
+  title   = {Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer},
+  journal = {Journal of Machine Learning Research},
+  year    = {2020},
+  volume  = {21},
+  number  = {140},
+  pages   = {1-67},
+  url     = {http://jmlr.org/papers/v21/20-074.html}
+}
+```

config.json

@@ -0,0 +1,9 @@
+{
+  "model_type": "tau/sled",
+  "underlying_config": "google/t5-v1_1-large",
+  "context_size": 256,
+  "window_fraction": 0.5,
+  "prepend_prefix": true,
+  "encode_prefix": true,
+  "sliding_method": "dynamic"
+}

tokenizer_config.json

@@ -0,0 +1,5 @@
+{
+  "tokenizer_class": "SledTokenizer",
+  "base_tokenizer": "google/t5-v1_1-large",
+  "model_max_length": 16384
+}