---
license: apache-2.0
base_model: google/t5-efficient-mini-nl24
datasets:
- Open-Orca/FLAN
language:
- en
library_name: transformers
pipeline_tag: text2text-generation
---


# t5e-mini-nl24-flan


25k steps on FLAN as an initial test/validation that [code](https://github.com/pszemraj/nanoT5/tree/flan-dataset) works. Not practically useful.

```py
from transformers import pipeline

pipe = pipeline(
    "text2text-generation",
    model="pszemraj/t5e-mini-nl24-flan",
)
res = pipe(
    "true or false: water is wet.",
    top_k=4,
    penalty_alpha=0.6,
    max_new_tokens=128,
)
print(res[0]["generated_text"])
```

## Quick eval

Quick eval for:	`pszemraj/t5e-mini-nl24-flan`


hf (pretrained=pszemraj/t5e-mini-nl24-flan,trust_remote_code=True,dtype=bfloat16,trust_remote_code=True), gen_kwargs: (None), limit: None, num_fewshot: None, batch_size: 8
|    Tasks    |Version|     Filter     |n-shot|  Metric   |   |Value |   |Stderr|
|-------------|------:|----------------|-----:|-----------|---|-----:|---|------|
|boolq        |      2|none            |     0|acc        |↑  |0.4541|±  |0.0087|
|openbookqa   |      1|none            |     0|acc        |↑  |0.1300|±  |0.0151|
|             |       |none            |     0|acc_norm   |↑  |0.2700|±  |0.0199|
|piqa         |      1|none            |     0|acc        |↑  |0.6159|±  |0.0113|
|             |       |none            |     0|acc_norm   |↑  |0.6077|±  |0.0114|
|social_iqa   |      0|none            |     0|acc        |↑  |0.3705|±  |0.0109|
|tinyArc      |      0|none            |    25|acc_norm   |↑  |0.2913|±  |   N/A|
|tinyGSM8k    |      0|flexible-extract|     5|exact_match|↑  |0.0269|±  |   N/A|
|             |       |strict-match    |     5|exact_match|↑  |0.0055|±  |   N/A|
|tinyHellaswag|      0|none            |    10|acc_norm   |↑  |0.3538|±  |   N/A|
|tinyMMLU     |      0|none            |     0|acc_norm   |↑  |0.2551|±  |   N/A|
|winogrande   |      1|none            |     0|acc        |↑  |0.5217|±  |0.0140|


<details>
  <summary>base model evals: click to expand</summary>
  
Quick eval for:	`google/t5-efficient-mini-nl24`


hf (pretrained=google/t5-efficient-mini-nl24,trust_remote_code=True,dtype=bfloat16,trust_remote_code=True), gen_kwargs: (None), limit: None, num_fewshot: None, batch_size: 8
|    Tasks    |Version|     Filter     |n-shot|  Metric   |   |Value |   |Stderr|
|-------------|------:|----------------|-----:|-----------|---|-----:|---|------|
|boolq        |      2|none            |     0|acc        |↑  |0.3783|±  |0.0085|
|openbookqa   |      1|none            |     0|acc        |↑  |0.1280|±  |0.0150|
|             |       |none            |     0|acc_norm   |↑  |0.2660|±  |0.0198|
|piqa         |      1|none            |     0|acc        |↑  |0.5473|±  |0.0116|
|             |       |none            |     0|acc_norm   |↑  |0.5267|±  |0.0116|
|social_iqa   |      0|none            |     0|acc        |↑  |0.3536|±  |0.0108|
|tinyArc      |      0|none            |    25|acc_norm   |↑  |0.3101|±  |   N/A|
|tinyGSM8k    |      0|flexible-extract|     5|exact_match|↑  |0.0145|±  |   N/A|
|             |       |strict-match    |     5|exact_match|↑  |0.0055|±  |   N/A|
|tinyHellaswag|      0|none            |    10|acc_norm   |↑  |0.2616|±  |   N/A|
|tinyMMLU     |      0|none            |     0|acc_norm   |↑  |0.2839|±  |   N/A|
|winogrande   |      1|none            |     0|acc        |↑  |0.4996|±  |0.0141|

</details>