|
--- |
|
license: apache-2.0 |
|
library_name: transformers |
|
--- |
|
|
|
i wanted to learn more about exposure bias mitigation in language models and came across [ReMask](https://huggingface.co/euclaise/ReMask-3B). |
|
it's a neat idea, and i wanted to give it a go. |
|
|
|
- during training, the model processes input sequences twice - once with the full sequence & once with masked sequence. |
|
- computes model outputs for both. |
|
- divergence loss is computed as the average of forward and backward KL divergences. |
|
- final loss is a weighted sum of the cross entropy losses and the divergence loss. |
|
|
|
impl on github |
|
|
|
``` |
|
<|user|> |
|
Could Moulin Rouge have been hypothetically used as Spain's Spanish American War triage center? |
|
<|logic|> |
|
The Moulin Rouge cabaret in France had a capacity of 850 people. Spain had 700-800 injured during Spanish American War. |
|
<|answer|> |
|
``` |
