In this step, the goal is to identify which tokens in the generated text were influenced by the preceding context.
First, a context-aware generation is produced using the model's inputs augmented with available context. Then, the same generation is force-decoded using the contextless inputs. During both processes, a contrastive metric (KL-divergence is used as default for the Context sensitivity metric parameter) are collected for every generated token. Intuitively, higher metric scores indicate that the current generation step was more influenced by the presence of context.
The generated tokens are ranked according to their metric scores, and the most salient tokens are selected for the next step (This demo provides a Context sensitivity threshold parameter to select tokens above N standard deviations from the in-example metric average, and Context sensitivity top-k to pick the K most salient tokens.)
In the example shown in the figure, elle is selected as the only context-sensitive token by the procedure.
Once context-sensitive tokens are identified, the next step is to link every one of these tokens to specific contextual cues that justified its prediction.
This is achieved by means of contrastive feature attribution (Yin and Neubig, 2022). More specifically, for a given context-sensitive token, a contrastive alternative to it is generated in absence of input context, and a function of the probabilities of the pair is used to identify salient parts of the context (By default, in this demo we use saliency, i.e. raw gradients, for the Attribution method and contrast_prob_diff, i.e. the probability difference between the two options, for the Attributed function).
Gradients are collected and aggregated to obtain a single score per context token, which is then used to rank the tokens and select the most influential ones (This demo provides a Attribution threshold parameter to select tokens above N standard deviations from the in-example metric average, and Attribution top-k to pick the K most salient tokens.)
In the example shown in the figure, the attribution process links elle to dishes and assiettes in the source and target contexts, respectively. This makes sense intuitively, as they in the original input is gender-neutral in English, and the presence of its gendered coreferent disambiguates the choice for the French pronoun in the translation.
This demo provides a convenient UI for the Inseq implementation of PECoRe (the inseq attribute-context CLI command).
In the demo tab, fill in the input and context fields with the text you want to analyze, and click the Run PECoRe button to produce an output where the tokens selected by PECoRe in the model generation and context are highlighted. For more details on the parameters and their meaning, check the Parameters tab.
The example shows the output of the CORA Multilingual QA model used as default in the interface, using default settings.
""" citation = r"""
To refer to the PECoRe framework for context usage detection, cite:
@inproceedings{sarti-etal-2023-quantifying,
title = "Quantifying the Plausibility of Context Reliance in Neural Machine Translation",
author = "Sarti, Gabriele and
Chrupa{\l}a, Grzegorz and
Nissim, Malvina and
Bisazza, Arianna",
booktitle = "The Twelfth International Conference on Learning Representations (ICLR 2024)",
month = may,
year = "2024",
address = "Vienna, Austria",
publisher = "OpenReview",
url = "https://openreview.net/forum?id=XTHfNGI3zT"
}
inseq attribute-context, including this demo), please also cite:
@inproceedings{sarti-etal-2023-inseq,
title = "Inseq: An Interpretability Toolkit for Sequence Generation Models",
author = "Sarti, Gabriele and
Feldhus, Nils and
Sickert, Ludwig and
van der Wal, Oskar and
Nissim, Malvina and
Bisazza, Arianna",
booktitle = "Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations)",
month = jul,
year = "2023",
address = "Toronto, Canada",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2023.acl-demo.40",
pages = "421--435",
}
