

TITLE
Sequential constraints?

COMMENTS
4

REACTIONS
+1: 3
-1: 0
laugh: 0
hooray: 0
heart: 0
rocket: 0
eyes: 0

BODY
fairseq has an implementation of unordered and ordered multi-token constraints: see their [PR](https://github.com/pytorch/fairseq/pull/2402) and [example](https://github.com/pytorch/fairseq/tree/1bba712622b8ae4efb3eb793a8a40da386fe11d0/examples/constrained_decoding).

This is more advanced than the single-token constraints that have been occasionally [requested here](https://github.com/huggingface/transformers/issues/10485) mainly due to the bookkeeping involved; see the papers referenced in the fairseq PR.

Has anyone looked into porting this feature? fairseq's [constraint tracking logic](https://github.com/pytorch/fairseq/blob/master/fairseq/token_generation_constraints.py) looks to be well-factored and could probably be adopted verbatim, license permitting. The beam search modifications ([fairseq implementation](https://github.com/pytorch/fairseq/blob/ee0d5a0f65a25e5f5372776402aac5cb9c4adbf1/fairseq/search.py#L210)) may be able to be implemented as a `LogitsProcessor`, or maybe even just a `prefix_allowed_tokens_fn`, but the papers propose some additional logic around making sure that a partial constraint stays in the beam; I'm not sure whether those hooks are sufficient to implement that logic (or how essential it is to the functionality).

(I've found one [related issue](https://github.com/huggingface/transformers/issues/1163).)