

TITLE
LayoutLMv2 processing doesn't handle tokenizer overflow

COMMENTS
9

REACTIONS
+1: 0
-1: 0
laugh: 0
hooray: 0
heart: 0
rocket: 0
eyes: 0

BODY
## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.10.2
- Platform:  MAC
- Python version: 3.8.9
- PyTorch version (GPU?): Not important
- Tensorflow version (GPU?):
- Using GPU in script? Mp
- Using distributed or parallel set-up in script?:

### Who can help
@NielsRogge @LysandreJik 

## Information

We are porting our layoutlmv2 project to use Transformers instead of the UniLMFt package.

The additional functionality of the tokenizer has helped us to eliminate a good deal of alignment code!! 

While evaluating `processing_layoutlmv2.py` I noticed that overflow wasn't being handled properly.

https://github.com/huggingface/transformers/blob/3ab0185b061baae207efed02799dd424ee8377f1/src/transformers/models/layoutlmv2/processing_layoutlmv2.py#L182-L205

In the above block the input is tokeninzed, potentially with allowing overflow when `return_overflowing_tokens==True` this will cause the length of encoded_inputs to be longer than the input sequence. EG if a page has 1k words and boxes it will be returned as two sequences and there will be an `overflow_to_sample_mapping` attached to the encoded_inputs. 

When adding the image

```
        encoded_inputs["image"] = features.pop("pixel_values")
```

The length of `image` will be less than the rest of the encoded inputs if there is any overflow. This will cause: 1) a mismatch between page images and examples, and 2) examples at the end of the sequence will lack image embeddings.


## Expected behavior

We handle this by using the `overflow_to_sample_mapping` to find which image to pair with each sequence in the batch:

```
    images = []
    for batch_index in range(len(tokenized_inputs["input_ids"])):
        org_batch_index = tokenized_inputs["overflow_to_sample_mapping"][batch_index]
        image = examples["image"][org_batch_index]
        images.append(image)
    tokenized_inputs['image] = images
```

`return_offsets_mapping=True` is required for this to work, but you could consider raising if `return_overflowing_tokens` is True and `return_offset_mapping` is False to maintain the ability to pair images with the correct sequences.

