

TITLE
Overflow when normalizing start and end logits in QA pipeline

COMMENTS
3

REACTIONS
+1: 0
-1: 0
laugh: 0
hooray: 0
heart: 1
rocket: 0
eyes: 0

BODY
### System Info

```shell
- `transformers` version: 4.18.0
- Platform: Linux-4.19.0-20-cloud-amd64-x86_64-with-debian-10.12
- Python version: 3.7.12
- Huggingface_hub version: 0.4.0
- PyTorch version (GPU?): 1.11.0+cu102 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>
```


### Who can help?

@Narsil @sgugger 

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

After succesfully training a question answering model using the [run_qa.py ](https://github.com/huggingface/transformers/blob/215e0681e4c3f6ade6e219d022a5e640b42fcb76/examples/pytorch/question-answering/run_qa.py) script, I tried to reuse it in for inference in a question-answering `pipeline` but the model predicts `no-answer` for all the (question, context) pairs even for those included in my evaluation file which was used when running the `run_qa.py` script. The evaluation part of `run_qa.py` was able to predict an answer for those examples. 

The issue is related to the normalization of start and end logits done in the QA pipeline https://github.com/huggingface/transformers/blob/215e0681e4c3f6ade6e219d022a5e640b42fcb76/src/transformers/pipelines/question_answering.py#L401 

`start_` and `end_` are provided `np.float32` arrays and all the logits calculated by the model were <-100  which leads to overflow when applying `exp` then `log`. The normalization differs from the one applied in `run_qa.py` file which explains this overflow error happening only while doing inference. 

https://github.com/huggingface/transformers/blob/215e0681e4c3f6ade6e219d022a5e640b42fcb76/examples/flax/question-answering/utils_qa.py#L191 


The trace showing this behavior is provided below:

``` 
> /opt/conda/lib/python3.7/site-packages/transformers/pipelines/question_answering.py(401)postprocess()
    400             # Normalize logits and spans to retrieve the answer
--> 401             start_ = np.exp(start_ - np.log(np.sum(np.exp(start_), axis=-1, keepdims=True)))
    402             end_ = np.exp(end_ - np.log(np.sum(np.exp(end_), axis=-1, keepdims=True)))

ipdb> start_.dtype
dtype('float32')

ipdb> start_.shape
(1, 512)
ipdb> start_[0, :20]
array([-10000.     , -10000.     , -10000.     , -10000.     ,
       -10000.     , -10000.     , -10000.     , -10000.     ,
       -10000.     , -10000.     , -10000.     , -10000.     ,
       -10000.     , -10000.     , -10000.     , -10000.     ,
       -10000.     ,   -155.77672,   -155.8097 ,   -155.81848],
      dtype=float32)
ipdb> n #running operation line 401

  start_ = np.exp(start_ - np.log(np.sum(np.exp(start_), axis=-1, keepdims=True)))
> /opt/conda/lib/python3.7/site-packages/transformers/pipelines/question_answering.py(402)postprocess()
    401             start_ = np.exp(start_ - np.log(np.sum(np.exp(start_), axis=-1, keepdims=True)))
--> 402             end_ = np.exp(end_ - np.log(np.sum(np.exp(end_), axis=-1, keepdims=True)))
    403 

ipdb> start_[0, :20]
array([inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf,
       inf, inf, inf, inf, inf, inf, inf

```

### Expected behavior

```shell
The LogSumExp trick should be applied similarly to https://github.com/huggingface/transformers/blob/215e0681e4c3f6ade6e219d022a5e640b42fcb76/examples/flax/question-answering/utils_qa.py#L191 and https://datumorphism.leima.is/cards/machine-learning/neural-networks/log-sum-exp-trick/ to prevent this overflow.
```
