

TITLE
Return type of ViTFeatureExtractor does not match 'return_tensors' parameter when input is torch.Tensor or PIL.Image.Image

COMMENTS
3

REACTIONS
+1: 0
-1: 0
laugh: 0
hooray: 0
heart: 0
rocket: 0
eyes: 0

BODY
## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.15.0
- Platform: Ubuntu 16.04
- Python version: 3.9.7
- PyTorch version (GPU?): 1.10.1+cu113
- Tensorflow version (GPU?): not installed
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- ALBERT, BERT, XLM, DeBERTa, DeBERTa-v2, ELECTRA, MobileBert, SqueezeBert: @LysandreJik
- T5, BART, Marian, Pegasus, EncoderDecoder: @patrickvonplaten
- Blenderbot, MBART: @patil-suraj
- Longformer, Reformer, TransfoXL, XLNet, FNet, BigBird: @patrickvonplaten
- FSMT: @stas00
- Funnel: @sgugger
- GPT-2, GPT: @patrickvonplaten, @LysandreJik
- RAG, DPR: @patrickvonplaten, @lhoestq
- TensorFlow: @Rocketknight1
- JAX/Flax: @patil-suraj
- TAPAS, LayoutLM, LayoutLMv2, LUKE, ViT, BEiT, DEiT, DETR, CANINE: @NielsRogge
- GPT-Neo, GPT-J, CLIP: @patil-suraj
- Wav2Vec2, HuBERT, SpeechEncoderDecoder, UniSpeech, UniSpeechSAT, SEW, SEW-D, Speech2Text: @patrickvonplaten, @anton-l

If the model isn't in the list, ping @LysandreJik who will redirect you to the correct contributor.

Library:

- Benchmarks: @patrickvonplaten
- Deepspeed: @stas00
- Ray/raytune: @richardliaw, @amogkam
- Text generation: @patrickvonplaten @narsil
- Tokenizers: @LysandreJik
- Trainer: @sgugger
- Pipelines: @Narsil
- Speech: @patrickvonplaten, @anton-l
- Vision: @NielsRogge, @sgugger

Documentation: @sgugger

Model hub:

- for issues with a model, report at https://discuss.huggingface.co/ and tag the model's creator.

HF projects:

- datasets: [different repo](https://github.com/huggingface/datasets)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj

For research projetcs, please ping the contributor directly. For example, on the following projects:

- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->

## Information

Model I am using (Bert, XLNet ...): ViT

The problem arises when using:
* [x] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)
Classification on ImageNet

## To reproduce
**Note**: Since `typing.List` is deprecated since python 3.9, I am using `builtins.list` in the following contents.

Steps to reproduce the behavior:

1. Set `do_normalize` and `do_resize` parameter of a `ViTFeatureExtractor`
2. Try different combinations
3. We call like this
```python
from transformers import ViTFeatureExtractor

extractor = ViTFeatureExtractor.from_pretrained("google/vit-base-patch16-224")
img = torch.randn(3, 256, 256)
input = extractor(img, return_tensors="pt")
# or
input = extractor([img, img], return_tensors="pt")
```
As the `__call__` function of ViTFeatureExtractor accepts `(PIL.Image.Image, np.ndarray, torch.Tensor, list[PIL.Image.Image], list[np.ndarray], list[torch.Tensor])` as its first parameter, it does't matter whether to call `extractor(img)` or `extractor([img])`. I also tried this:
```jupyter
>>> ViTFeatureExtractor()
ViTFeatureExtractor {
  "do_normalize": true,
  "do_resize": true,
  "feature_extractor_type": "ViTFeatureExtractor",
  "image_mean": [
    0.5,
    0.5,
    0.5
  ],
  "image_std": [
    0.5,
    0.5,
    0.5
  ],
  "resample": 2,
  "size": 224
}

>>> ViTFeatureExtractor.from_pretrained("google/vit-base-patch16-224")
 ViTFeatureExtractor {
   "do_normalize": true,
   "do_resize": true,
   "feature_extractor_type": "ViTFeatureExtractor",
   "image_mean": [
     0.5,
     0.5,
     0.5
   ],
   "image_std": [
     0.5,
     0.5,
     0.5
   ],
   "resample": 2,
   "size": 224
 }
```
which indicates the two extractors are exactly same. And the results are very weird, shown in table below:
|  `do_resize`  |  `do_normalize`  |  `return_tensors`  |  actual return type  |
|  :----------:  |  :---------------:  |  :----------------:  |  :-----------:  |
|  ✅  |  ✅  |  not specified  |  `list[np.ndarray]`  |
|  ✅  |  ✅  |  "pt"  |  4-D `torch.Tensor` <br> with shape `(B, C, H, W)`  | 
|  ❌  |  ✅  |   not specified |  `list[torch.Tensor]`  |
|  ❌  |  ✅  |  "pt"  |  ValueError  |
|  ✅  |  ❌  |   not specified  |  `list[PIL.Image.Image]`  |
|  ✅  |  ❌  |  "pt"  |  ValueError  |
|  ❌  |  ❌  |   not specified  |  `list[torch.Tensor]`  |
|  ❌  |  ❌  |  "pt"  |  ValueError  |

When return type is `list[torch.Tensor]`, each element in the list is a 3-D `torch.Tensor` with shape `(C, H, W)`. The `ValueError` in table refers to
>ValueError: Unable to create tensor, you should probably activate padding with 'padding=True' to have batched tensors with the same length.

I cannot understand why `__call__` returns a list of `PIL.Image.Image` when `do_normalize = False` and `do_resize = True`, which is the most weird thing. It seems that tensors are only converted to PIL images, resized but no more operations.
According to the doc, the default value of `return_tensors` is `"np"` when not specified. But it does not correspond to the real type of return value when `do_normalize` or `do_resize` is changed. The doc also says
>NumPy arrays and PyTorch tensors are converted to PIL images when resizing, so the most efficient is to pass PIL images.

As we can do resize and normalization using `torchvision.transforms`, there are 3 solutions:
### Solution 1
Do feature extract before using `torch.utils.data.DataLoader` (pass PIL images to `__call__`).
```python
from torchvision.datasets import ImageFolder

dataset = ImageFolder(root)
extractor = ViTFeatureExtractor.from_pretrained("google/vit-base-patch16-224")
extractor.do_normalize = True
extractor.do_resize = True
# X is list of PIL.Image.Image
X = [x for x, _ in dataset]  # This will consume all of your memory
X = extractor(X, return_tensors="pt")["pixel_values"]
# or
for x, _ in dataset:
    x = extractor(x, return_tensors="pt")["pixel_values"].squeeze()  # This is very slow and inefficient
```
### Solution 2
Do feature extract after `torch.utils.data.DataLoader` and `torchvision.datasets.ImageFolder` in a small batch (pass PIL images to `__call__`).
```python
from torchvision.datasets import ImageFolder
from torch.utils.data import DataLoader

dataset = ImageFolder(root)
extractor = ViTFeatureExtractor.from_pretrained("google/vit-base-patch16-224")
extractor.do_normalize = True
extractor.do_resize = True
loader = DataLoader(dataset, batch_size=64)
X, y = next(iter(loader))
X = extractor(X, return_tensors="pt")["pixel_values"]
```
This will raise an error:
>TypeError: default_collate: batch must contain tensors, numpy arrays, numbers, dicts or lists; found <class 'PIL.Image.Image'>

### Solution 3
#### Warning
It is not recommended to use `ViTFeatureExtractor` along with `torchvision.transforms`. Inappropriate combinations can lead to a decrease in accuracy. Generally, never do resize after normalization. Examples:
* Use `torchvision.transforms` only
```python
import torch
from torchvision.datasets import ImageFolder
from torch.utils.data import DataLoader
from torchvision import transforms
from transformers.image_utils import (
    IMAGENET_STANDARD_MEAN,
    IMAGENET_STANDARD_STD
)
from transformers import ViTForImageClassification

img_size = 224

normalize = transforms.Normalize(mean=IMAGENET_STANDARD_MEAN, std=IMAGENET_STANDARD_STD)

tf = transforms.Compose([
    transforms.Resize((img_size, img_size)),
    transforms.ToTensor(), normalize])

dataset = ImageFolder(root=root, transform=tf)
loader = DataLoader(dataset, batch_size=64)
model = ViTForImageClassification.from_pretrained("google/vit-base-patch16-224")
outputs = model(dataset[0][0].unsqueeze(0))
# or
outputs = model(next(iter(loader))[0])
```
* Use `ViTFeatureExtractor` only (not recommended, images are converted to tensors then to PIL images again)
```python
import torch
from torchvision.datasets import ImageFolder
from torch.utils.data import DataLoader
from torchvision import transforms
from transformers import ViTFeatureExtractor, ViTForImageClassification

tf = transforms.ToTensor()

dataset = ImageFolder(root=root, transform=tf)
loader = DataLoader(dataset, batch_size=64)
extractor = ViTFeatureExtractor.from_pretrained("google/vit-base-patch16-224")
model = ViTForImageClassification.from_pretrained("google/vit-base-patch16-224")
inputs = extractor(images=dataset[0][0], return_tensors="pt")
outputs = model(**inputs)
```
`DataLoader` cannot be used here as it requires that each tensor in the mini-batch has the same shape.
* Do resize using `torchvision.transforms`, do normalization using `ViTFeatureExtractor`
```python
import torch
from torchvision.datasets import ImageFolder
from torch.utils.data import DataLoader
from torchvision import transforms
from transformers import ViTFeatureExtractor, ViTForImageClassification

img_size = 224

tf = transforms.Compose([
    transforms.Resize((img_size, img_size)),
    transforms.ToTensor()])

dataset = ImageFolder(root=root, transform=tf)
loader = DataLoader(dataset, batch_size=64)
extractor = ViTFeatureExtractor.from_pretrained("google/vit-base-patch16-224")
model = ViTForImageClassification.from_pretrained("google/vit-base-patch16-224")
extractor.do_resize = False
inputs = extractor(images=dataset[0][0])
outputs = model(torch.stack(inputs["pixel_values"]))
# or
images = list(next(iter(loader))[0].unbind())
inputs = extractor(images=images)
outputs = model(torch.stack(inputs["pixel_values"]))
```
And I proposed a flexible workaround for those who want to use `ViTModel` or `ViTForImageClassification` with `torch.utils.data.DataLoader`.
```python
import torch
from torch import Tensor
from torchvision.datasets import ImageFolder
from torch.utils.data import DataLoader
from torchvision import transforms
from transformers.image_utils import (
    IMAGENET_STANDARD_MEAN,
    IMAGENET_STANDARD_STD
)

img_size = 224

normalize = transforms.Normalize(mean=IMAGENET_STANDARD_MEAN, std=IMAGENET_STANDARD_STD)

tf = transforms.Compose([
    transforms.Resize((img_size, img_size)),
    transforms.RandomHorizontalFlip(),
    transforms.ToTensor(), normalize])

dataset = ImageFolder(root=root, transform=tf)
loader = DataLoader(dataset, batch_size=64)

extractor = ViTFeatureExtractor.from_pretrained("google/vit-base-patch16-224")

# X must be a 4-D tensor with shape (B, C, H, W)
def feature_extract(X: Tensor, extractor: ViTFeatureExtractor, do_resize=False, do_normalize=False) -> Tensor:
    X = list(X.unbind())
    extractor.do_resize = do_resize
    extractor.do_normalize = do_normalize
    if do_resize:
        if do_normalize:
            batch_feature = extractor(images=X, return_tensors="pt")
            return batch_feature["pixel_values"]
        else:
            batch_feature = extractor(images=X)
            imgs = [transforms.ToTensor()(img) for img in batch_feature["pixel_values"]]
            return torch.stack(imgs)
    else:
        batch_feature = extractor(images=X)
        return torch.stack(batch_feature["pixel_values"])
```
Usage:
```python
from torch import nn

def model_fn(batch: list[Tensor], extractor: ViTFeatureExtractor, model: nn.Module, device: str, criterion: nn.Module) -> tuple[Tensor, Tensor]:
    X, y = batch
    X = feature_extract(X, extractor)
    X, y = X.to(device), y.to(device)

    o = model(X)

    if hasattr(o, "logits"):  # Use for ViTForImageClassification
        outs: Tensor = o.logits
    else:  # Use for other model containing ViTModel
        outs: Tensor = o

    loss = criterion(outs, y)
    preds = outs.argmax(-1)
    accuracy = torch.mean((preds == y).float())

    return loss, accuracy

loss, accuracy = model_fn(next(iter(loader)), extractor, model, "cuda", nn.CrossEntropyLoss())
```
### Update
Tested on official docs, the return type is not functioning either, see table below. I modified on code provided by offical docs:
```python
from transformers import ViTFeatureExtractor, ViTForImageClassification
from PIL import Image
import requests

url = "http://images.cocodataset.org/val2017/000000039769.jpg"
image = Image.open(requests.get(url, stream=True).raw)

feature_extractor = ViTFeatureExtractor.from_pretrained("google/vit-base-patch16-224")
model = ViTForImageClassification.from_pretrained("google/vit-base-patch16-224")

feature_extractor.do_resize = False
feature_extractor.do_normalize = True
inputs = feature_extractor(images=image, return_tensors="pt")
# or
inputs = feature_extractor(images=[image, image], return_tensors="pt")
```
type of `image` is
```
>>> type(image)
PIL.JpegImagePlugin.JpegImageFile
```
and results are
|  `do_resize`  |  `do_normalize`  |  `return_tensors`  |  actual return type  |
|  :----------:  |  :---------------:  |  :----------------:  |  :-----------:  |
|  ✅  |  ✅  |  not specified  |  `list[np.ndarray]`  |
|  ✅  |  ✅  |  "pt"  |  4-D `torch.Tensor` <br> with shape `(B, C, H, W)`  | 
|  ❌  |  ✅  |   not specified |  `list[np.ndarray]`  |
|  ❌  |  ✅  |  "pt"  | 4-D `torch.Tensor` <br> with shape `(B, C, H, W)` |
|  ✅  |  ❌  |   not specified  |  `list[PIL.Image.Image]`  |
|  ✅  |  ❌  |  "pt"  |  ValueError  |
|  ❌  |  ❌  |   not specified  |  `list[PIL.JpegImagePlugin.JpegImageFile]`  |
|  ❌  |  ❌  |  "pt"  |  ValueError  |

I have only tested when input is `torch.Tensor | list[torch.Tensor]` or `PIL.Image.Image | list[PIL.Image.Image]`. Not sure if other conditions work properly.
<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior
Return type of `ViTFeatureExtractor.__call__` should match `return_tensors`, and there should be no error in table above.
<!-- A clear and concise description of what you would expect to happen. -->
