

TITLE
HuggingFace/BigBird RuntimeError: Internal: src/sentencepiece_processor.cc

COMMENTS
2

REACTIONS
+1: 0
-1: 0
laugh: 0
hooray: 0
heart: 0
rocket: 0
eyes: 0

BODY
### System Info

```shell
I'm able run the HuggingFace/BigBird code for a binary classification on a proprietary essay dataset in Google Colab with no errors.  I wanted to access more powerful GPU's and converted the code from .ipynb to .py to run on Marquette's supercomputer (called Raj).  Raj does not allow me to access the roberta model remotely so I changed the first line of code below to the second for local access (and also copied the bigbird-roberta-base files to Raj):

tokenizer = BigBirdTokenizer.from_pretrained('google/bigbird-roberta-base')
tokenizer = BigBirdTokenizer.from_pretrained('<my user path on Raj>/bigbird-roberta-base')

However, this gives me the following error:
RuntimeError: Internal: src/sentencepiece_processor.cc(890) [model_proto->ParseFromArray(serialized.data(), serialized.size())] 

I did confirm that sentencepiece 0.1.96 is installed and I'm using Python version 3.6.8.

Any help or suggestions is appreciated!
```


### Who can help?

@ydshieh, @SaulLu

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

from transformers import BigBirdTokenizer, BigBirdForSequenceClassification
print('Loading tokenizer...')
tokenizer = BigBirdTokenizer.from_pretrained(<my user path>/bigbird-roberta-base')
# Tokenize all of the sentences and map the tokens to thier word IDs.
input_ids = []
# Record the length of each sequence (in terms of BERT tokens).
lengths = []
print('Tokenizing comments...')
# For every sentence...
for sen in train.data:
    # Report progress.
    if ((len(input_ids) % 1000) == 0):
        print('  Read {:,} comments.'.format(len(input_ids)))
    # `encode` will:
    #   (1) Tokenize the sentence.
    #   (2) Prepend the `[CLS]` token to the start.
    #   (3) Append the `[SEP]` token to the end.
    #   (4) Map tokens to their IDs.
    encoded_sent = tokenizer.encode(
                        str(sen), # Sentence to encode.  Added str due to error.
                        add_special_tokens = True, # Add '[CLS]' and '[SEP]'
                   )
    # Add the encoded sentence to the list.
    input_ids.append(encoded_sent)
    # Record the non-truncated length.
    lengths.append(len(encoded_sent))
print('DONE.')

print('   Min length: {:,} tokens'.format(min(lengths)))
print('   Max length: {:,} tokens'.format(max(lengths)))
print('Median length: {:,} tokens'.format(np.median(lengths)))

### Expected behavior

```shell
The first print statement should generate:
Tokenizing comments...
  Read 0 comments.
DONE.
       454 comments

The second group of three print statements should generate:
Min length: 90 tokens
   Max length: 995 tokens
Median length: 826.5 tokens
```
