

TITLE
squad data preprocessor error (list index out of range) while finetuning bert on squad 1.1

COMMENTS
2

REACTIONS
+1: 0
-1: 0
laugh: 0
hooray: 0
heart: 0
rocket: 0
eyes: 0

BODY
run_squad.py throws this error on squad v1.1 dataset 

Traceback (most recent call last):
  File "run_squad.py", line 820, in <module>
    main()
  File "run_squad.py", line 762, in main
    train_dataset = load_and_cache_examples(args, tokenizer, evaluate=False, output_examples=False)
  File "run_squad.py", line 446, in load_and_cache_examples
    examples = processor.get_train_examples(args.data_dir, filename=args.train_file)
  File "/home/din/question_answering_deepQA/venv_indic_deepQA/lib/python3.6/site-packages/transformers/data/processors/squad.py", line 602, in get_train_examples
    return self._create_examples(input_data, "train")
  File "/home/din/question_answering_deepQA/venv_indic_deepQA/lib/python3.6/site-packages/transformers/data/processors/squad.py", line 656, in _create_examples
    answers=answers,
  File "/home/din/question_answering_deepQA/venv_indic_deepQA/lib/python3.6/site-packages/transformers/data/processors/squad.py", line 729, in __init__
    self.start_position = char_to_word_offset[start_position_character]
IndexError: list index out of range
