

TITLE
Loading a set of tokenized files for training

COMMENTS
3

REACTIONS
+1: 0
-1: 0
laugh: 0
hooray: 0
heart: 0
rocket: 0
eyes: 0

BODY
I have a directory of files that are already tokenized using a pretrained tokenizer. Each file is a pickle file containing a list of objects where each object corresponds to a text sequence containing input_ids and attention_masks. The directory has thousands of files. I'm looking for an efficient way to load the data for training using Trainer. Do I have to write my own Dataloader or do I create a custom dataset using Datasets? 

Thank you. 