

TITLE
Correctly tokenize sentence pairs

COMMENTS
3

REACTIONS
+1: 0
-1: 0
laugh: 0
hooray: 0
heart: 0
rocket: 0
eyes: 0

BODY
Hey,

I saw different ways to tokenize sentence pairs and the intuitive one is not shown here:

https://huggingface.co/transformers/preprocessing.html#preprocessing-pairs-of-sentences

So, I am asking here if I do right.

I encode pairs of sentence using list of lists. Instead handing over two seperate lists for each sentences, I handover a list of list, where each element is a list with  a single sentence pair. So:

pairs=[[sen1, sen2],[sen1,sen2],....]

Is this hopefully right too?