crossdelenna's picture
add tokenizer
765b4cd
raw
history blame
286 Bytes
{"m": 0, "z": 1, "b": 2, "i": 3, "r": 4, "-": 5, "'": 6, "q": 7, "u": 8, "a": 9, "c": 10, "g": 11, "f": 12, "v": 13, "p": 14, "j": 15, "h": 16, "y": 17, "l": 18, "s": 19, "e": 20, "o": 21, "t": 22, "k": 23, "n": 24, "d": 25, "x": 26, "w": 28, ",": 29, "|": 27, "[UNK]": 30, "[PAD]": 31}