Update tokenization_indictrans.py

#2
by VarunGumma - opened

Fix tokenization as per new HF Tokenizer

prajdabre changed pull request status to merged

Sign up or log in to comment