Hugging Face's logo

Spaces:
mponty
/
some_tokenization
Sleeping

App Files Community
mponty commited on
Commit
4d09e88
·
verified ·
1 Parent(s): 8456741

Update app.py

Browse files
Files changed (1) hide show
  1. app.py +1 -0
app.py CHANGED
@@ -29,6 +29,7 @@ tokenizer = st.selectbox(
29
  "White Space",
30
  "Qwen2.5 Tokenizer",
31
  "RuAdapt Tokenizer",
 
32
  "Byte Pair Encoding (Open AI GPT-4o)",
33
  ),
34
  index=None,
 
29
  "White Space",
30
  "Qwen2.5 Tokenizer",
31
  "RuAdapt Tokenizer",
32
+ "Aya-Expanse Tokenizer"
33
  "Byte Pair Encoding (Open AI GPT-4o)",
34
  ),
35
  index=None,