Spaces:

Dovakiins
/

qwerrwe

Build error

App Files Files Community

qwerrwe / src /axolotl /utils /tokenization.py

Commit History

prepared dataset caching, other misc fixes (#665)

e50a64e
unverified

winglian commited on Oct 3, 2023

use fastchat conversations template (#578)

e7d3e2d
unverified

winglian commited on Sep 27, 2023

minor tweaks to simplify (#597)

31b9e0c
unverified

winglian commited on Sep 18, 2023

Debug tokenization output: Add ability to output text only (no tokens), and/or specify num samples to see (#511)

48434be
unverified

Tom Jobbins commited on Aug 31, 2023

Adding logging enhancement

553a86b

theobjectivedad commited on Jul 14, 2023

add tests and supoort for loader for sys prompt data

3a38271

winglian commited on Jun 18, 2023

Apply isort then black

37293dc

Nanobit commited on May 29, 2023

Lint tokenization

e6b57de

Nanobit commited on May 29, 2023

black formatting

2bc1a5b

winglian commited on May 10, 2023

fix sharegpt tokenization, refactor tokenization debugging

5159d00

winglian commited on Apr 30, 2023