perplexity-lenses / README.md

Commit History

Sync with data tooling repo, using edugp/kenlm models, updating viz to use quantiles for coloring and ad-hoc viz for the registry dataset
3c30fa3

edugp commited on

Run tokenizer before computing perplexity and format
7b62017

edugp commited on

Add tests and fix issue when splitting into sentences, to grab the minimum number between total sentences and sample size, rather than total original documents and sample size
d131aa3

edugp commited on

Update README
6d1a001

edugp commited on

initial commit
77d22a6

system HF staff commited on