Update README.md

README.md

@@ -13,10 +13,9 @@ inference: false
 
 ## **"Garbage in, garbage out. A language model is only as good as its training data irrespective of its parameter count."**
 
-This educational value classifier is deeply inspired by [Textbooks Are All You Need](https://arxiv.org/abs/2306.11644), where a classifier was developed to predict the educational value of data, and was then used for data filtering.  
-Model is built on fasttext - it can classify more than 2000 examples per second in CPU, and so it can be used **on-the-fly** during pretraining.  
-The model can classify if a text has high educational value (more explicitly defined then textbook quality).  The definition change is a substantial change vs [kenhktsui/llm-data-textbook-quality-fasttext-classifer-v1](https://huggingface.co/kenhktsui/llm-data-textbook-quality-fasttext-classifer-v1).   
-It can be used as a filter for data curation when training a LLM.  
+📚  The educational value classifier can classify if a text from web has high educational value (more explicitly defined then textbook quality).  It is deeply inspired by [Textbooks Are All You Need](https://arxiv.org/abs/2306.11644), where a classifier was developed to predict the educational value of data, and was then used for data filtering. 
+The model is trained on web/ raw text, not on data formatted as instruction dataset (yet).  
+It can be used as a filter for pretraining data curation when training a LLM 🤖.  
 There are 3 labels instead of 2 labels, as it offers higher granularity of educational value.
 - High (Top 25% educational value)
 - Mid (Middle 25-75% educational value)
@@ -25,6 +24,8 @@ There are 3 labels instead of 2 labels, as it offers higher granularity of educa
 A detailed report/ paper will follow when more downstream experiments of this classifier become available.    
 The classifier had been applied to various pretraining dataset. See [**Benchmark**](https://huggingface.co/kenhktsui/llm-data-textbook-quality-fasttext-classifer-v2#benchmark)  
 
+⚡ Model is built on fasttext - it can classify more than 2000 examples per second in CPU, and so it can be used **on-the-fly** during pretraining.  
+
 Please note textbook quality is a subset of high quality.
 
 ## Feedback welcomed!
@@ -148,7 +149,10 @@ The classifier aligns with the expectation.
 - In general, the later a dataset is released, the higher the educational value it is because of increasing focus on data quality in the research community.
 - Textbook category (mostly synethetic) scores the highest, becuase they are created for educational value, reflecting the effectiveness of this model.   
 - Maths/ paper category scores the second highest, because of its density of knowledge.
-- Instruction dataset scores less than textbook because depth of knowledge in conversation is usually less dense in textbook, but they are in general more educative than unfiltered web.
 - Wikipedia scores comparatively lower because it also contains information (e.g. result of a match, award of a movie star) that has smaller educational value.  
 - Web scores low (if no filtering is applied) because it contains all domains.
-- Meme scores the lowest as expected. Hateful memes almost got zero point.
+- Meme scores the lowest as expected. Hateful memes almost got zero point.
+
+Some instruction datasets are added for curiosity sake although model is not trained on instruction data. There are two possible interpretations:
+- They score less than textbook because depth of knowledge in conversation is usually less dense in textbook, but they are in general more educative than unfiltered web.
+- The model does not perform well enough to tell educational value in instruction datasets.