|
--- |
|
language: en |
|
license: apache-2.0 |
|
library_name: transformers |
|
model-index: |
|
- name: digital-socrates-13b |
|
results: |
|
- task: |
|
type: text-generation |
|
name: Text Generation |
|
dataset: |
|
name: AI2 Reasoning Challenge (25-Shot) |
|
type: ai2_arc |
|
config: ARC-Challenge |
|
split: test |
|
args: |
|
num_few_shot: 25 |
|
metrics: |
|
- type: acc_norm |
|
value: 58.36 |
|
name: normalized accuracy |
|
source: |
|
url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=allenai/digital-socrates-13b |
|
name: Open LLM Leaderboard |
|
- task: |
|
type: text-generation |
|
name: Text Generation |
|
dataset: |
|
name: HellaSwag (10-Shot) |
|
type: hellaswag |
|
split: validation |
|
args: |
|
num_few_shot: 10 |
|
metrics: |
|
- type: acc_norm |
|
value: 80.14 |
|
name: normalized accuracy |
|
source: |
|
url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=allenai/digital-socrates-13b |
|
name: Open LLM Leaderboard |
|
- task: |
|
type: text-generation |
|
name: Text Generation |
|
dataset: |
|
name: MMLU (5-Shot) |
|
type: cais/mmlu |
|
config: all |
|
split: test |
|
args: |
|
num_few_shot: 5 |
|
metrics: |
|
- type: acc |
|
value: 57.01 |
|
name: accuracy |
|
source: |
|
url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=allenai/digital-socrates-13b |
|
name: Open LLM Leaderboard |
|
- task: |
|
type: text-generation |
|
name: Text Generation |
|
dataset: |
|
name: TruthfulQA (0-shot) |
|
type: truthful_qa |
|
config: multiple_choice |
|
split: validation |
|
args: |
|
num_few_shot: 0 |
|
metrics: |
|
- type: mc2 |
|
value: 44.47 |
|
source: |
|
url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=allenai/digital-socrates-13b |
|
name: Open LLM Leaderboard |
|
- task: |
|
type: text-generation |
|
name: Text Generation |
|
dataset: |
|
name: Winogrande (5-shot) |
|
type: winogrande |
|
config: winogrande_xl |
|
split: validation |
|
args: |
|
num_few_shot: 5 |
|
metrics: |
|
- type: acc |
|
value: 74.59 |
|
name: accuracy |
|
source: |
|
url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=allenai/digital-socrates-13b |
|
name: Open LLM Leaderboard |
|
- task: |
|
type: text-generation |
|
name: Text Generation |
|
dataset: |
|
name: GSM8k (5-shot) |
|
type: gsm8k |
|
config: main |
|
split: test |
|
args: |
|
num_few_shot: 5 |
|
metrics: |
|
- type: acc |
|
value: 29.49 |
|
name: accuracy |
|
source: |
|
url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=allenai/digital-socrates-13b |
|
name: Open LLM Leaderboard |
|
--- |
|
|
|
This is the Digital Socrates 13B (DS-13B) model described in our paper: <b>Digital Socrates: Evaluating LLMs through explanation critiques</b> (ACL Anthology link: https://aclanthology.org/2024.acl-long.302, arXiv link: https://arxiv.org/abs/2311.09613). |
|
|
|
The less recommended, smaller 7B model can be found at https://huggingface.co/allenai/digital-socrates-7b |
|
|
|
The DS-13B model is a fine-tuned version of [Llama-2-13b-Chat](https://huggingface.co/meta-llama/Llama-2-13b-chat), please |
|
review its [guidelines](https://huggingface.co/meta-llama/Llama-2-13b-chat/blob/main/USE_POLICY.md) and |
|
[licensing](https://huggingface.co/meta-llama/Llama-2-13b-chat#model-details) before use. |
|
|
|
|
|
|
|
|
|
# What is Digital Socrates? |
|
Digital Socrates is an open-source, automatic explanation-critiquing model. It is useful for revealing insights about student |
|
models by examining their reasoning chains. The use of Digital Socrates allows for nuanced, interpretable automatic |
|
evaluation of explanations without expensive API calls or human annotations. |
|
|
|
While LLMs can provide reasoned explanations along with their answers, the nature and |
|
quality of those explanations are still poorly understood. In response, our goal is to define a detailed way of characterizing the |
|
explanation capabilities of modern models and to create a nuanced, interpretable explanation |
|
evaluation tool that can generate such characterizations automatically. |
|
|
|
Given a **question** (together with the multiple-choice options and gold answer), |
|
```commandline |
|
When Dennis operates his lawnmower, he notices the engine makes a lot of noise. He also notices that the engine gets very hot. Which best describes the heat and noise generated from the lawnmower? |
|
(A) a change in phase |
|
(B) thermal expansion |
|
(C) an increase in entropy |
|
(D) mechanical advantage |
|
|
|
Answer sheet: (C) |
|
``` |
|
as well as a student model explanation and answer, |
|
``` |
|
1) The question states that the lawnmower engine makes a lot of noise. |
|
2) The question states that the lawnmower engine gets very hot. |
|
3) Noise and heat are both forms of energy. |
|
4) The noise and heat generated from the lawnmower are a result of the conversion of energy from the fuel to mechanical energy. |
|
|
|
Answer: (D) |
|
``` |
|
|
|
Digital Socrates gives a **critique of the model-generated explanation** that provides localized feedback on the |
|
most significant flaw (if any) in the explanation, suggestions to address the identified flaw, as well as a numeric rating indicating the explanation quality: |
|
``` |
|
The explanation states or suggests the following: |
|
* Main flaw (standalone statement): "The noise and heat generated from the lawnmower are a result of the conversion of energy from the fuel to mechanical energy." |
|
* Dimension: incorrect_information |
|
|
|
Consider these points for revising the explanation: |
|
* General: It's important to understand the difference between the different types of energy. Mechanical energy is the energy of motion, while thermal energy is the energy of heat. |
|
* Specific: In the case of the lawnmower, the noise and heat are not a result of the conversion of energy from the fuel to mechanical energy. The noise is a result of the vibration of the engine, while the heat is a result of the friction and combustion of the fuel. |
|
|
|
Explanation score: 2 |
|
``` |
|
|
|
|
|
Remarkably, despite being orders of magnitude smaller than GPT-4, our Digital Socrates models are |
|
capable of generating critiques close to GPT-4 critiques in terms of human rating and other |
|
quantitative measures (correlation of explanation scores given and error category matches). |
|
Through quantitative and qualitative analysis, we demonstrate how Digital Socrates is useful for |
|
revealing insights about student models by examining their reasoning chains. |
|
|
|
We invite you to try out Digital Socrates for your own application! |
|
|
|
|
|
|
|
# How to use Digital Socrates? |
|
We provide a quick example of how you can try out Digital Socrates with just a few lines of code: |
|
|
|
'DSCritiqueBank-V1' used below can be downloaded from our [dataset page](https://allenai.org/data/digital-socrates). |
|
``` |
|
import json |
|
from transformers import AutoTokenizer, AutoModelForCausalLM |
|
# Load model and tokenizer |
|
model_path = "allenai/digital-socrates-13b" |
|
model = AutoModelForCausalLM.from_pretrained(model_path, device_map="auto") |
|
tokenizer = AutoTokenizer.from_pretrained(model_path) |
|
|
|
# Define input data |
|
question = "When Dennis operates his lawnmower, he notices the engine makes a lot of noise. He also notices that the engine gets very hot. Which best describes the heat and noise generated from the lawnmower? (A) a change in phase (B) thermal expansion (C) an increase in entropy (D) mechanical advantage" |
|
explanation = "1) The question states that the lawnmower engine makes a lot of noise.\n2) The question states that the lawnmower engine gets very hot.\n3) Noise and heat are both forms of energy.\n4) The noise and heat generated from the lawnmower are a result of the conversion of energy from the fuel to mechanical energy." |
|
answerkey = "C" |
|
predictedanswer = "D" |
|
|
|
# construct prompt (Llama conventions) |
|
with open("../DSCritiqueBank-V1/DSCB-prompts.json") as file: |
|
prompts = json.load(file) |
|
|
|
system_prompt = prompts['digital_socrates_v1']['system'] |
|
user_prompt = prompts['digital_socrates_v1']['main'].replace("[[QUESTION]]", question).replace("[[EXPLANATION]]", explanation).replace("[[PREDICTEDANSWER]]", predictedanswer).replace("[[ANSWERKEY]]", answerkey) |
|
|
|
full_prompt = f"[INST] <<SYS>>\n{system_prompt}\n<</SYS>{user_prompt} [/INST]\n\n" |
|
|
|
# Run model |
|
input_ids = tokenizer.encode(full_prompt, return_tensors="pt").to("cuda:0") |
|
output = model.generate(input_ids, max_new_tokens=512, temperature=0) |
|
res = tokenizer.batch_decode(output, skip_special_tokens=True) |
|
``` |
|
Print the output: |
|
``` |
|
>>> print(res[0].split("[/INST]")[-1]) |
|
|
|
The explanation states or suggests the following: |
|
* Main flaw (standalone statement): "The noise and heat generated from the lawnmower are a result of the conversion of energy from the fuel to mechanical energy." |
|
* Dimension: incorrect_information |
|
|
|
Consider these points for revising the explanation: |
|
* General: It's important to understand the difference between the different types of energy. Mechanical energy is the energy of motion, while thermal energy is the energy of heat. |
|
* Specific: In the case of the lawnmower, the noise and heat are not a result of the conversion of energy from the fuel to mechanical energy. The noise is a result of the vibration of the engine, while the heat is a result of the friction and combustion of the fuel. |
|
|
|
Explanation score: 2 |
|
``` |
|
|
|
|
|
|
|
# More details about Digital Socrates ... |
|
For more details about Digital Socrates, please refer to our: |
|
* 📄Paper: https://arxiv.org/abs/2311.09613 |
|
* 💻Dataset: https://allenai.org/data/digital-socrates |
|
|
|
|
|
# Citation |
|
|
|
``` |
|
@inproceedings{gu-etal-2024-digital, |
|
title = "Digital Socrates: Evaluating {LLM}s through Explanation Critiques", |
|
author = "Gu, Yuling and |
|
Tafjord, Oyvind and |
|
Clark, Peter", |
|
editor = "Ku, Lun-Wei and |
|
Martins, Andre and |
|
Srikumar, Vivek", |
|
booktitle = "Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)", |
|
month = aug, |
|
year = "2024", |
|
address = "Bangkok, Thailand", |
|
publisher = "Association for Computational Linguistics", |
|
url = "https://aclanthology.org/2024.acl-long.302", |
|
pages = "5559--5586", |
|
} |
|
``` |
|
|
|
# [Open LLM Leaderboard Evaluation Results](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard) |
|
Detailed results can be found [here](https://huggingface.co/datasets/open-llm-leaderboard/details_allenai__digital-socrates-13b) |
|
|
|
| Metric |Value| |
|
|---------------------------------|----:| |
|
|Avg. |57.34| |
|
|AI2 Reasoning Challenge (25-Shot)|58.36| |
|
|HellaSwag (10-Shot) |80.14| |
|
|MMLU (5-Shot) |57.01| |
|
|TruthfulQA (0-shot) |44.47| |
|
|Winogrande (5-shot) |74.59| |
|
|GSM8k (5-shot) |29.49| |
|
|
|
|
