Update README.md
Browse files
README.md
CHANGED
|
@@ -24,30 +24,13 @@ base_model:
|
|
| 24 |
Evaluation results on ZebraGrid
|
| 25 |
|
| 26 |
```
|
| 27 |
-
╒══════════════════════════════════╤════════╤══════════╤══════════╤══════════════╤═══════════════════╤═══════════════════╤════════════╤═════════════╤═════════════════╤═══════════════╕
|
| 28 |
│ Model │ Mode │ N_Mode │ N_Size │ Puzzle Acc │ Easy Puzzle Acc │ Hard Puzzle Acc │ Cell Acc │ No answer │ Total Puzzles │ Reason Lens │
|
| 29 |
-
╞══════════════════════════════════╪════════╪══════════╪══════════╪══════════════╪═══════════════════╪═══════════════════╪════════════╪═════════════╪═════════════════╪═══════════════╡
|
| 30 |
│ o1-preview-2024-09-12 │ greedy │ single │ 1 │ 71.4 │ 98.57 │ 60.83 │ 75.14 │ 0.3 │ 1000 │ 1565.88 │
|
| 31 |
-
│ o1-preview-2024-09-12-v2 │ greedy │ single │ 1 │ 70.4 │ 98.21 │ 59.58 │ 74.18 │ 0.4 │ 1000 │ 1559.71 │
|
| 32 |
-
│ o1-mini-2024-09-12-v3 │ greedy │ single │ 1 │ 59.7 │ 86.07 │ 49.44 │ 70.32 │ 1 │ 1000 │ 1166.38 │
|
| 33 |
-
│ o1-mini-2024-09-12-v2 │ greedy │ single │ 1 │ 56.8 │ 82.86 │ 46.67 │ 69.87 │ 1.3 │ 1000 │ 1164.95 │
|
| 34 |
-
│ o1-mini-2024-09-12 │ greedy │ single │ 1 │ 52.6 │ 87.14 │ 39.17 │ 52.29 │ 0.8 │ 1000 │ 993.28 │
|
| 35 |
│ claude-3-5-sonnet-20241022 │ greedy │ single │ 1 │ 36.2 │ 91.07 │ 14.86 │ 54.27 │ 0 │ 1000 │ 861.18 │
|
| 36 |
-
│ claude-3-5-sonnet-20240620 │ greedy │ single │ 1 │ 33.4 │ 87.5 │ 12.36 │ 54.34 │ 0 │ 1000 │ 1141.94 │
|
| 37 |
│ Llama-3.1-405B-Inst-fp8@together │ greedy │ single │ 1 │ 32.6 │ 87.14 │ 11.39 │ 45.8 │ 12.5 │ 1000 │ 314.66 │
|
| 38 |
-
│ gpt-4o-2024-08-06 │ greedy │ single │ 1 │ 31.7 │ 84.64 │ 11.11 │ 50.34 │ 3.6 │ 1000 │ 1106.51 │
|
| 39 |
-
│ gemini-1.5-pro-exp-0827 │ greedy │ single │ 1 │ 30.5 │ 79.64 │ 11.39 │ 50.84 │ 0.8 │ 1000 │ 1594.47 │
|
| 40 |
-
│ Llama-3.1-405B-Inst@sambanova │ greedy │ single │ 1 │ 30.1 │ 84.64 │ 8.89 │ 39.06 │ 24.7 │ 1000 │ 2001.12 │
|
| 41 |
-
│ chatgpt-4o-latest-24-09-07 │ greedy │ single │ 1 │ 29.9 │ 81.43 │ 9.86 │ 48.83 │ 4.2 │ 1000 │ 1539.99 │
|
| 42 |
-
│ Mistral-Large-2 │ greedy │ single │ 1 │ 29 │ 80.36 │ 9.03 │ 47.64 │ 1.7 │ 1000 │ 1592.39 │
|
| 43 |
-
│ gpt-4-turbo-2024-04-09 │ greedy │ single │ 1 │ 28.4 │ 80.71 │ 8.06 │ 47.9 │ 0.1 │ 1000 │ 1148.46 │
|
| 44 |
-
│ gpt-4o-2024-05-13 │ greedy │ single │ 1 │ 28.2 │ 77.86 │ 8.89 │ 38.72 │ 19.3 │ 1000 │ 1643.51 │
|
| 45 |
│ Athene-V2-Chat-AWQ │ greedy │ single │ 1 │ 27.8 │ 77.14 │ 8.61 │ 45.83 │ 6.4 │ 1000 │ 1785.7 │
|
| 46 |
-
│ gpt-4-0314 │ greedy │ single │ 1 │ 27.1 │ 77.14 │ 7.64 │ 47.43 │ 0.2 │ 1000 │ 1203.17 │
|
| 47 |
-
│ claude-3-opus-20240229 │ greedy │ single │ 1 │ 27 │ 78.21 │ 7.08 │ 48.91 │ 0 │ 1000 │ 855.72 │
|
| 48 |
│ Qwen2.5-72B-Instruct │ greedy │ single │ 1 │ 26.6 │ 76.43 │ 7.22 │ 40.92 │ 11.9 │ 1000 │ 1795.9 │
|
| 49 |
│ Qwen2.5-32B-Instruct │ greedy │ single │ 1 │ 26.1 │ 77.5 │ 6.11 │ 43.39 │ 6.3 │ 1000 │ 1333.07 │
|
| 50 |
│ Athene-70B │ greedy │ single │ 1 │ 16.7 │ 52.5 │ 2.78 │ 32.98 │ 21.1 │ 1000 │ 391.19 │
|
| 51 |
-
╘══════════════════════════════════╧════════╧══════════╧══════════╧══════════════╧═══════════════════╧═══════════════════╧════════════╧═════════════╧═════════════════╧═══════════════╛
|
| 52 |
```
|
| 53 |
|
|
|
|
| 24 |
Evaluation results on ZebraGrid
|
| 25 |
|
| 26 |
```
|
|
|
|
| 27 |
│ Model │ Mode │ N_Mode │ N_Size │ Puzzle Acc │ Easy Puzzle Acc │ Hard Puzzle Acc │ Cell Acc │ No answer │ Total Puzzles │ Reason Lens │
|
|
|
|
| 28 |
│ o1-preview-2024-09-12 │ greedy │ single │ 1 │ 71.4 │ 98.57 │ 60.83 │ 75.14 │ 0.3 │ 1000 │ 1565.88 │
|
|
|
|
|
|
|
|
|
|
|
|
|
| 29 |
│ claude-3-5-sonnet-20241022 │ greedy │ single │ 1 │ 36.2 │ 91.07 │ 14.86 │ 54.27 │ 0 │ 1000 │ 861.18 │
|
|
|
|
| 30 |
│ Llama-3.1-405B-Inst-fp8@together │ greedy │ single │ 1 │ 32.6 │ 87.14 │ 11.39 │ 45.8 │ 12.5 │ 1000 │ 314.66 │
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 31 |
│ Athene-V2-Chat-AWQ │ greedy │ single │ 1 │ 27.8 │ 77.14 │ 8.61 │ 45.83 │ 6.4 │ 1000 │ 1785.7 │
|
|
|
|
|
|
|
| 32 |
│ Qwen2.5-72B-Instruct │ greedy │ single │ 1 │ 26.6 │ 76.43 │ 7.22 │ 40.92 │ 11.9 │ 1000 │ 1795.9 │
|
| 33 |
│ Qwen2.5-32B-Instruct │ greedy │ single │ 1 │ 26.1 │ 77.5 │ 6.11 │ 43.39 │ 6.3 │ 1000 │ 1333.07 │
|
| 34 |
│ Athene-70B │ greedy │ single │ 1 │ 16.7 │ 52.5 │ 2.78 │ 32.98 │ 21.1 │ 1000 │ 391.19 │
|
|
|
|
| 35 |
```
|
| 36 |
|