Add F1 analysis

results/analysis.ipynb

@@ -0,0 +1,465 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# F1 Scores"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "vscode": {
+     "languageId": "r"
+    }
+   },
+   "outputs": [],
+   "source": [
+    "library(\"ggpubr\")\n",
+    "library(readr)\n",
+    "library(ggplot2)\n",
+    "library(tidyverse)\n",
+    "library(ARTool)\n",
+    "library(emmeans)\n",
+    "library(multcomp)\n",
+    "library(car)\n",
+    "library(rstatix)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 10,
+   "metadata": {
+    "vscode": {
+     "languageId": "r"
+    }
+   },
+   "outputs": [
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "New names:\n",
+      "* `` -> ...1\n",
+      "\n",
+      "\u001b[1mRows: \u001b[22m\u001b[34m59\u001b[39m \u001b[1mColumns: \u001b[22m\u001b[34m5\u001b[39m\n",
+      "\u001b[36m--\u001b[39m \u001b[1mColumn specification\u001b[22m \u001b[36m--------------------------------------------------------\u001b[39m\n",
+      "\u001b[1mDelimiter:\u001b[22m \",\"\n",
+      "\u001b[32mdbl\u001b[39m (5): ...1, faiss_dpr, faiss_longformer, es_dpr, es_longformer\n",
+      "\n",
+      "\u001b[36mi\u001b[39m Use `spec()` to retrieve the full column specification for this data.\n",
+      "\u001b[36mi\u001b[39m Specify the column types or set `show_col_types = FALSE` to quiet this message.\n"
+     ]
+    },
+    {
+     "data": {
+      "text/html": [
+       "<table class=\"dataframe\">\n",
+       "<caption>A tibble: 6 × 4</caption>\n",
+       "<thead>\n",
+       "\t<tr><th scope=col>question</th><th scope=col>retriever</th><th scope=col>reader</th><th scope=col>f1</th></tr>\n",
+       "\t<tr><th scope=col>&lt;dbl&gt;</th><th scope=col>&lt;fct&gt;</th><th scope=col>&lt;fct&gt;</th><th scope=col>&lt;dbl&gt;</th></tr>\n",
+       "</thead>\n",
+       "<tbody>\n",
+       "\t<tr><td>0</td><td>faiss</td><td>dpr       </td><td>0.0000000</td></tr>\n",
+       "\t<tr><td>0</td><td>faiss</td><td>longformer</td><td>0.0000000</td></tr>\n",
+       "\t<tr><td>0</td><td>es   </td><td>dpr       </td><td>0.1300813</td></tr>\n",
+       "\t<tr><td>0</td><td>es   </td><td>longformer</td><td>0.7692308</td></tr>\n",
+       "\t<tr><td>1</td><td>faiss</td><td>dpr       </td><td>0.0000000</td></tr>\n",
+       "\t<tr><td>1</td><td>faiss</td><td>longformer</td><td>0.0000000</td></tr>\n",
+       "</tbody>\n",
+       "</table>\n"
+      ],
+      "text/latex": [
+       "A tibble: 6 × 4\n",
+       "\\begin{tabular}{llll}\n",
+       " question & retriever & reader & f1\\\\\n",
+       " <dbl> & <fct> & <fct> & <dbl>\\\\\n",
+       "\\hline\n",
+       "\t 0 & faiss & dpr        & 0.0000000\\\\\n",
+       "\t 0 & faiss & longformer & 0.0000000\\\\\n",
+       "\t 0 & es    & dpr        & 0.1300813\\\\\n",
+       "\t 0 & es    & longformer & 0.7692308\\\\\n",
+       "\t 1 & faiss & dpr        & 0.0000000\\\\\n",
+       "\t 1 & faiss & longformer & 0.0000000\\\\\n",
+       "\\end{tabular}\n"
+      ],
+      "text/markdown": [
+       "\n",
+       "A tibble: 6 × 4\n",
+       "\n",
+       "| question &lt;dbl&gt; | retriever &lt;fct&gt; | reader &lt;fct&gt; | f1 &lt;dbl&gt; |\n",
+       "|---|---|---|---|\n",
+       "| 0 | faiss | dpr        | 0.0000000 |\n",
+       "| 0 | faiss | longformer | 0.0000000 |\n",
+       "| 0 | es    | dpr        | 0.1300813 |\n",
+       "| 0 | es    | longformer | 0.7692308 |\n",
+       "| 1 | faiss | dpr        | 0.0000000 |\n",
+       "| 1 | faiss | longformer | 0.0000000 |\n",
+       "\n"
+      ],
+      "text/plain": [
+       "  question retriever reader     f1       \n",
+       "1 0        faiss     dpr        0.0000000\n",
+       "2 0        faiss     longformer 0.0000000\n",
+       "3 0        es        dpr        0.1300813\n",
+       "4 0        es        longformer 0.7692308\n",
+       "5 1        faiss     dpr        0.0000000\n",
+       "6 1        faiss     longformer 0.0000000"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    }
+   ],
+   "source": [
+    "f1_scores <- read_csv(\"f1_scores.csv\") %>%\n",
+    "    rename(question = `...1`) %>%\n",
+    "    pivot_longer(!question, names_to=c(\"retriever\", \"reader\"), names_sep=\"_\", values_to=\"f1\")\n",
+    "\n",
+    "f1_scores$retriever = as.factor(f1_scores$retriever)\n",
+    "f1_scores$reader = as.factor(f1_scores$reader)\n",
+    "\n",
+    "head(f1_scores)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "To test which tests we can use, we need to check for normality. For this, we use a Shapiro-Wilk test of normality. As you can see in the results below, all $p$-values are lower than 0.001, so we reject the null-hypothesis of normality and now know that none of the f1-scores are normally distributed."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 21,
+   "metadata": {
+    "vscode": {
+     "languageId": "r"
+    }
+   },
+   "outputs": [
+    {
+     "data": {
+      "text/html": [
+       "<table class=\"dataframe\">\n",
+       "<caption>A tibble: 1 × 3</caption>\n",
+       "<thead>\n",
+       "\t<tr><th scope=col>variable</th><th scope=col>statistic</th><th scope=col>p</th></tr>\n",
+       "\t<tr><th scope=col>&lt;chr&gt;</th><th scope=col>&lt;dbl&gt;</th><th scope=col>&lt;dbl&gt;</th></tr>\n",
+       "</thead>\n",
+       "<tbody>\n",
+       "\t<tr><td>f1</td><td>0.5086706</td><td>3.999447e-18</td></tr>\n",
+       "</tbody>\n",
+       "</table>\n"
+      ],
+      "text/latex": [
+       "A tibble: 1 × 3\n",
+       "\\begin{tabular}{lll}\n",
+       " variable & statistic & p\\\\\n",
+       " <chr> & <dbl> & <dbl>\\\\\n",
+       "\\hline\n",
+       "\t f1 & 0.5086706 & 3.999447e-18\\\\\n",
+       "\\end{tabular}\n"
+      ],
+      "text/markdown": [
+       "\n",
+       "A tibble: 1 × 3\n",
+       "\n",
+       "| variable &lt;chr&gt; | statistic &lt;dbl&gt; | p &lt;dbl&gt; |\n",
+       "|---|---|---|\n",
+       "| f1 | 0.5086706 | 3.999447e-18 |\n",
+       "\n"
+      ],
+      "text/plain": [
+       "  variable statistic p           \n",
+       "1 f1       0.5086706 3.999447e-18"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "data": {
+      "text/html": [
+       "<table class=\"dataframe\">\n",
+       "<caption>A tibble: 1 × 3</caption>\n",
+       "<thead>\n",
+       "\t<tr><th scope=col>variable</th><th scope=col>statistic</th><th scope=col>p</th></tr>\n",
+       "\t<tr><th scope=col>&lt;chr&gt;</th><th scope=col>&lt;dbl&gt;</th><th scope=col>&lt;dbl&gt;</th></tr>\n",
+       "</thead>\n",
+       "<tbody>\n",
+       "\t<tr><td>f1</td><td>0.7704567</td><td>2.671656e-12</td></tr>\n",
+       "</tbody>\n",
+       "</table>\n"
+      ],
+      "text/latex": [
+       "A tibble: 1 × 3\n",
+       "\\begin{tabular}{lll}\n",
+       " variable & statistic & p\\\\\n",
+       " <chr> & <dbl> & <dbl>\\\\\n",
+       "\\hline\n",
+       "\t f1 & 0.7704567 & 2.671656e-12\\\\\n",
+       "\\end{tabular}\n"
+      ],
+      "text/markdown": [
+       "\n",
+       "A tibble: 1 × 3\n",
+       "\n",
+       "| variable &lt;chr&gt; | statistic &lt;dbl&gt; | p &lt;dbl&gt; |\n",
+       "|---|---|---|\n",
+       "| f1 | 0.7704567 | 2.671656e-12 |\n",
+       "\n"
+      ],
+      "text/plain": [
+       "  variable statistic p           \n",
+       "1 f1       0.7704567 2.671656e-12"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "data": {
+      "text/html": [
+       "<table class=\"dataframe\">\n",
+       "<caption>A tibble: 1 × 3</caption>\n",
+       "<thead>\n",
+       "\t<tr><th scope=col>variable</th><th scope=col>statistic</th><th scope=col>p</th></tr>\n",
+       "\t<tr><th scope=col>&lt;chr&gt;</th><th scope=col>&lt;dbl&gt;</th><th scope=col>&lt;dbl&gt;</th></tr>\n",
+       "</thead>\n",
+       "<tbody>\n",
+       "\t<tr><td>f1</td><td>0.6741031</td><td>7.912632e-15</td></tr>\n",
+       "</tbody>\n",
+       "</table>\n"
+      ],
+      "text/latex": [
+       "A tibble: 1 × 3\n",
+       "\\begin{tabular}{lll}\n",
+       " variable & statistic & p\\\\\n",
+       " <chr> & <dbl> & <dbl>\\\\\n",
+       "\\hline\n",
+       "\t f1 & 0.6741031 & 7.912632e-15\\\\\n",
+       "\\end{tabular}\n"
+      ],
+      "text/markdown": [
+       "\n",
+       "A tibble: 1 × 3\n",
+       "\n",
+       "| variable &lt;chr&gt; | statistic &lt;dbl&gt; | p &lt;dbl&gt; |\n",
+       "|---|---|---|\n",
+       "| f1 | 0.6741031 | 7.912632e-15 |\n",
+       "\n"
+      ],
+      "text/plain": [
+       "  variable statistic p           \n",
+       "1 f1       0.6741031 7.912632e-15"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "data": {
+      "text/html": [
+       "<table class=\"dataframe\">\n",
+       "<caption>A tibble: 1 × 3</caption>\n",
+       "<thead>\n",
+       "\t<tr><th scope=col>variable</th><th scope=col>statistic</th><th scope=col>p</th></tr>\n",
+       "\t<tr><th scope=col>&lt;chr&gt;</th><th scope=col>&lt;dbl&gt;</th><th scope=col>&lt;dbl&gt;</th></tr>\n",
+       "</thead>\n",
+       "<tbody>\n",
+       "\t<tr><td>f1</td><td>0.6558935</td><td>3.037616e-15</td></tr>\n",
+       "</tbody>\n",
+       "</table>\n"
+      ],
+      "text/latex": [
+       "A tibble: 1 × 3\n",
+       "\\begin{tabular}{lll}\n",
+       " variable & statistic & p\\\\\n",
+       " <chr> & <dbl> & <dbl>\\\\\n",
+       "\\hline\n",
+       "\t f1 & 0.6558935 & 3.037616e-15\\\\\n",
+       "\\end{tabular}\n"
+      ],
+      "text/markdown": [
+       "\n",
+       "A tibble: 1 × 3\n",
+       "\n",
+       "| variable &lt;chr&gt; | statistic &lt;dbl&gt; | p &lt;dbl&gt; |\n",
+       "|---|---|---|\n",
+       "| f1 | 0.6558935 | 3.037616e-15 |\n",
+       "\n"
+      ],
+      "text/plain": [
+       "  variable statistic p           \n",
+       "1 f1       0.6558935 3.037616e-15"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    }
+   ],
+   "source": [
+    "f1_scores %>%\n",
+    "    filter(retriever == \"faiss\") %>%\n",
+    "    shapiro_test(f1)\n",
+    "\n",
+    "f1_scores %>%\n",
+    "    filter(retriever == \"es\") %>%\n",
+    "    shapiro_test(f1)\n",
+    "\n",
+    "f1_scores %>%\n",
+    "    filter(reader == \"dpr\") %>%\n",
+    "    shapiro_test(f1)\n",
+    "\n",
+    "f1_scores %>%\n",
+    "    filter(reader == \"longformer\") %>%\n",
+    "    shapiro_test(f1)\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Since our data is not normally distributed, we cannot use an ANOVA to compare our results. Therefore, we use an aligned-rank test, which is a non-parameteric version of a factorial repeated measures ANOVA."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 22,
+   "metadata": {
+    "vscode": {
+     "languageId": "r"
+    }
+   },
+   "outputs": [
+    {
+     "data": {
+      "text/html": [
+       "<table class=\"dataframe\">\n",
+       "<caption>A anova.art: 3 × 7</caption>\n",
+       "<thead>\n",
+       "\t<tr><th></th><th scope=col>Term</th><th scope=col>Df</th><th scope=col>Df.res</th><th scope=col>Sum Sq</th><th scope=col>Sum Sq.res</th><th scope=col>F value</th><th scope=col>Pr(&gt;F)</th></tr>\n",
+       "\t<tr><th></th><th scope=col>&lt;chr&gt;</th><th scope=col>&lt;dbl&gt;</th><th scope=col>&lt;dbl&gt;</th><th scope=col>&lt;dbl&gt;</th><th scope=col>&lt;dbl&gt;</th><th scope=col>&lt;dbl&gt;</th><th scope=col>&lt;dbl&gt;</th></tr>\n",
+       "</thead>\n",
+       "<tbody>\n",
+       "\t<tr><th scope=row>retriever</th><td>retriever       </td><td>1</td><td>232</td><td>200452.90</td><td>793168.0</td><td>58.63206</td><td>5.105423e-13</td></tr>\n",
+       "\t<tr><th scope=row>reader</th><td>reader          </td><td>1</td><td>232</td><td> 66045.36</td><td>944311.6</td><td>16.22613</td><td>7.620176e-05</td></tr>\n",
+       "\t<tr><th scope=row>retriever:reader</th><td>retriever:reader</td><td>1</td><td>232</td><td>158290.44</td><td>843714.0</td><td>43.52587</td><td>2.804257e-10</td></tr>\n",
+       "</tbody>\n",
+       "</table>\n"
+      ],
+      "text/latex": [
+       "A anova.art: 3 × 7\n",
+       "\\begin{tabular}{r|lllllll}\n",
+       "  & Term & Df & Df.res & Sum Sq & Sum Sq.res & F value & Pr(>F)\\\\\n",
+       "  & <chr> & <dbl> & <dbl> & <dbl> & <dbl> & <dbl> & <dbl>\\\\\n",
+       "\\hline\n",
+       "\tretriever & retriever        & 1 & 232 & 200452.90 & 793168.0 & 58.63206 & 5.105423e-13\\\\\n",
+       "\treader & reader           & 1 & 232 &  66045.36 & 944311.6 & 16.22613 & 7.620176e-05\\\\\n",
+       "\tretriever:reader & retriever:reader & 1 & 232 & 158290.44 & 843714.0 & 43.52587 & 2.804257e-10\\\\\n",
+       "\\end{tabular}\n"
+      ],
+      "text/markdown": [
+       "\n",
+       "A anova.art: 3 × 7\n",
+       "\n",
+       "| <!--/--> | Term &lt;chr&gt; | Df &lt;dbl&gt; | Df.res &lt;dbl&gt; | Sum Sq &lt;dbl&gt; | Sum Sq.res &lt;dbl&gt; | F value &lt;dbl&gt; | Pr(&gt;F) &lt;dbl&gt; |\n",
+       "|---|---|---|---|---|---|---|---|\n",
+       "| retriever | retriever        | 1 | 232 | 200452.90 | 793168.0 | 58.63206 | 5.105423e-13 |\n",
+       "| reader | reader           | 1 | 232 |  66045.36 | 944311.6 | 16.22613 | 7.620176e-05 |\n",
+       "| retriever:reader | retriever:reader | 1 | 232 | 158290.44 | 843714.0 | 43.52587 | 2.804257e-10 |\n",
+       "\n"
+      ],
+      "text/plain": [
+       "                 Term             Df Df.res Sum Sq    Sum Sq.res F value \n",
+       "retriever        retriever        1  232    200452.90 793168.0   58.63206\n",
+       "reader           reader           1  232     66045.36 944311.6   16.22613\n",
+       "retriever:reader retriever:reader 1  232    158290.44 843714.0   43.52587\n",
+       "                 Pr(>F)      \n",
+       "retriever        5.105423e-13\n",
+       "reader           7.620176e-05\n",
+       "retriever:reader 2.804257e-10"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "NOTE: Results may be misleading due to involvement in interactions\n",
+      "\n"
+     ]
+    },
+    {
+     "data": {
+      "text/plain": [
+       " contrast   estimate   SE  df t.ratio p.value\n",
+       " es - faiss     58.3 7.61 232   7.657  <.0001\n",
+       "\n",
+       "Results are averaged over the levels of: reader "
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "NOTE: Results may be misleading due to involvement in interactions\n",
+      "\n"
+     ]
+    },
+    {
+     "data": {
+      "text/plain": [
+       " contrast         estimate   SE  df t.ratio p.value\n",
+       " dpr - longformer    -33.5 8.31 232  -4.028  0.0001\n",
+       "\n",
+       "Results are averaged over the levels of: retriever "
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    }
+   ],
+   "source": [
+    "model.acc <- art(f1 ~ retriever * reader, data = f1_scores)\n",
+    "anova(model.acc)\n",
+    "art.con(model.acc, ~ retriever)\n",
+    "art.con(model.acc, ~ reader)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "From these results, we can see that both the retriever and the reader have a significant effect on the F1 score ($F = 58.63$ and $F = 16.23$ respectively, $p < 0.0001$ for both). However, there is also an interaction between the retriever and reader ($F = 43.53$, $p < 0.0001$). The post-hoc analysis of contrasts shows that ElasticSearch performs better than FAISS ($p < 0.0001$) and Longformer performs better than DPR ($p < 0.0001$)."
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "R",
+   "language": "R",
+   "name": "ir"
+  },
+  "language_info": {
+   "codemirror_mode": "r",
+   "file_extension": ".r",
+   "mimetype": "text/x-r-source",
+   "name": "R",
+   "pygments_lexer": "r",
+   "version": "4.1.2"
+  },
+  "orig_nbformat": 4
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}