benschlagman
/

tapas_fine_tuning

@@ -1,481 +0,0 @@
-# -*- coding: utf-8 -*-
-"""Fine-tuning Google Colab
-Automatically generated by Colaboratory.
-Original file is located at
-    https://colab.research.google.com/drive/1owUKmaxQ_M6Ybsc3R9pR7_3a-J24ZKwk
-## Introduction: TAPAS
-* Original TAPAS paper (ACL 2020): https://www.aclweb.org/anthology/2020.acl-main.398/
-* Follow-up paper on intermediate pre-training (EMMNLP Findings 2020): https://www.aclweb.org/anthology/2020.findings-emnlp.27/
-* Original Github repository: https://github.com/google-research/tapas
-* Blog post: https://ai.googleblog.com/2020/04/using-neural-networks-to-find-answers.html
-TAPAS is an algorithm that (among other tasks) can answer questions about tabular data. It is essentially a BERT model with relative position embeddings and additional token type ids that encode tabular structure, and 2 classification heads on top: one for **cell selection** and one for (optionally) performing an **aggregation** among selected cells (such as summing or counting).
-Similar to BERT, the base `TapasModel` is pre-trained using the masked language modeling (MLM) objective on a large collection of tables from Wikipedia and associated texts. In addition, the authors further pre-trained the model on an second task (table entailment) to increase the numerical reasoning capabilities of TAPAS (as explained in the follow-up paper), which further improves performance on downstream tasks.
-In this notebook, we are going to fine-tune `TapasForQuestionAnswering` on [Sequential Question Answering (SQA)](https://www.microsoft.com/en-us/research/publication/search-based-neural-structured-learning-sequential-question-answering/), a dataset built by Microsoft Research which deals with asking questions related to a table in a **conversational set-up**. We are going to do so as in the original paper, by adding a randomly initialized cell selection head on top of the pre-trained base model (note that SQA does not have questions that involve aggregation and hence no aggregation head), and then fine-tuning them altogether.
-First, we install both the Transformers library as well as the dependency on [`torch-scatter`](https://github.com/rusty1s/pytorch_scatter), which the model requires.
-"""
-! rm -r transformers
-! git clone https://github.com/huggingface/transformers.git
-! cd transformers
-! pip install ./transformers
-!pip3 install torch==1.7.1+cu101 -f https://download.pytorch.org/whl/torch_stable.html
-!pip3 install --no-index torch-scatter -f https://pytorch-geometric.com/whl/torch-1.7.0+cu101.html
-"""We also install a small portion from the SQA training dataset, for demonstration purposes. This is a TSV file containing table-question pairs. Besides this, we also download the `table_csv` directory, which contains the actual tabular data.
-Note that you can download the entire SQA dataset on the [official website](https://www.microsoft.com/en-us/download/details.aspx?id=54253).
-"""
-import requests, zipfile, io
-import os
-def download_files(dir_name):
-  if not os.path.exists(dir_name):
-    # 28 training examples from the SQA training set + table csv data
-    urls = ["https://www.dropbox.com/s/j6n2kq4n7ugippc/qa.zip?dl=1",
-            "https://www.dropbox.com/s/mlg46w8lwsn90ne/table_csv.zip?dl=1"
-    ]
-    for url in urls:
-      r = requests.get(url)
-      z = zipfile.ZipFile(io.BytesIO(r.content))
-      z.extractall()
-dir_name = "sqa_data"
-download_files(dir_name)
-"""## Prepare the data
-Let's look at the first few rows of the dataset:
-"""
-import pandas as pd
-data = pd.read_excel("set_28_examples.xlsx")
-data.head()
-"""As you can see, each row corresponds to a question related to a table.
-* The `position` column identifies whether the question is the first, second, ... in a sequence of questions related to a table.
-* The `table_file` column identifies the name of the table file, which refers to a CSV file in the `table_csv` directory.
-* The `answer_coordinates` and `answer_text` columns indicate the answer to the question. The `answer_coordinates` is a list of tuples, each tuple being a (row_index, column_index) pair. The `answer_text` column is a list of strings, indicating the cell values.
-However, the `answer_coordinates` and `answer_text` columns are currently not recognized as real Python lists of Python tuples and strings respectively. Let's do that first using the `.literal_eval()`function of the `ast` module:
-"""
-import ast
-def _parse_answer_coordinates(answer_coordinate_str):
-  """Parses the answer_coordinates of a question.
-  Args:
-    answer_coordinate_str: A string representation of a Python list of tuple
-      strings.
-      For example: "['(1, 4)','(1, 3)', ...]"
-  """
-  try:
-    answer_coordinates = []
-    # make a list of strings
-    coords = ast.literal_eval(answer_coordinate_str)
-    # parse each string as a tuple
-    for row_index, column_index in sorted(
-        ast.literal_eval(coord) for coord in coords):
-      answer_coordinates.append((row_index, column_index))
-  except SyntaxError:
-    raise ValueError('Unable to evaluate %s' % answer_coordinate_str)
-  return answer_coordinates
-def _parse_answer_text(answer_text):
-  """Populates the answer_texts field of `answer` by parsing `answer_text`.
-  Args:
-    answer_text: A string representation of a Python list of strings.
-      For example: "[u'test', u'hello', ...]"
-    answer: an Answer object.
-  """
-  try:
-    answer = []
-    for value in ast.literal_eval(answer_text):
-      answer.append(value)
-  except SyntaxError:
-    raise ValueError('Unable to evaluate %s' % answer_text)
-  return answer
-data['answer_coordinates'] = data['answer_coordinates'].apply(lambda coords_str: _parse_answer_coordinates(coords_str))
-data['answer_text'] = data['answer_text'].apply(lambda txt: _parse_answer_text(txt))
-data.head(10)
-"""Let's create a new dataframe that groups questions which are asked in a sequence related to the table. We can do this by adding a `sequence_id` column, which is a combination of the `id` and `annotator` columns:"""
-def get_sequence_id(example_id, annotator):
-  if "-" in str(annotator):
-    raise ValueError('"-" not allowed in annotator.')
-  return f"{example_id}-{annotator}"
-data['sequence_id'] = data.apply(lambda x: get_sequence_id(x.id, x.annotator), axis=1)
-data.head()
-# let's group table-question pairs by sequence id, and remove some columns we don't need
-grouped = data.groupby(by='sequence_id').agg(lambda x: x.tolist())
-grouped = grouped.drop(columns=['id', 'annotator', 'position'])
-grouped['table_file'] = grouped['table_file'].apply(lambda x: x[0])
-grouped.head(10)
-"""Each row in the dataframe above now consists of a **table and one or more questions** which are asked in a **sequence**. Let's visualize the first row, i.e. a table, together with its queries:"""
-# path to the directory containing all csv files
-table_csv_path = "table_csv"
-item = grouped.iloc[0]
-table = pd.read_csv(table_csv_path + item.table_file[9:]).astype(str)
-display(table)
-print("")
-print(item.question)
-"""We can see that there are 3 sequential questions asked related to the contents of the table.
-We can now use `TapasTokenizer` to batch encode this, as follows:
-"""
-import torch
-from transformers import TapasTokenizer
-# initialize the tokenizer
-tokenizer = TapasTokenizer.from_pretrained("google/tapas-base")
-encoding = tokenizer(table=table, queries=item.question, answer_coordinates=item.answer_coordinates, answer_text=item.answer_text,
-                     truncation=True, padding="max_length", return_tensors="pt")
-encoding.keys()
-"""TAPAS basically flattens every table-question pair before feeding it into a BERT like model:"""
-tokenizer.decode(encoding["input_ids"][0])
-"""The `token_type_ids` created here will be of shape (batch_size, sequence_length, 7), as TAPAS uses 7 different token types to encode tabular structure. Let's verify this:"""
-assert encoding["token_type_ids"].shape == (3, 512, 7)
-"""
-One thing we can verify is whether the `prev_label` token type ids are created correctly. These indicate which tokens were (part of) an answer to the previous table-question pair.
-The prev_label token type ids of the first example in a batch must always be zero (since there's no previous table-question pair). Let's verify this:"""
-assert encoding["token_type_ids"][0][:,3].sum() == 0
-"""However, the `prev_label` token type ids of the second table-question pair in the batch must be set to 1 for the tokens which were an answer to the previous (i.e. the first) table question pair in the batch. The answers to the first table-question pair are the following:"""
-print(item.answer_text[0])
-"""So let's now verify whether the `prev_label` ids of the second table-question pair are set correctly:"""
-for id, prev_label in zip (encoding["input_ids"][1], encoding["token_type_ids"][1][:,3]):
-  if id != 0: # we skip padding tokens
-    print(tokenizer.decode([id]), prev_label.item())
-"""This looks OK! Be sure to check this, because the token type ids are critical for the performance of TAPAS.
-Let's create a PyTorch dataset and corresponding dataloader. Note the __getitem__ method here: in order to properly set the prev_labels token types, we must check whether a table-question pair is the first in a sequence or not. In case it is, we can just encode it. In case it isn't, we need to encode it together with the previous table-question pair.
-Note that this is not the most efficient approach, because we're effectively tokenizing each table-question pair twice when applied on the entire dataset (feel free to ping me a more efficient solution).
-"""
-class TableDataset(torch.utils.data.Dataset):
-    def __init__(self, df, tokenizer):
-        self.df = df
-        self.tokenizer = tokenizer
-    def __getitem__(self, idx):
-        item = self.df.iloc[idx]
-        table = pd.read_csv(table_csv_path + item.table_file[9:]).astype(str) # TapasTokenizer expects the table data to be text only
-        if item.position != 0:
-          # use the previous table-question pair to correctly set the prev_labels token type ids
-          previous_item = self.df.iloc[idx-1]
-          encoding = self.tokenizer(table=table,
-                                    queries=[previous_item.question, item.question],
-                                    answer_coordinates=[previous_item.answer_coordinates, item.answer_coordinates],
-                                    answer_text=[previous_item.answer_text, item.answer_text],
-                                    padding="max_length",
-                                    truncation=True,
-                                    return_tensors="pt"
-          )
-          # use encodings of second table-question pair in the batch
-          encoding = {key: val[-1] for key, val in encoding.items()}
-        else:
-          # this means it's the first table-question pair in a sequence
-          encoding = self.tokenizer(table=table,
-                                    queries=item.question,
-                                    answer_coordinates=item.answer_coordinates,
-                                    answer_text=item.answer_text,
-                                    padding="max_length",
-                                    truncation=True,
-                                    return_tensors="pt"
-          )
-          # remove the batch dimension which the tokenizer adds
-          encoding = {key: val.squeeze(0) for key, val in encoding.items()}
-        return encoding
-    def __len__(self):
-        return len(self.df)
-train_dataset = TableDataset(df=data, tokenizer=tokenizer)
-train_dataloader = torch.utils.data.DataLoader(train_dataset, batch_size=2)
-train_dataset[0]["token_type_ids"].shape
-train_dataset[1]["input_ids"].shape
-batch = next(iter(train_dataloader))
-batch["input_ids"].shape
-batch["token_type_ids"].shape
-"""Let's decode the first table-question pair:"""
-tokenizer.decode(batch["input_ids"][0])
-#first example should not have any prev_labels set
-assert batch["token_type_ids"][0][:,3].sum() == 0
-"""Let's decode the second table-question pair and verify some more:"""
-tokenizer.decode(batch["input_ids"][1])
-assert batch["labels"][0].sum() == batch["token_type_ids"][1][:,3].sum()
-print(batch["token_type_ids"][1][:,3].sum())
-for id, prev_label in zip(batch["input_ids"][1], batch["token_type_ids"][1][:,3]):
-  if id != 0:
-    print(tokenizer.decode([id]), prev_label.item())
-"""## Define the model
-Here we initialize the model with a pre-trained base and randomly initialized cell selection head, and move it to the GPU (if available).
-Note that the `google/tapas-base` checkpoint has (by default) an SQA configuration, so we don't need to specify any additional hyperparameters.
-"""
-from transformers import TapasForQuestionAnswering
-model = TapasForQuestionAnswering.from_pretrained("google/tapas-base")
-device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
-model.to(device)
-"""## Training the model
-Let's fine-tune the model in well-known PyTorch fashion:
-"""
-from transformers import AdamW
-optimizer = AdamW(model.parameters(), lr=5e-5)
-for epoch in range(10):  # loop over the dataset multiple times
-   print("Epoch:", epoch)
-   for idx, batch in enumerate(train_dataloader):
-        # get the inputs;
-        input_ids = batch["input_ids"].to(device)
-        attention_mask = batch["attention_mask"].to(device)
-        token_type_ids = batch["token_type_ids"].to(device)
-        labels = batch["labels"].to(device)
-        # zero the parameter gradients
-        optimizer.zero_grad()
-        # forward + backward + optimize
-        outputs = model(input_ids=input_ids, attention_mask=attention_mask, token_type_ids=token_type_ids,
-                       labels=labels)
-        loss = outputs.loss
-        print("Loss:", loss.item())
-        loss.backward()
-        optimizer.step()
-"""## Inference
-As SQA is a bit different due to its conversational nature, we need to run every training example of the a batch one by one through the model (sequentially), overwriting the `prev_labels` token types (which were created by the tokenizer) by the answer predicted by the model. It is based on the [following code](https://github.com/google-research/tapas/blob/f458b6624b8aa75961a0ab78e9847355022940d3/tapas/experiments/prediction_utils.py#L92) from the official implementation:
-"""
-import collections
-import numpy as np
-def compute_prediction_sequence(model, data, device):
-  """Computes predictions using model's answers to the previous questions."""
-  # prepare data
-  input_ids = data["input_ids"].to(device)
-  attention_mask = data["attention_mask"].to(device)
-  token_type_ids = data["token_type_ids"].to(device)
-  all_logits = []
-  prev_answers = None
-  num_batch = data["input_ids"].shape[0]
-  for idx in range(num_batch):
-    if prev_answers is not None:
-        coords_to_answer = prev_answers[idx]
-        # Next, set the label ids predicted by the model
-        prev_label_ids_example = token_type_ids_example[:,3] # shape (seq_len,)
-        model_label_ids = np.zeros_like(prev_label_ids_example.cpu().numpy()) # shape (seq_len,)
-        # for each token in the sequence:
-        token_type_ids_example = token_type_ids[idx] # shape (seq_len, 7)
-        for i in range(model_label_ids.shape[0]):
-          segment_id = token_type_ids_example[:,0].tolist()[i]
-          col_id = token_type_ids_example[:,1].tolist()[i] - 1
-          row_id = token_type_ids_example[:,2].tolist()[i] - 1
-          if row_id >= 0 and col_id >= 0 and segment_id == 1:
-            model_label_ids[i] = int(coords_to_answer[(col_id, row_id)])
-        # set the prev label ids of the example (shape (1, seq_len) )
-        token_type_ids_example[:,3] = torch.from_numpy(model_label_ids).type(torch.long).to(device)
-    prev_answers = {}
-    # get the example
-    input_ids_example = input_ids[idx] # shape (seq_len,)
-    attention_mask_example = attention_mask[idx] # shape (seq_len,)
-    token_type_ids_example = token_type_ids[idx] # shape (seq_len, 7)
-    # forward pass to obtain the logits
-    outputs = model(input_ids=input_ids_example.unsqueeze(0),
-                    attention_mask=attention_mask_example.unsqueeze(0),
-                    token_type_ids=token_type_ids_example.unsqueeze(0))
-    logits = outputs.logits
-    all_logits.append(logits)
-    # convert logits to probabilities (which are of shape (1, seq_len))
-    dist_per_token = torch.distributions.Bernoulli(logits=logits)
-    probabilities = dist_per_token.probs * attention_mask_example.type(torch.float32).to(dist_per_token.probs.device)
-    # Compute average probability per cell, aggregating over tokens.
-    # Dictionary maps coordinates to a list of one or more probabilities
-    coords_to_probs = collections.defaultdict(list)
-    prev_answers = {}
-    for i, p in enumerate(probabilities.squeeze().tolist()):
-      segment_id = token_type_ids_example[:,0].tolist()[i]
-      col = token_type_ids_example[:,1].tolist()[i] - 1
-      row = token_type_ids_example[:,2].tolist()[i] - 1
-      if col >= 0 and row >= 0 and segment_id == 1:
-        coords_to_probs[(col, row)].append(p)
-    # Next, map cell coordinates to 1 or 0 (depending on whether the mean prob of all cell tokens is > 0.5)
-    coords_to_answer = {}
-    for key in coords_to_probs:
-      coords_to_answer[key] = np.array(coords_to_probs[key]).mean() > 0.5
-    prev_answers[idx+1] = coords_to_answer
-  logits_batch = torch.cat(tuple(all_logits), 0)
-  return logits_batch
-data = {'Actors': ["Brad Pitt", "Leonardo Di Caprio", "George Clooney"],
-        'Age': ["56", "45", "59"],
-        'Number of movies': ["87", "53", "69"],
-        'Date of birth': ["7 february 1967", "10 june 1996", "28 november 1967"]}
-queries = ["How many movies has George Clooney played in?", "How old is he?", "What's his date of birth?"]
-table = pd.DataFrame.from_dict(data)
-inputs = tokenizer(table=table, queries=queries, padding='max_length', return_tensors="pt")
-logits = compute_prediction_sequence(model, inputs, device)
-"""Finally, we can use the handy `convert_logits_to_predictions` function of `TapasTokenizer` to convert the logits into predicted coordinates, and print out the result:"""
-predicted_answer_coordinates, = tokenizer.convert_logits_to_predictions(inputs, logits.cpu().detach())
-# handy helper function in case inference on Pandas dataframe
-answers = []
-for coordinates in predicted_answer_coordinates:
-  if len(coordinates) == 1:
-    # only a single cell:
-    answers.append(table.iat[coordinates[0]])
-  else:
-    # multiple cells
-    cell_values = []
-    for coordinate in coordinates:
-      cell_values.append(table.iat[coordinate])
-    answers.append(", ".join(cell_values))
-display(table)
-print("")
-for query, answer in zip(queries, answers):
-  print(query)
-  print("Predicted answer: " + answer)
-"""Note that the results here are not correct, that's obvious since we only trained on 28 examples and tested it on an entire different example. In reality, you should train on the entire dataset. The result of this is the `google/tapas-base-finetuned-sqa` checkpoint.
-## Legacy
-The code below was considered during the creation of this tutorial, but eventually not used.
-"""
-# grouped = data.groupby(data.position)
-# test = grouped.get_group(0)
-# test.index
-def custom_collate_fn(data):
-  """
-  A custom collate function to batch input_ids, attention_mask, token_type_ids and so on of different batch sizes.
-  Args:
-    data:
-      a list of dictionaries (each dictionary is what the __getitem__ method of TableDataset returns)
-  """
-  result = {}
-  for k in data[0].keys():
-      result[k] = torch.cat([x[k] for x in data], dim=0)
-  return result
-class TableDataset(torch.utils.data.Dataset):
-    def __init__(self, df, tokenizer):
-        self.df = df
-        self.tokenizer = tokenizer
-    def __getitem__(self, idx):
-        item = self.df.iloc[idx]
-        table = pd.read_csv(table_csv_path + item.table_file[9:]).astype(str) # TapasTokenizer expects the table data to be text only
-        if item.position != 0:
-          # use the previous table-question pair
-          previous_item = self.df.iloc[idx-1]
-          encoding = self.tokenizer(table=table,
-                                    queries=[previous_item.question, item.question],
-                                    answer_coordinates=[previous_item.answer_coordinates, item.answer_coordinates],
-                                    answer_text=[previous_item.answer_text, item.answer_text],
-                                    padding="max_length",
-                                    truncation=True,
-                                    return_tensors="pt"
-          )
-          # remove the batch dimension which the tokenizer adds
-          encoding = {key: val[-1] for key, val in encoding.items()}
-          #encoding = {key: val.squeeze(0) for key, val in encoding.items()}
-        else:
-          # this means it's the first table-question pair in a sequence
-          encoding = self.tokenizer(table=table,
-                                    queries=item.question,
-                                    answer_coordinates=item.answer_coordinates,
-                                    answer_text=item.answer_text,
-                                    padding="max_length",
-                                    truncation=True,
-                                    return_tensors="pt"
-          )
-        return encoding
-    def __len__(self):
-        return len(self.df)
-train_dataset = TableDataset(df=grouped, tokenizer=tokenizer)
-train_dataloader = torch.utils.data.DataLoader(train_dataset, batch_size=2, collate_fn=custom_collate_fn)