Sadat07/bert_squad

google-bert/bert-base-uncased Fine-Tuned on SQuAD

bert_squad

Pretrained model on context-based Question Answering using the SQuAD dataset. This model is fine-tuned from the BERT architecture for extracting answers from passages.

Model Description

bert_squad is a transformer-based model trained for context-based question answering tasks. It leverages the pretrained BERT architecture and adapts it for extracting precise answers given a question and a related context. This model uses the Stanford Question Answering Dataset (SQuAD), available via Hugging Face datasets, for training and fine-tuning.

The model was trained using free computational resources, demonstrating its accessibility for educational and small-scale research purposes.

Fine-tuned by: SADAT PARVEJ, RAFIFA BINTE JAHIR

Shared by: SADAT PARVEJ

Language(s) (NLP): ENGLISH

Finetuned from model: https://huggingface.co/google-bert/bert-base-uncased

Training Objective

The model predicts the most relevant span of text in a given passage that answers a specific question. It fine-tunes BERT's ability to analyze context using supervised data from SQuAD.

Performance Benchmarks

Training Loss: 0.477800

Validation Loss: 0.465936

Exact Match (EM): 87.568590%

Intended Uses & Limitations

This model is designed for tasks such as:

Extractive Question Answering Reading comprehension applications Known Limitations:

As BERT is inherently a masked language model (MLM), its original pretraining limits its ability for generative tasks or handling queries outside the SQuAD-style question-answering setup. The model's predictions may be biased or overly reliant on the training dataset, as SQuAD comprises structured and fact-based question-answer pairs.

How to Get Started with the Model

Use the code below to get started with the model.

import torch
from transformers import AutoTokenizer, AutoModelForQuestionAnswering

# Load the model and tokenizer
model_name = "Sadat07/bert_squad"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForQuestionAnswering.from_pretrained(model_name)

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model.to(device)


context = """
The person who invented light was 
Thomas Edison.He was born in 1879.
"""
question = "When did Thomas Edison invent?"


inputs = tokenizer(question, context, return_tensors="pt", truncation=True, max_length=512)
input_ids = inputs["input_ids"].to(device)
attention_mask = inputs["attention_mask"].to(device)


print("Tokenized Input:", tokenizer.decode(input_ids[0]))

# Perform inference
with torch.no_grad():
    outputs = model(input_ids=input_ids, attention_mask=attention_mask)
    start_scores = outputs.start_logits
    end_scores = outputs.end_logits

# Logits
print("Start logits:", start_scores)
print("End logits:", end_scores)

# Get start and end indices
start_idx = torch.argmax(start_scores)
end_idx = torch.argmax(end_scores) + 1

# Decode the answer
if start_idx >= end_idx:
    print("Model did not predict a valid answer. Please check context and question.")
else:
    answer = tokenizer.convert_tokens_to_string(
        tokenizer.convert_ids_to_tokens(input_ids[0][start_idx:end_idx])
    )
    print(f"Question: {question}")
    print(f"Answer: {answer}")
  

Training Details

Step	Training Loss	Validation Loss	Exact Match	Squad F1	Start Accuracy	End Accuracy
100	0.632200	0.811809	84.749290	84.749290	0.847493	0.899243
200	0.751500	0.627198	84.768212	84.768212	0.847682	0.899243
300	0.662600	0.557515	86.244087	86.244087	0.862441	0.899243
400	0.600400	0.567693	86.177862	86.177862	0.861779	0.899243
500	0.613200	0.523546	86.499527	86.499527	0.864995	0.899243
600	0.495200	0.539225	86.565752	86.565752	0.865658	0.899243
700	0.645300	0.552358	85.354778	85.354778	0.853548	0.899243
800	0.499100	0.562317	86.338694	86.338694	0.863387	0.899243
900	0.482800	0.499747	86.811731	86.811731	0.868117	0.899243
1000	0.372800	0.543513	86.972564	86.972564	0.869726	0.900000
1100	0.554000	0.502747	85.969726	85.969726	0.859697	0.894797
1200	0.459800	0.484941	87.019868	87.019868	0.870199	0.900662
1300	0.463600	0.477527	87.407758	87.407758	0.874078	0.899905
1400	0.356800	0.499119	87.549669	87.549669	0.875497	0.901608
1500	0.494200	0.485287	87.549669	87.549669	0.875497	0.901703
1600	0.521100	0.466062	87.284768	87.284768	0.872848	0.899243
1700	0.461200	0.462704	87.540208	87.540208	0.875402	0.901419
1800	0.415700	0.474295	87.691580	87.691580	0.876916	0.901892
1900	0.622900	0.462900	87.417219	87.417219	0.874172	0.901987
2000	0.477800	0.465936	87.568590	87.568590	0.875686	0.901892

Training Data

The model was trained on the SQuAD dataset, a widely used benchmark for context-based question-answering tasks. It consists of passages from Wikipedia and corresponding questions, with human-annotated answers.

During training, the dataset was processed to extract contexts, questions, and answers, ensuring compatibility with the BERT architecture for QA. The training utilized free resources to minimize costs and focus on model efficiency.

Training Procedure

Training Objective
The model was trained with the objective of performing context-based question answering using the SQuAD dataset. The fine-tuning process adapts BERT's masked language model (MLM) architecture for QA tasks by leveraging its ability to encode contextual relationships between the passage, question, and answer.

Optimization
The training utilized the AdamW optimizer with a linear learning rate scheduler and warm-up steps to ensure effective weight updates and prevent overfitting. The training was run for 2000 steps, with early stopping applied based on the validation loss and exact match score.

Hardware and Resources
Training was conducted on free resources, such as Google Colab or equivalent free GPU resources. While this limited the scale, adjustments in batch size and learning rate were optimized to make the training efficient within these constraints.

Unique Features
The model fine-tuning procedure emphasizes efficient learning, leveraging BERT's pre-trained knowledge while adapting it specifically to QA tasks in a resource-constrained environment.

Metrics

Performance was evaluated using the following metrics:

• Exact Match (EM): Measures the percentage of predictions that match the ground-truth answers exactly.
• F1 Score: Assesses the overlap between the predicted and true answers at a token level, balancing precision and recall.
• Start and End Accuracy: Tracks the model’s ability to correctly identify the start and end indices of answers within the context.

Results

The model trained on the SQuAD dataset achieved the following key performance metrics:

Exact Match (EM): Up to 87.69%

F1 Score: Up to 87.69%

Validation Loss: Reduced to 0.46

Start Accuracy: Peaked at 87.69%

End Accuracy: Peaked at 90.19%

Summary

The model, bert_squad, was fine-tuned for context-based question answering using the SQuAD dataset from Hugging Face. Key metrics include an Exact Match (EM) and F1 score of up to 87.69%, demonstrating strong accuracy. Performance benchmarks show consistent improvement in loss and accuracy over 2000 steps, with validation loss reaching as low as 0.46.

The training utilized free resources, leveraging BERT’s robust pretraining, although BERT’s limitation as a Masked Language Model (MLM) remains a consideration. This work highlights the potential for effective question-answering systems built on pre-existing datasets and infrastructure.

Model Architecture and Objective

The model uses BERT, a pre-trained Transformer-based architecture, fine-tuned for context-based question answering tasks. It aims to predict answers based on the given input text and context.

Compute Infrastructure

Hardware

GPU: Tesla P100, NVIDIA T4

Software

Framework: Hugging Face Transformers

Dataset: SQuAD (from Hugging Face)

Other tools: Python, PyTorch

BibTeX:

@misc{bert_squad_finetune,
  title = {BERT Fine-tuned for SQuAD},
  author = {Your Name or Team Name},
  year = {2024},
  url = {https://huggingface.co/your-model-repository}
}

Glossary

Exact Match (EM): A metric measuring the percentage of predictions that match the ground truth exactly.

Masked Language Model (MLM): Pre-training objective for BERT, predicting masked words in input sentences.