metrics:
- Recall @10 0.438
- MRR @10 0.247
base_model:
- unicamp-dl/mt5-base-mmarco-v2
tags:
- Information Retrieval
- Natural Language Processing
- Question Answering
license: apache-2.0
Urdu-mT5-mmarco: Fine-Tuned mT5 Model for Urdu Information Retrieval
As part of ongoing efforts to make Information Retrieval (IR) more inclusive, this model addresses the needs of low-resource languages, focusing specifically on Urdu. We created this model by translating the MS-Marco dataset into Urdu using the IndicTrans2 model. To establish baseline performance, we initially tested for zero-shot learning for IR in Urdu using the unicamp-dl/mt5-base-mmarco-v2 model and then applied fine-tuning with the mMARCO multilingual IR methodology on the translated dataset.
Model Details
Model Description
- Developed by: Umer Butt
- Model type: MT5ForConditionalGeneration
- Language(s) (NLP): Python/pytorch
Uses
Direct Use
Bias, Risks, and Limitations
Although this model performs well and is state-of-the-art for now. But still this model is finetuned on mmarco model and a translated dataset(which was created using indicTrans2 model). Hence the limitations of those apply here too.
Evaluation
The evaluation was done using the scripts in the pygaggle library. Specifically these files: evaluate_monot5_reranker.py ms_marco_eval.py
Metrics
Following the approach in the mmarco work. The same two metrics were used.
Recal @10 : 0.438 MRR @10 : 0.247
Results
| Model | Name | Data | Recall@10 | MRR@10 | Queries Ranked |
|---|---|---|---|---|---|
| bm25 (k = 1000) | BM25 - Baseline from mmarco paper | English data | 0.391 | 0.187 | 6980 |
| unicamp-dl/mt5-base-mmarco-v2 | mmarco reranker - Baseline from paper | English data | 0.370 | 6980 | |
| bm25 (k = 1000) | BM25 | Urdu data | 0.2675 | 0.129 | 6980 |
| unicamp-dl/mt5-base-mmarco-v2 | Zero-shot mmarco | Urdu data | 0.408 | 0.204 | 6980 |
| This work | Mavkif/urdu-mt5-mmarco | Urdu data | 0.438 | 0.247 | 6980 |
Model Architecture and Objective
{ "_name_or_path": "unicamp-dl/mt5-base-mmarco-v2", "architectures": ["MT5ForConditionalGeneration"], "d_model": 768, "num_heads": 12, "num_layers": 12, "dropout_rate": 0.1, "vocab_size": 250112, "model_type": "mt5", "transformers_version": "4.38.2" } For more details on how to customize the decoding parameters (such as max_length, num_beams, and early_stopping), refer to the Hugging Face documentation.
How to Get Started with the Model
Example Code for Scoring Query-Document Pairs: In an IR setting, you provide a query and one or more candidate documents. The model scores each document for relevance to the query, which can be used for ranking.
from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
import torch
import torch.nn.functional as F
# Load the tokenizer and model
tokenizer = AutoTokenizer.from_pretrained("Mavkif/urdu-mt5-mmarco")
model = AutoModelForSeq2SeqLM.from_pretrained("Mavkif/urdu-mt5-mmarco")
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model.to(device)
def rank_documents(query, documents):
# Create input pairs of query and documents
query_document_pairs = [f"{query} [SEP] {doc}" for doc in documents]
# Tokenize the input pairs
inputs = tokenizer(query_document_pairs, padding=True, truncation=True, return_tensors="pt", max_length=512)
inputs = {k: v.to(device) for k, v in inputs.items()}
# Generate decoder input ids (starting with the decoder start token)
decoder_input_ids = torch.full(
(inputs["input_ids"].shape[0], 1), model.config.decoder_start_token_id, dtype=torch.long, device=device
)
# Perform inference to get the logits
with torch.no_grad():
outputs = model(**inputs, decoder_input_ids=decoder_input_ids)
# Get the logits for the sequence output
logits = outputs.logits
# Extract the probabilities for the generated sequence
scores = []
for idx, doc in enumerate(documents):
# Calculate the softmax over the entire vocabulary for each token in the sequence
doc_logits = logits[idx]
doc_probs = F.softmax(doc_logits, dim=-1)
# Get the probability score for "ہاں" token in the output sequence
token_true_id = tokenizer.convert_tokens_to_ids("ہاں")
token_probs = doc_probs[:, token_true_id]
sum_prob = token_probs.sum().item() # Sum probability over the sequence
scores.append((doc, sum_prob)) # Use the summed probability directly as the score
# Normalize scores to be between 0 and 1
max_score = max(score for _, score in scores)
min_score = min(score for _, score in scores)
normalized_scores = [((score - min_score) / (max_score - min_score) if max_score > min_score else 0.5) for _, score in scores]
# Create a list of documents with normalized scores
ranked_documents = [(documents[idx], normalized_scores[idx]) for idx in range(len(documents))]
# Sort documents based on scores (descending order)
ranked_documents = sorted(ranked_documents, key=lambda x: x[1], reverse=True)
return ranked_documents
# Example query and documents
query = "پاکستان کی معیشت کی موجودہ صورتحال کیا ہے؟"
documents = [
"پاکستان کی معیشت میں بہتری کے اشارے ہیں۔",
"زر مبادلہ کے ذخائر میں کمی دیکھی گئی ہے۔",
"فٹبال پاکستان میں تیزی سے مقبول ہو رہا ہے۔"
]
# Get ranked documents
ranked_docs = rank_documents(query, documents)
# Print the ranked documents
for idx, (doc, score) in enumerate(ranked_docs):
print(f"Rank {idx + 1}: Score: {score}, Document: {doc}")
Rank 1: Score: 1.0, Document: پاکستان کی معیشت میں بہتری کے اشارے ہیں۔
Rank 2: Score: 0.547, Document: فٹبال پاکستان میں تیزی سے مقبول ہو رہا ہے۔
Rank 3: Score: 0.0, Document: زر مبادلہ کے ذخائر میں کمی دیکھی گئی ہے۔
Model Card Authors [optional]
Umer Butt