metadata

language:
  - en
license: mit
library_name: transformers
tags:
  - batch prompting
  - batch
  - BatchPrompt
  - BatchPrompting
  - GLUE
  - Llama
  - fine-tuned
  - Llama3
  - Llama-3-8B-Instruct
datasets:
  - achandlr/BatchPrompting
metrics:
  - accuracy
pipeline_tag: question-answering

Model Card for Model ID

This model is a fine-tuned version of Llama-3-8B-Instruct on the BatchPrompting dataset, which spans 13 diverse NLP tasks. The model has been fine-tuned to effectively perform batch prompting - answering multiple questions concatenated into a single prompt in one inference pass.

Model Details

Model Description

Developed by: Alex Chandler, Sebastian Joseph
Model type: Large Language Model (Llama-3 variant
Language(s) (NLP): English
License: MIT
Finetuned from model [optional]: Llama-3-8B-Instruct

Model Sources [optional]

Repository: Forthcoming
Paper: Forthcoming
Demo: Forthcoming

Uses

How to Use

Use with transformers See the snippet below for usage with Transformers:

import transformers
import torch

model_id = "achandlr/Llama-3-8B-Instruct-BatchPromptQA"

# Load the model pipeline
pipeline = transformers.pipeline("text-generation", model=model_id)

# Generate text using the pipeline
generated_text = pipeline("Hey how are you doing today?")
print(generated_text)

Direct Use

The model can be used for efficient question-answering on a variety of NLP tasks by concatenating multiple questions into a single prompt. It demonstrates strong generalization to unseen tasks and maintains performance with larger batch sizes compared to the non-fine-tuned model.

Out-of-Scope Use

The model should not be used for tasks that may cause harm or for generating factually incorrect or biased content. Caution should be exercised if using the model for high-stakes decision making.

Bias, Risks, and Limitations

The model may exhibit biases present in its pretraining data or the BatchPrompting dataset. It has not been extensively tested for fairness or potential misuse. Performance may degrade on out-of-distribution examples or tasks very dissimilar to the training data.

Recommendations

Users should be made aware of the model's potential limitations and biases. The model's outputs should be carefully monitored, especially when used for sensitive applications. More testing is needed to fully characterize its capabilities and shortcomings.

How to Get Started with the Model

Use the code below to get started with the model.

[More Information Needed]

Training Details

Training Data

The model was fine-tuned on our BatchPrompting dataset consisting of 13 NLP tasks:

GLUE Benchmark Tasks: A collection of datasets used for evaluating the performance of models on a variety of natural language understanding tasks.
Mathematical Reasoning Datasets:
- GSM8K: Focuses on numerical and logical reasoning challenges.
- GSM8K-Hard: Contains more complex problems from the GSM8K dataset.
CommonsenseQA: Tests the model's commonsense reasoning ability through multiple-choice question answering.
RACE Reading Comprehension Dataset: Consists of passages and questions designed to assess reading comprehension, derived from English exams.

Training Procedure

The model was fine-tuned using the LoRA method.

Training Hyperparameters

Training regime: Forthcoming

Speeds, Sizes, Times [optional]

Forthcoming

[More Information Needed]

Evaluation

Testing Data, Factors & Metrics Evaluation was performed on tasks that were excluded from the training run. Key metrics included accuracy and BatchPrompt error rate (failure to answer a question or conform to the specified format). A table of our results is forthcoming.

Testing Data, Factors & Metrics

Forthcoming

Testing Data

Forthcoming

[More Information Needed]

Metrics

Forthcoming

[More Information Needed]

Results

Forthcoming [More Information Needed]

Summary

Forthcoming

Model Examination [optional]

Forthcoming

[More Information Needed]

Model Architecture and Objective

[More Information Needed]

Compute Infrastructure

[More Information Needed]

Hardware

[More Information Needed]

Software

[More Information Needed]

Citation

Forthcoming

BibTeX:

Forthcoming

APA:

[More Information Needed]

Glossary [optional]

[More Information Needed]

More Information [optional]

[More Information Needed]

Model Card Authors [optional]

[More Information Needed]

Model Card Contact

[More Information Needed]