Marissa commited on
Commit
baf2144
1 Parent(s): 8f4b3fc

Add model card

Browse files

Hi! This PR has a preliminary model card, based on the format we are using as part of our effort to standardize model cards at Hugging Face. It is very similar to the proposed model card at https://huggingface.co/facebook/dpr-reader-single-nq-base/discussions/1

Feel free to merge if you are ok with the changes! (cc

@Ezi



@Meg



@Nazneen

)

Files changed (1) hide show
  1. README.md +152 -0
README.md ADDED
@@ -0,0 +1,152 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ language: en
3
+ license: cc-by-nc-4.0
4
+ tags:
5
+ - dpr
6
+ datasets:
7
+ - nq_open
8
+ - trivia_qa
9
+ - web_questions
10
+ - trec
11
+ ---
12
+
13
+ # `dpr-reader-multiset-base`
14
+
15
+ ## Table of Contents
16
+ - [Model Details](#model-details)
17
+ - [How To Get Started With the Model](#how-to-get-started-with-the-model)
18
+ - [Uses](#uses)
19
+ - [Risks, Limitations and Biases](#risks-limitations-and-biases)
20
+ - [Training](#training)
21
+ - [Evaluation](#evaluation-results)
22
+ - [Environmental Impact](#environmental-impact)
23
+ - [Technical Specifications](#technical-specifications)
24
+ - [Citation Information](#citation-information)
25
+ - [Model Card Authors](#model-card-authors)
26
+
27
+ ## Model Details
28
+
29
+ **Model Description:** [Dense Passage Retrieval (DPR)](https://github.com/facebookresearch/DPR) is a set of tools and models for state-of-the-art open-domain Q&A research. `dpr-reader-multiset-base` is the reader model trained using the [Natural Questions (NQ) dataset](https://huggingface.co/datasets/nq_open), [TriviaQA](https://huggingface.co/datasets/trivia_qa), [WebQuestions (WQ)](https://huggingface.co/datasets/web_questions), and [CuratedTREC (TREC)](https://huggingface.co/datasets/trec).
30
+
31
+ - **Developed by:** See [GitHub repo](https://github.com/facebookresearch/DPR) for model developers
32
+ - **Model Type:** BERT-based encoder
33
+ - **Language(s):** [CC-BY-NC-4.0](https://github.com/facebookresearch/DPR/blob/main/LICENSE), also see [Code of Conduct](https://github.com/facebookresearch/DPR/blob/main/CODE_OF_CONDUCT.md)
34
+ - **License:** English
35
+ - **Related Models:**
36
+ - [`dpr-question_encoder-multiset-base`](https://huggingface.co/facebook/dpr-question_encoder-multiset-base)
37
+ - [`dpr-ctx_encoder-multiset-base`](https://huggingface.co/facebook/dpr-ctx_encoder-multiset-base)
38
+ - [`dpr-question-encoder-single-nq-base`](https://huggingface.co/facebook/dpr-question_encoder-single-nq-base)
39
+ - [`dpr-reader-single-nq-base`](https://huggingface.co/facebook/dpr-reader-single-nq-base)
40
+ - [`dpr-ctx_encoder-single-nq-base`](https://huggingface.co/facebook/dpr-ctx_encoder-single-nq-base)
41
+ - **Resources for more information:**
42
+ - [Research Paper](https://arxiv.org/abs/2004.04906)
43
+ - [GitHub Repo](https://github.com/facebookresearch/DPR)
44
+ - [Hugging Face DPR docs](https://huggingface.co/docs/transformers/main/en/model_doc/dpr)
45
+ - [BERT Base Uncased Model Card](https://huggingface.co/bert-base-uncased)
46
+
47
+ ## How to Get Started with the Model
48
+
49
+ Use the code below to get started with the model.
50
+
51
+ ```python
52
+ from transformers import DPRReader, DPRReaderTokenizer
53
+
54
+ tokenizer = DPRReaderTokenizer.from_pretrained("facebook/dpr-reader-multiset-base")
55
+ model = DPRReader.from_pretrained("facebook/dpr-reader-multiset-base")
56
+ encoded_inputs = tokenizer(
57
+ questions=["What is love ?"],
58
+ titles=["Haddaway"],
59
+ texts=["'What Is Love' is a song recorded by the artist Haddaway"],
60
+ return_tensors="pt",
61
+ )
62
+ outputs = model(**encoded_inputs)
63
+ start_logits = outputs.start_logits
64
+ end_logits = outputs.end_logits
65
+ relevance_logits = outputs.relevance_logits
66
+ ```
67
+
68
+ ## Uses
69
+
70
+ #### Direct Use
71
+
72
+ `dpr-reader-multiset-base`, [`dpr-question_encoder-multiset-base`](https://huggingface.co/facebook/dpr-question_encoder-multiset-base), and [`dpr-ctx_encoder-multiset-base`](https://huggingface.co/facebook/dpr-ctx_encoder-multiset-base) can be used for the task of open-domain question answering.
73
+
74
+ #### Misuse and Out-of-scope Use
75
+
76
+ The model should not be used to intentionally create hostile or alienating environments for people. In addition, the set of DPR models was not trained to be factual or true representations of people or events, and therefore using the models to generate such content is out-of-scope for the abilities of this model.
77
+
78
+ ## Risks, Limitations and Biases
79
+
80
+ **CONTENT WARNING: Readers should be aware this section may contain content that is disturbing, offensive, and can propogate historical and current stereotypes.**
81
+
82
+ Significant research has explored bias and fairness issues with language models (see, e.g., [Sheng et al. (2021)](https://aclanthology.org/2021.acl-long.330.pdf) and [Bender et al. (2021)](https://dl.acm.org/doi/pdf/10.1145/3442188.3445922)). Predictions generated by the model can include disturbing and harmful stereotypes across protected classes; identity characteristics; and sensitive, social, and occupational groups.
83
+
84
+ ## Training
85
+
86
+ #### Training Data
87
+
88
+ This model was trained using the following datasets:
89
+ - **[Natural Questions (NQ) dataset](https://huggingface.co/datasets/nq_open)** ([Lee et al., 2019](https://aclanthology.org/P19-1612/); [Kwiatkowski et al., 2019](https://aclanthology.org/Q19-1026/))
90
+ - **[TriviaQA](https://huggingface.co/datasets/trivia_qa)** ([Joshi et al., 2017](https://aclanthology.org/P17-1147/))
91
+ - **[WebQuestions (WQ)](https://huggingface.co/datasets/web_questions)** ([Berant et al., 2013](https://aclanthology.org/D13-1160/))
92
+ - **[CuratedTREC (TREC)](https://huggingface.co/datasets/trec)** ([Baudiš & Šedivý, 2015](https://www.aminer.cn/pub/599c7953601a182cd263079b/reading-wikipedia-to-answer-open-domain-questions))
93
+
94
+ #### Training Procedure
95
+
96
+ The training procedure is described in the [associated paper](https://arxiv.org/pdf/2004.04906.pdf):
97
+
98
+ > Given a collection of M text passages, the goal of our dense passage retriever (DPR) is to index all the passages in a low-dimensional and continuous space, such that it can retrieve efficiently the top k passages relevant to the input question for the reader at run-time.
99
+
100
+ > Our dense passage retriever (DPR) uses a dense encoder EP(·) which maps any text passage to a d- dimensional real-valued vectors and builds an index for all the M passages that we will use for retrieval. At run-time, DPR applies a different encoder EQ(·) that maps the input question to a d-dimensional vector, and retrieves k passages of which vectors are the closest to the question vector.
101
+
102
+ The authors report that for encoders, they used two independent BERT ([Devlin et al., 2019](https://aclanthology.org/N19-1423/)) networks (base, un-cased) and use FAISS ([Johnson et al., 2017](https://arxiv.org/abs/1702.08734)) during inference time to encode and index passages. See the paper for further details on training, including encoders, inference, positive and negative passages, and in-batch negatives.
103
+
104
+ ## Evaluation
105
+
106
+ The following evaluation information is extracted from the [associated paper](https://arxiv.org/pdf/2004.04906.pdf).
107
+
108
+ #### Testing Data, Factors and Metrics
109
+
110
+ The model developers report the performance of the model on five QA datasets, using the top-k accuracy (k ∈ {20, 100}). The datasets were [NQ](https://huggingface.co/datasets/nq_open), [TriviaQA](https://huggingface.co/datasets/trivia_qa), [WebQuestions (WQ)](https://huggingface.co/datasets/web_questions), [CuratedTREC (TREC)](https://huggingface.co/datasets/trec), and [SQuAD v1.1](https://huggingface.co/datasets/squad).
111
+
112
+ #### Results
113
+
114
+ | | Top 20 | | | | | Top 100| | | | |
115
+ |:----:|:------:|:---------:|:--:|:----:|:-----:|:------:|:---------:|:--:|:----:|:-----:|
116
+ | | NQ | TriviaQA | WQ | TREC | SQuAD | NQ | TriviaQA | WQ | TREC | SQuAD |
117
+ | | 79.4 | 78.8 |75.0| 89.1 | 51.6 | 86.0 | 84.7 |82.9| 93.9 | 67.6 |
118
+
119
+ ## Environmental Impact
120
+
121
+ Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700). We present the hardware type and based on the [associated paper](https://arxiv.org/abs/2004.04906).
122
+
123
+ - **Hardware Type:** 8 32GB GPUs
124
+ - **Hours used:** Unknown
125
+ - **Cloud Provider:** Unknown
126
+ - **Compute Region:** Unknown
127
+ - **Carbon Emitted:** Unknown
128
+
129
+ ## Technical Specifications
130
+
131
+ See the [associated paper](https://arxiv.org/abs/2004.04906) for details on the modeling architecture, objective, compute infrastructure, and training details.
132
+
133
+ ## Citation Information
134
+
135
+ ```bibtex
136
+ @inproceedings{karpukhin-etal-2020-dense,
137
+ title = "Dense Passage Retrieval for Open-Domain Question Answering",
138
+ author = "Karpukhin, Vladimir and Oguz, Barlas and Min, Sewon and Lewis, Patrick and Wu, Ledell and Edunov, Sergey and Chen, Danqi and Yih, Wen-tau",
139
+ booktitle = "Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)",
140
+ month = nov,
141
+ year = "2020",
142
+ address = "Online",
143
+ publisher = "Association for Computational Linguistics",
144
+ url = "https://www.aclweb.org/anthology/2020.emnlp-main.550",
145
+ doi = "10.18653/v1/2020.emnlp-main.550",
146
+ pages = "6769--6781",
147
+ }
148
+ ```
149
+
150
+ ## Model Card Authors
151
+
152
+ This model card was written by the team at Hugging Face.