### Putting it all together You can use Doc2Query or Doc2Query-- in an indexing pipeline to build an index of the expanded documents: <div class="pipeline"> <div class="df" title="Document Frame">D</div> <div class="transformer attn" title="Doc2Query or Doc2Query−− Transformer">Doc2Query[−−]</div> <div class="df" title="Document Frame">D</div> <div class="transformer" title="Indexer">Indexer</div> <div class="artefact" title="Doc2Query Index">IDX</div> </div> ```python import pyterrier as pt pt.init() import pyterrier_doc2query doc2query = pyterrier_doc2query.Doc2Query(append=True) dataset = pt.get_dataset('irds:msmarco-passage') indexer = pt.IterDictIndexer('./msmarco_psg') indxer_pipe = doc2query >> indexer indxer_pipe.index(dataset.get_corpus_iter()) ``` Once you built an index, you can retrieve from it using any retrieval function (often BM25): <div class="pipeline"> <div class="df" title="Query Frame">Q</div> <div class="transformer" title="BM25 Transformer">BM25 Retriever <div class="artefact" title="Doc2Query Index">IDX</div></div> <div class="df" title="Result Frame">R</div> </div> ```python bm25 = pt.BatchRetrieve('./msmarco_psg', wmodel="BM25") ``` ### References & Credits - Rodrigo Nogueira and Jimmy Lin. [From doc2query to docTTTTTquery](https://cs.uwaterloo.ca/~jimmylin/publications/Nogueira_Lin_2019_docTTTTTquery-v2.pdf). - Mitko Gospodinov, Sean MacAvaney, and Craig Macdonald. Doc2Query--: When Less is More. ECIR 2023. - Craig Macdonald, Nicola Tonellotto, Sean MacAvaney, Iadh Ounis. [PyTerrier: Declarative Experimentation in Python from BM25 to Dense Retrieval](https://dl.acm.org/doi/abs/10.1145/3459637.3482013). CIKM 2021.