BERTopic_ArXiv

This is a BERTopic model. BERTopic is a flexible and modular topic modeling framework that allows for the generation of easily interpretable topics from large datasets.

This pre-trained model demonstrates the use of several representation models that can be used within BERTopic. This model was trained on ~30000 ArXiv abstracts with the following topic representation methods (bertopic.representation):

• POS
• KeyBERTInspired
• MaximalMarginalRelevance
• KeyBERT + MaximalMarginalRelevance
• ChatGPT labels
• ChatGPT summaries

An example of the default c-TF-IDF representations:

An example of labels generated by ChatGPT (gpt-3.5-turbo):

To generate these images, you can follow along with this tutorial:

Usage

To use this model, please install BERTopic:

pip install -U bertopic
pip install -U safetensors

You can use the model as follows:

from bertopic import BERTopic
topic_model = BERTopic.load("MaartenGr/BERTopic_ArXiv")

topic_model.get_topic_info()

To view all different topic representations (keywords, labels, summary, etc.) you can run the following:

>>> topic_model.get_topic(0, full=True)
{'Main': [['dialogue', 0.02704485163341523],
  ['dialog', 0.01677038224466311],
  ['response', 0.011692640237477233],
  ['responses', 0.01002788412923778],
  ['intent', 0.00990720856306287],
  ['oriented', 0.009217253131615378],
  ['slot', 0.009177118721490055],
  ['conversational', 0.009129311385144046],
  ['systems', 0.009101146153425574],
  ['conversation', 0.008845392252307181]],
 'POS': [['dialogue', 0.02704485163341523],
  ['dialog', 0.01677038224466311],
  ['response', 0.011692640237477233],
  ['responses', 0.01002788412923778],
  ['intent', 0.00990720856306287],
  ['slot', 0.009177118721490055],
  ['conversational', 0.009129311385144046],
  ['systems', 0.009101146153425574],
  ['conversation', 0.008845392252307181],
  ['user', 0.008753551043296965]],
 'KeyBERTInspired': [['task oriented dialogue', 0.6559894680976868],
  ['dialogue systems', 0.6249060034751892],
  ['oriented dialogue', 0.5788208246231079],
  ['dialog systems', 0.530449628829956],
  ['dialogue state', 0.5167528390884399],
  ['response generation', 0.5143576860427856],
  ['spoken language understanding', 0.46739083528518677],
  ['oriented dialog', 0.4600704610347748],
  ['dialog', 0.4534587264060974],
  ['dialogues', 0.44082391262054443]],
 'MMR': [['dialogue', 0.02704485163341523],
  ['dialog', 0.01677038224466311],
  ['response', 0.011692640237477233],
  ['responses', 0.01002788412923778],
  ['intent', 0.00990720856306287],
  ['oriented', 0.009217253131615378],
  ['slot', 0.009177118721490055],
  ['conversational', 0.009129311385144046],
  ['systems', 0.009101146153425574],
  ['conversation', 0.008845392252307181]],
 'KeyBERT + MMR': [['task oriented dialogue', 0.6559894680976868],
  ['dialogue systems', 0.6249060034751892],
  ['oriented dialogue', 0.5788208246231079],
  ['dialog systems', 0.530449628829956],
  ['dialogue state', 0.5167528390884399],
  ['response generation', 0.5143576860427856],
  ['spoken language understanding', 0.46739083528518677],
  ['oriented dialog', 0.4600704610347748],
  ['dialog', 0.4534587264060974],
  ['dialogues', 0.44082391262054443]],
 'OpenAI_Label': [['Challenges and Approaches in Developing Task-oriented Dialogue Systems',
   1]],
 'OpenAI_Summary': [['Task-oriented dialogue systems and their components, such as dialogue policy, natural language understanding, dialogue state tracking, response generation, and end-to-end training using neural networks. These components are crucial in assisting users to complete various activities such as booking tickets and restaurant reservations through spoken language understanding dialogue. The challenge lies in tracking dialogue states of multiple domains and obtaining annotations for training. Effective SLU is achieved by utilizing context from the prior dialogue history.',
   1]]}

Topic overview

• Number of topics: 107
• Number of training documents: 33189

Click here for an overview of all topics.

Topic ID	Topic Keywords	Topic Frequency	Label
-1	language - models - model - data - based	20	-1_language_models_model_data
0	dialogue - dialog - response - responses - intent	14247	0_dialogue_dialog_response_responses
1	speech - asr - speech recognition - recognition - end	1833	1_speech_asr_speech recognition_recognition
2	tuning - tasks - prompt - models - language	1369	2_tuning_tasks_prompt_models
3	summarization - summaries - summary - abstractive - document	1109	3_summarization_summaries_summary_abstractive
4	question - answer - qa - answering - question answering	893	4_question_answer_qa_answering
5	sentiment - sentiment analysis - aspect - analysis - opinion	837	5_sentiment_sentiment analysis_aspect_analysis
6	clinical - medical - biomedical - notes - patient	691	6_clinical_medical_biomedical_notes
7	translation - nmt - machine translation - neural machine - neural machine translation	586	7_translation_nmt_machine translation_neural machine
8	generation - text generation - text - language generation - nlg	558	8_generation_text generation_text_language generation
9	hate - hate speech - offensive - speech - detection	484	9_hate_hate speech_offensive_speech
10	news - fake - fake news - stance - fact	455	10_news_fake_fake news_stance
11	relation - relation extraction - extraction - relations - entity	450	11_relation_relation extraction_extraction_relations
12	ner - named - named entity - entity - named entity recognition	376	12_ner_named_named entity_entity
13	parsing - parser - dependency - treebank - parsers	370	13_parsing_parser_dependency_treebank
14	event - temporal - events - event extraction - extraction	314	14_event_temporal_events_event extraction
15	emotion - emotions - multimodal - emotion recognition - emotional	300	15_emotion_emotions_multimodal_emotion recognition
16	word - embeddings - word embeddings - embedding - words	292	16_word_embeddings_word embeddings_embedding
17	explanations - explanation - rationales - rationale - interpretability	212	17_explanations_explanation_rationales_rationale
18	morphological - arabic - morphology - languages - inflection	204	18_morphological_arabic_morphology_languages
19	topic - topics - topic models - lda - topic modeling	200	19_topic_topics_topic models_lda
20	bias - gender - biases - gender bias - debiasing	195	20_bias_gender_biases_gender bias
21	law - frequency - zipf - words - length	185	21_law_frequency_zipf_words
22	legal - court - law - legal domain - case	182	22_legal_court_law_legal domain
23	adversarial - attacks - attack - adversarial examples - robustness	181	23_adversarial_attacks_attack_adversarial examples
24	commonsense - commonsense knowledge - reasoning - knowledge - commonsense reasoning	180	24_commonsense_commonsense knowledge_reasoning_knowledge
25	quantum - semantics - calculus - compositional - meaning	171	25_quantum_semantics_calculus_compositional
26	correction - error - error correction - grammatical - grammatical error	161	26_correction_error_error correction_grammatical
27	argument - arguments - argumentation - argumentative - mining	160	27_argument_arguments_argumentation_argumentative
28	sarcasm - humor - sarcastic - detection - humorous	157	28_sarcasm_humor_sarcastic_detection
29	coreference - resolution - coreference resolution - mentions - mention	156	29_coreference_resolution_coreference resolution_mentions
30	sense - word sense - wsd - word - disambiguation	153	30_sense_word sense_wsd_word
31	knowledge - knowledge graph - graph - link prediction - entities	149	31_knowledge_knowledge graph_graph_link prediction
32	parsing - semantic parsing - amr - semantic - parser	146	32_parsing_semantic parsing_amr_semantic
33	cross lingual - lingual - cross - transfer - languages	146	33_cross lingual_lingual_cross_transfer
34	mt - translation - qe - quality - machine translation	139	34_mt_translation_qe_quality
35	sql - text sql - queries - spider - schema	138	35_sql_text sql_queries_spider
36	classification - text classification - label - text - labels	136	36_classification_text classification_label_text
37	style - style transfer - transfer - text style - text style transfer	136	37_style_style transfer_transfer_text style
38	question - question generation - questions - answer - generation	129	38_question_question generation_questions_answer
39	authorship - authorship attribution - attribution - author - authors	127	39_authorship_authorship attribution_attribution_author
40	sentence - sentence embeddings - similarity - sts - sentence embedding	123	40_sentence_sentence embeddings_similarity_sts
41	code - identification - switching - cs - code switching	121	41_code_identification_switching_cs
42	story - stories - story generation - generation - storytelling	118	42_story_stories_story generation_generation
43	discourse - discourse relation - discourse relations - rst - discourse parsing	117	43_discourse_discourse relation_discourse relations_rst
44	code - programming - source code - code generation - programming languages	117	44_code_programming_source code_code generation
45	paraphrase - paraphrases - paraphrase generation - paraphrasing - generation	114	45_paraphrase_paraphrases_paraphrase generation_paraphrasing
46	agent - games - environment - instructions - agents	111	46_agent_games_environment_instructions
47	covid - covid 19 - 19 - tweets - pandemic	108	47_covid_covid 19_19_tweets
48	linking - entity linking - entity - el - entities	107	48_linking_entity linking_entity_el
49	poetry - poems - lyrics - poem - music	103	49_poetry_poems_lyrics_poem
50	image - captioning - captions - visual - caption	100	50_image_captioning_captions_visual
51	nli - entailment - inference - natural language inference - language inference	96	51_nli_entailment_inference_natural language inference
52	keyphrase - keyphrases - extraction - document - phrases	95	52_keyphrase_keyphrases_extraction_document
53	simplification - text simplification - ts - sentence - simplified	95	53_simplification_text simplification_ts_sentence
54	empathetic - emotion - emotional - empathy - emotions	95	54_empathetic_emotion_emotional_empathy
55	depression - mental - health - mental health - social media	93	55_depression_mental_health_mental health
56	segmentation - word segmentation - chinese - chinese word segmentation - chinese word	93	56_segmentation_word segmentation_chinese_chinese word segmentation
57	citation - scientific - papers - citations - scholarly	85	57_citation_scientific_papers_citations
58	agreement - syntactic - verb - grammatical - subject verb	85	58_agreement_syntactic_verb_grammatical
59	metaphor - literal - figurative - metaphors - idiomatic	83	59_metaphor_literal_figurative_metaphors
60	srl - semantic role - role labeling - semantic role labeling - role	82	60_srl_semantic role_role labeling_semantic role labeling
61	privacy - private - federated - privacy preserving - federated learning	82	61_privacy_private_federated_privacy preserving
62	change - semantic change - time - semantic - lexical semantic	82	62_change_semantic change_time_semantic
63	bilingual - lingual - cross lingual - cross - embeddings	80	63_bilingual_lingual_cross lingual_cross
64	political - media - news - bias - articles	77	64_political_media_news_bias
65	medical - qa - question - questions - clinical	75	65_medical_qa_question_questions
66	math - mathematical - math word - word problems - problems	73	66_math_mathematical_math word_word problems
67	financial - stock - market - price - news	69	67_financial_stock_market_price
68	table - tables - tabular - reasoning - qa	69	68_table_tables_tabular_reasoning
69	readability - complexity - assessment - features - reading	65	69_readability_complexity_assessment_features
70	layout - document - documents - document understanding - extraction	64	70_layout_document_documents_document understanding
71	brain - cognitive - reading - syntactic - language	62	71_brain_cognitive_reading_syntactic
72	sign - gloss - language - signed - language translation	61	72_sign_gloss_language_signed
73	vqa - visual - visual question - visual question answering - question	59	73_vqa_visual_visual question_visual question answering
74	biased - biases - spurious - nlp - debiasing	57	74_biased_biases_spurious_nlp
75	visual - dialogue - multimodal - image - dialog	55	75_visual_dialogue_multimodal_image
76	translation - machine translation - machine - smt - statistical	54	76_translation_machine translation_machine_smt
77	multimodal - visual - image - translation - machine translation	52	77_multimodal_visual_image_translation
78	geographic - location - geolocation - geo - locations	51	78_geographic_location_geolocation_geo
79	reasoning - prompting - llms - chain thought - chain	48	79_reasoning_prompting_llms_chain thought
80	essay - scoring - aes - essay scoring - essays	45	80_essay_scoring_aes_essay scoring
81	crisis - disaster - traffic - tweets - disasters	45	81_crisis_disaster_traffic_tweets
82	graph - text classification - text - gcn - classification	44	82_graph_text classification_text_gcn
83	annotation - tools - linguistic - resources - xml	43	83_annotation_tools_linguistic_resources
84	entity alignment - alignment - kgs - entity - ea	43	84_entity alignment_alignment_kgs_entity
85	personality - traits - personality traits - evaluative - text	42	85_personality_traits_personality traits_evaluative
86	ad - alzheimer - alzheimer disease - disease - speech	40	86_ad_alzheimer_alzheimer disease_disease
87	taxonomy - hypernymy - taxonomies - hypernym - hypernyms	39	87_taxonomy_hypernymy_taxonomies_hypernym
88	active learning - active - al - learning - uncertainty	37	88_active learning_active_al_learning
89	reviews - summaries - summarization - review - opinion	36	89_reviews_summaries_summarization_review
90	emoji - emojis - sentiment - message - anonymous	35	90_emoji_emojis_sentiment_message
91	table - table text - tables - table text generation - text generation	35	91_table_table text_tables_table text generation
92	domain - domain adaptation - adaptation - domains - source	35	92_domain_domain adaptation_adaptation_domains
93	alignment - word alignment - parallel - pairs - alignments	34	93_alignment_word alignment_parallel_pairs
94	indo - languages - indo european - names - family	34	94_indo_languages_indo european_names
95	patent - claim - claim generation - chemical - technical	32	95_patent_claim_claim generation_chemical
96	agents - emergent - communication - referential - games	32	96_agents_emergent_communication_referential
97	graph - amr - graph text - graphs - text generation	31	97_graph_amr_graph text_graphs
98	moral - ethical - norms - values - social	29	98_moral_ethical_norms_values
99	acronym - acronyms - abbreviations - abbreviation - disambiguation	27	99_acronym_acronyms_abbreviations_abbreviation
100	typing - entity typing - entity - type - types	27	100_typing_entity typing_entity_type
101	coherence - discourse - discourse coherence - coherence modeling - text	26	101_coherence_discourse_discourse coherence_coherence modeling
102	pos - taggers - tagging - tagger - pos tagging	25	102_pos_taggers_tagging_tagger
103	drug - social - social media - media - health	25	103_drug_social_social media_media
104	gender - translation - bias - gender bias - mt	24	104_gender_translation_bias_gender bias
105	job - resume - skills - skill - soft	21	105_job_resume_skills_skill

Training Procedure

The model was trained as follows:

from cuml.manifold import UMAP
from cuml.cluster import HDBSCAN
from bertopic import BERTopic
from sklearn.feature_extraction.text import CountVectorizer
from bertopic.representation import PartOfSpeech, KeyBERTInspired, MaximalMarginalRelevance, OpenAI

# Prepare sub-models
embedding_model = SentenceTransformer('sentence-transformers/all-mpnet-base-v2')
umap_model = UMAP(n_components=5, n_neighbors=50, random_state=42, metric="cosine", verbose=True)
hdbscan_model = HDBSCAN(min_samples=20, gen_min_span_tree=True, prediction_data=False, min_cluster_size=20, verbose=True)
vectorizer_model = CountVectorizer(stop_words="english", ngram_range=(1, 3), min_df=5)

# Summarization with ChatGPT
summarization_prompt = """
I have a topic that is described by the following keywords: [KEYWORDS]
In this topic, the following documents are a small but representative subset of all documents in the topic:
[DOCUMENTS]

Based on the information above, please give a description of this topic in the following format:
topic: <description>
"""
summarization_model = OpenAI(model="gpt-3.5-turbo", chat=True, prompt=summarization_prompt, nr_docs=5, exponential_backoff=True, diversity=0.1)

# Representation models
representation_models = {
    "POS": PartOfSpeech("en_core_web_lg"),
    "KeyBERTInspired": KeyBERTInspired(),
    "MMR": MaximalMarginalRelevance(diversity=0.3),
    "KeyBERT + MMR": [KeyBERTInspired(), MaximalMarginalRelevance(diversity=0.3)],
    "OpenAI_Label": OpenAI(model="gpt-3.5-turbo", exponential_backoff=True, chat=True, diversity=0.1),
    "OpenAI_Summary": [KeyBERTInspired(), summarization_model],
}

# Fit BERTopic
topic_model= BERTopic(
        embedding_model=embedding_model,
        umap_model=umap_model,
        hdbscan_model=hdbscan_model,
        vectorizer_model=vectorizer_model,
        representation_model=representation_models,
        verbose=True
).fit(docs)

Training hyperparameters

• calculate_probabilities: False
• language: None
• low_memory: False
• min_topic_size: 10
• n_gram_range: (1, 1)
• nr_topics: None
• seed_topic_list: None
• top_n_words: 10
• verbose: True

Framework versions

• Numpy: 1.22.4
• HDBSCAN: 0.8.29
• UMAP: 0.5.3
• Pandas: 1.5.3
• Scikit-Learn: 1.2.2
• Sentence-transformers: 2.2.2
• Transformers: 4.29.2
• Numba: 0.56.4
• Plotly: 5.13.1
• Python: 3.10.11