Update geneformer/tokenizer.py

Update geneformer/tokenizer.py

Add find_packages to automatically include mtl and subpackages in setup.py (#410)

adding tensorboard to setup.py (#409)

tensorboard add to reqs (#407)

dictionaries from parent dir (#405)

remove token dictionary and unpickling from init (#403)

move dict loading to function in eval utils

edit doc formatting

edit docs formatting

add mtl_classifier to docs

add input size tip to instructions

update tokenizer to defaults for 95M models for special token and input size

update instructions to include reminder about token dictionary

pointing dictionaries from the mtl module's init (#397)

Refactored token dictionary loading and encapsulated dictionary (#398)

Refactor: Convert mask_token_id, pad_token_id, and all_special_ids to properties (#395)

sync token_dictionary variable name w/ classifier

update setup with req and manifest with updated filenames

fix imports mtl/eval_utils

allow model_type valid options to take params model_type : {"Pretrained", "GeneClassifier", "CellClassifier", "MTLCellClassifier", "MTLCellClassifier-Quantized"} (#390)

peft>=0.11.1 (#387)

"save_model_without_heads" is redundant (#385)

comment out "def save_model_without_heads(original_model_save_directory)"; redundant for ISP/Emb extractor (#382)

fixed bug related to dynamic ranges in dictionary with 'min' and 'max' value mismatch in optuna suggest fn (#380)

Update README.md

precommit formatting

update with 12L and 20L i4096 gc95M models, multitask and quantiz code

rename for consistency

delete old gene name dict

update to only have gene names as keys in gene_name_id_dict

Add function for summing of Ensembl IDs (#377)

save pval

add typing list import

move dicts to init

add random state to umap

update get_embs with token_gene_dict arg

update refs to get_model_emb_dims

embs_df with all model embeddings (#363)

Add function to get number of model embeddings (#364)

clone embs_i to resolve memory leak in cls embs

update perturber stats to reflect cos sim and emb_extractor to suppress warnings for non-cls

update to account for set of perturbed genes with aggregate_data

update to enable cls emb

update tokenizer to include eos token

Update geneformer/emb_extractor.py (#350)

Upload in_silico_perturber_stats.py (#313)

fix cell state gene embeddings bug (#345)

patch datasets save_to_disk

update kwargs for pretrainer