1678 145 58

Stefan Schweter PRO

stefan-it

AI & ML interests

Flair Library, NER & PoS Tagging, LM Pretraining (mostly encoder-only), Historical Language Models

Recent Activity

updated a model about 10 hours ago

stefan-it/bert5urk

reacted to davanstrien's post with 🔥 2 days ago

🌍 Big step for multilingual AI data! The Hugging Face community has rated educational content in languages spoken by 1.6 billion people! New additions: • Japanese • Italian • Old High German Learn more and contribute: https://huggingface.co/blog/davanstrien/fineweb2-community These ratings can help enhance training data for major world languages.

upvoted a paper 3 days ago

Analyzing the Effect of Linguistic Similarity on Cross-Lingual Transfer: Tasks and Experimental Setups Matter

View all activity

Articles

Fine-tune Flair Models on NER Dataset with 🤗 AutoTrain SpaceRunner

Oct 20, 2023

Organizations

Posts 1

Post

1401

My latest project is the outcome of the last 2+ years working with TPUs from the amazing TPU Research Cloud (TRC) program and training Encoder-only LMs with the TensorFlow Model Garden library.

👉 Link: https://github.com/stefan-it/model-garden-lms

An overview of some features:

- Cheatsheet for setting-up a TPU VM Pod (with all necessary dependencies) to pretrain LMs with TF Model Garden
- Conversion scripts that convert TF Model Garden weights to Hugging Face Transformers-compatible models
- Supported architectures include BERT, BERT with Token Dropping and TEAMS

I also released BERT-based models pretrained on the great Hugging Face FineWeb and FineWeb-Edu datasets (10BT subset). With more to come!

👉 Model Hub Link: https://huggingface.co/model-garden-lms

If you find these resources useful, please give them a like!

Made from Bavarian Oberland with ❤️ and 🥨.