Towards Best Practices for Open Datasets for LLM Training
Paper
•
2501.08365
•
Published
•
39
LeRobot
community
AI & ML interests
State-of-the-art Machine Learning for real-world robotics
Recent Activity
View all activity
lerobot's activity
thomwolf
authored
a
paper
2 days ago
Post
1653
AI agents are coming. But who's in control?
@meg , one of the best researchers in AI ethics, makes a critical point about autonomy: fully autonomous systems carry unknowable risks because they operate on computer logic rather than human logic.
The solution? Build systems that support & assist rather than override human decisions.
I highly recommend reading the blog post written by Meg, @evijit @sasha and @giadap . They define different levels of agent autonomy & provide a values-based analysis of risks, benefits, and uses of AI agents to help you make better decisions.
👉 https://huggingface.co/blog/ethics-soc-7
@meg , one of the best researchers in AI ethics, makes a critical point about autonomy: fully autonomous systems carry unknowable risks because they operate on computer logic rather than human logic.
The solution? Build systems that support & assist rather than override human decisions.
I highly recommend reading the blog post written by Meg, @evijit @sasha and @giadap . They define different levels of agent autonomy & provide a values-based analysis of risks, benefits, and uses of AI agents to help you make better decisions.
👉 https://huggingface.co/blog/ethics-soc-7
Post
2235
🔥 The AI Agent hype is real! This blog post deep dives into everything you need to know before deploying them: from key definitions to practical recommendations. A must-read for anyone building the future of autonomous systems.
📊 Key insight: A clear table breaking down the 5 levels of AI agents - from simple processors to fully autonomous systems. Essential framework for understanding where your agent stands on the autonomy spectrum
⚖️ Deep analysis of 15 core values reveals critical trade-offs: accuracy, privacy, safety, equity & more. The same features that make agents powerful can make them risky. Understanding these trade-offs is crucial for responsible deployment
🎯 6 key recommendations for the road ahead:
- Create rigorous evaluation protocols
- Study societal effects
- Understand ripple effects
- Improve transparency
- Open source can make a positive difference
- Monitor base model evolution
Read the blog post: https://huggingface.co/blog/ethics-soc-7 Brillant work by @meg @evijit @sasha @giadap
📊 Key insight: A clear table breaking down the 5 levels of AI agents - from simple processors to fully autonomous systems. Essential framework for understanding where your agent stands on the autonomy spectrum
⚖️ Deep analysis of 15 core values reveals critical trade-offs: accuracy, privacy, safety, equity & more. The same features that make agents powerful can make them risky. Understanding these trade-offs is crucial for responsible deployment
🎯 6 key recommendations for the road ahead:
- Create rigorous evaluation protocols
- Study societal effects
- Understand ripple effects
- Improve transparency
- Open source can make a positive difference
- Monitor base model evolution
Read the blog post: https://huggingface.co/blog/ethics-soc-7 Brillant work by @meg @evijit @sasha @giadap
BrigitteTousi
posted
an
update
8 days ago
Post
977
Community fine-tuned models are more carbon efficient than the models they are derived from! 🥳🌿
@alozowski @clefourrier @SaylorTwift @albertvillanova evaluated CO₂ emissions associated with model inference for over 3000 models on the Open LLM Leaderboard. Interesting trends and new insights emerged...👀
Blog Post: https://huggingface.co/blog/leaderboard-emissions-analysis
Leaderboard: open-llm-leaderboard/open_llm_leaderboard
@alozowski @clefourrier @SaylorTwift @albertvillanova evaluated CO₂ emissions associated with model inference for over 3000 models on the Open LLM Leaderboard. Interesting trends and new insights emerged...👀
Blog Post: https://huggingface.co/blog/leaderboard-emissions-analysis
Leaderboard: open-llm-leaderboard/open_llm_leaderboard
cadene
updated
a
dataset
18 days ago
mishig
updated
a
Space
26 days ago
Post
1350
🔍 From instruction-following to creative storytelling, dive into 2024's most impactful AI datasets! These gems are shaping everything from scientific research to video understanding.
Check it out: huggingface/open-source-ai-year-in-review-2024
Check it out: huggingface/open-source-ai-year-in-review-2024
Post
1289
🤝 Want to share your AI models while protecting your work? Licenses are key!
Fascinating to see that nearly 60% of models on the Hub use Apache & MIT licenses.
Explore the viz here: huggingface/open-source-ai-year-in-review-2024
Fascinating to see that nearly 60% of models on the Hub use Apache & MIT licenses.
Explore the viz here: huggingface/open-source-ai-year-in-review-2024
Post
1335
Did a fun experiment: What are the main themes emerging from the 100+ Nieman Journalism Lab predictions for 2025?
I used natural language processing to cluster and map them — really helps spot patterns that weren't obvious when reading predictions one by one. So what will shape journalism next year? A lot of AI and US politics (surprise!), but there's also this horizontal axis that spans from industry strategies to deep reflections on how to talk to the public.
Click any dot to explore the original prediction. What themes surprise/interest you the most?
👉 fdaudens/nieman_lab_2025_predictions_visualization
P.s.: I discovered that Nieman Lab's content is under Creative Commons license!
I used natural language processing to cluster and map them — really helps spot patterns that weren't obvious when reading predictions one by one. So what will shape journalism next year? A lot of AI and US politics (surprise!), but there's also this horizontal axis that spans from industry strategies to deep reflections on how to talk to the public.
Click any dot to explore the original prediction. What themes surprise/interest you the most?
👉 fdaudens/nieman_lab_2025_predictions_visualization
P.s.: I discovered that Nieman Lab's content is under Creative Commons license!
Post
690
The #NeurIPS2024 Class: Explore which are the leading research institutions 🎓🔬
huggingface/open-source-ai-year-in-review-2024
huggingface/open-source-ai-year-in-review-2024
Post
1561
Are you at #NeurIPS2024? Check out our cool data visualizations about research papers in the Year in Review!
huggingface/open-source-ai-year-in-review-2024
huggingface/open-source-ai-year-in-review-2024
huggingface/open-source-ai-year-in-review-2024
huggingface/open-source-ai-year-in-review-2024
Post
4868
We are proud to announce
HuggingFaceFW/fineweb-2: A sparkling update to
HuggingFaceFW/fineweb with 1000s of 🗣️languages.
We applied the same data-driven approach that led to SOTA English performance in🍷 FineWeb to thousands of languages.
🥂 FineWeb2 has 8TB of compressed text data and outperforms other multilingual datasets in our experiments.
The dataset is released under the permissive 📜 ODC-By 1.0 license, and the 💻 code to reproduce it and our evaluations is public.
We will very soon announce a big community project, and are working on a 📝 blogpost walking you through the entire dataset creation process. Stay tuned!
In the mean time come ask us question on our chat place: HuggingFaceFW/discussion
H/t @guipenedo @hynky @lvwerra as well as @vsabolcec Bettina Messmer @negar-foroutan and @mjaggi
We applied the same data-driven approach that led to SOTA English performance in🍷 FineWeb to thousands of languages.
🥂 FineWeb2 has 8TB of compressed text data and outperforms other multilingual datasets in our experiments.
The dataset is released under the permissive 📜 ODC-By 1.0 license, and the 💻 code to reproduce it and our evaluations is public.
We will very soon announce a big community project, and are working on a 📝 blogpost walking you through the entire dataset creation process. Stay tuned!
In the mean time come ask us question on our chat place: HuggingFaceFW/discussion
H/t @guipenedo @hynky @lvwerra as well as @vsabolcec Bettina Messmer @negar-foroutan and @mjaggi
- 2 replies
Post
1398
The viz of the day for the Year in review: Network graph showing likes similarity between models.
Instructive to see which models serve as the "nodes" of the Hub!
Check it out: huggingface/open-source-ai-year-in-review-2024
Instructive to see which models serve as the "nodes" of the Hub!
Check it out: huggingface/open-source-ai-year-in-review-2024
Post
1262
Exponentially growing number of open-source AI models over the course of the past 30 months – from a few thousands to over 1 million and more
Interactive data viz: huggingface/open-source-ai-year-in-review-2024
Interactive data viz: huggingface/open-source-ai-year-in-review-2024
Post
345
🎯 New day, new viz!
This teaser barely captures the heat between Meta 🇺🇸, Stability 🇬🇧 & Black Forest Labs 🇩🇪 racing for HF Hub likes. Want to see the full Fast & Furious AI showdown? Check the link below! 🏎️💨
huggingface/open-source-ai-year-in-review-2024
This teaser barely captures the heat between Meta 🇺🇸, Stability 🇬🇧 & Black Forest Labs 🇩🇪 racing for HF Hub likes. Want to see the full Fast & Furious AI showdown? Check the link below! 🏎️💨
huggingface/open-source-ai-year-in-review-2024
Post
1440
Most liked and most downloaded open-source AI models from 2022 to 2024
Interactive viz: https://aiworld.eu/embed/model/model/treemap
Discussion: huggingface/open-source-ai-year-in-review-2024
Interactive viz: https://aiworld.eu/embed/model/model/treemap
Discussion: huggingface/open-source-ai-year-in-review-2024
Post
1081
📈👀 Just dropped: visualization mapping Hugging Face's most liked & downloaded models from 2022 to now. Small models are clearly on the rise - fascinating shift in both likes and download patterns.
Check it out: huggingface/open-source-ai-year-in-review-2024
Check it out: huggingface/open-source-ai-year-in-review-2024
Post
1764
Keeping up with open-source AI in 2024 = overwhelming.
Here's help: We're launching our Year in Review on what actually matters, starting today!
Fresh content dropping daily until year end. Come along for the ride - first piece out now with @clem 's predictions for 2025.
Think of it as your end-of-year AI chocolate calendar.
Kudos to @BrigitteTousi @clefourrier @Wauplin @thomwolf for making it happen. We teamed up with aiworld.eu for awesome visualizations to make this digestible—it's a charm to work with their team.
Check it out: huggingface/open-source-ai-year-in-review-2024
Here's help: We're launching our Year in Review on what actually matters, starting today!
Fresh content dropping daily until year end. Come along for the ride - first piece out now with @clem 's predictions for 2025.
Think of it as your end-of-year AI chocolate calendar.
Kudos to @BrigitteTousi @clefourrier @Wauplin @thomwolf for making it happen. We teamed up with aiworld.eu for awesome visualizations to make this digestible—it's a charm to work with their team.
Check it out: huggingface/open-source-ai-year-in-review-2024
Post
502
Want the best of both worlds? I’m refining my test by combining a deep dive (today: Musk’s xAI rivalry) with shorter links to other news of the day (AI agent funding, healthcare improvements, and more!) in my daily newsletter. Let me know what you think.
https://www.linkedin.com/pulse/ai-news-musks-11b-bet-his-rivalry-altman-florent-daudens-rysoe/
https://www.linkedin.com/pulse/ai-news-musks-11b-bet-his-rivalry-altman-florent-daudens-rysoe/
Post
1069
The rapid progress in small audio models is mind-blowing! 🤯 Just tested OuteTTS v0.2 - cloned my voice from a 10s clip with impressive accuracy and natural prosody.
At 500M parameters, it's efficient enough to run on basic hardware but powerful enough for professional use.
This could transform how we produce audio content for new - think instant translated interviews keeping original voices, or scaled audio article production!
Demo and Model on the Hub: OuteAI/OuteTTS-0.2-500M h/t @reach-vb
At 500M parameters, it's efficient enough to run on basic hardware but powerful enough for professional use.
This could transform how we produce audio content for new - think instant translated interviews keeping original voices, or scaled audio article production!
Demo and Model on the Hub: OuteAI/OuteTTS-0.2-500M h/t @reach-vb
- 3 replies