The Beginners Guide to Cleaning a Dataset
โข
23
Caleb Fahlgren PRO
cfahlgren1
AI & ML interests
None yet
Recent Activity
liked
a Space
about 15 hours ago
lhoestq/duckdb-spreadsheets
posted
an
update
about 22 hours ago
observers ๐ญ - automatically log all OpenAI compatible requests to a dataset๐ฝ
โข supports any OpenAI compatible endpoint ๐ช
โข supports DuckDB, Hugging Face Datasets, and Argilla as stores
> pip install observers
No complex framework. Just a few lines of code to start sending your traces somewhere. Let us know what you think! @davidberenstein1957 and I will continue iterating!
Here's an example dataset that was logged to Hugging Face from Ollama: https://huggingface.co/datasets/cfahlgren1/llama-3.1-awesome-chatgpt-prompts
updated
a dataset
1 day ago
cfahlgren1/llama-3.1-awesome-chatgpt-prompts
Articles
Organizations
cfahlgren1's activity
posted
an
update
about 22 hours ago
Post
348
observers ๐ญ - automatically log all OpenAI compatible requests to a dataset๐ฝ
โข supports any OpenAI compatible endpoint ๐ช
โข supports DuckDB, Hugging Face Datasets, and Argilla as stores
> pip install observers
No complex framework. Just a few lines of code to start sending your traces somewhere. Let us know what you think! @davidberenstein1957 and I will continue iterating!
Here's an example dataset that was logged to Hugging Face from Ollama: cfahlgren1/llama-3.1-awesome-chatgpt-prompts
โข supports any OpenAI compatible endpoint ๐ช
โข supports DuckDB, Hugging Face Datasets, and Argilla as stores
> pip install observers
No complex framework. Just a few lines of code to start sending your traces somewhere. Let us know what you think! @davidberenstein1957 and I will continue iterating!
Here's an example dataset that was logged to Hugging Face from Ollama: cfahlgren1/llama-3.1-awesome-chatgpt-prompts
replied to
their
post
3 days ago
or shakespeare ๐คฃ @not-lain
posted
an
update
3 days ago
Post
778
You can create charts, leaderboards, and filters on top of any Hugging Face dataset in less than a minute
โข ASCII Bar Charts ๐
โข Powered by DuckDB WASM โก
โข Download results to Parquet ๐ฝ
โข Embed and Share results with friends ๐ฌ
Do you have any interesting queries?
โข ASCII Bar Charts ๐
โข Powered by DuckDB WASM โก
โข Download results to Parquet ๐ฝ
โข Embed and Share results with friends ๐ฌ
Do you have any interesting queries?
Reacted to
davanstrien's
post with โค๏ธ
3 days ago
Post
1220
huggingface.co/DIBT is dead! Long live https://huggingface.co/data-is-better-together!
We're working on some very cool projects so we're doing a bit of tidying of the Data is Better Together Hub org ๐ค
replied to
their
post
3 days ago
Heavy is the head that wears the crown
Reacted to
fracapuano's
post with โค๏ธ
3 days ago
Post
961
Sharing what we have built over the course of the weekend at the
@llamameta
hackathon, by Cerebral Valley in London ๐ฌ๐ง ๐
@gabrycina @calebgcc and I competed with 200+ participants and 50+ teams for a 24-hrs sprint centered around hacking for impact! We focused on applications of robotics to those in need of assisted living, moving our focus to enable greater autonomy and accessibility of robotics in everyday life.
complete list of assets ๐
๐ค trained robotics policies
v1:
- fracapuano/moss-pills
- fracapuano/moss-cup
v2:
- fracapuano/meta-grasp
๐ค datasets
v1:
- fracapuano/pills
- fracapuano/cup
v2:
- fracapuano/cupim
You can find a live demo of our submission at: https://x.com/_fracapuano/status/1858102728691458554
If you want to know more about how we collected 100GB+ of data, trained multiple RL-policies using @lerobot and used Llama-3.2 models to handle user interactions and switch between tasks, go ahead and have a look! Also, don't be a stranger, and reach out ๐ฆพ
Our project is fully open-source, for the community (and ourselves, ๐จโ๐ณ) to build! A huge thank you to @cadene for the help (and the robot ๐คญ) - truly feeling these hugs-vibes ๐ค , and to @thomwolf and @clem for sharing our work across
Little extra:
โก๏ธ Our ๐ง EEG waves๐ง -based control of the ๐ฆพrobotic arm๐ฆพ
@gabrycina @calebgcc and I competed with 200+ participants and 50+ teams for a 24-hrs sprint centered around hacking for impact! We focused on applications of robotics to those in need of assisted living, moving our focus to enable greater autonomy and accessibility of robotics in everyday life.
complete list of assets ๐
๐ค trained robotics policies
v1:
- fracapuano/moss-pills
- fracapuano/moss-cup
v2:
- fracapuano/meta-grasp
๐ค datasets
v1:
- fracapuano/pills
- fracapuano/cup
v2:
- fracapuano/cupim
You can find a live demo of our submission at: https://x.com/_fracapuano/status/1858102728691458554
If you want to know more about how we collected 100GB+ of data, trained multiple RL-policies using @lerobot and used Llama-3.2 models to handle user interactions and switch between tasks, go ahead and have a look! Also, don't be a stranger, and reach out ๐ฆพ
Our project is fully open-source, for the community (and ourselves, ๐จโ๐ณ) to build! A huge thank you to @cadene for the help (and the robot ๐คญ) - truly feeling these hugs-vibes ๐ค , and to @thomwolf and @clem for sharing our work across
Little extra:
โก๏ธ Our ๐ง EEG waves๐ง -based control of the ๐ฆพrobotic arm๐ฆพ
Reacted to
LukeNeumann's
post with ๐คฏ
3 days ago
Post
1150
Nine years ago, I uploaded the first 8K resolution video to YouTube and I've been stockpiling 8K footage ever since: https://www.youtube.com/watch?v=sLprVF6d7Ug&t
Should @Overlaiapp release the first open-source 8K video dataset?
Could anyone even fine tune a model with this?๐
Should @Overlaiapp release the first open-source 8K video dataset?
Could anyone even fine tune a model with this?๐
-
- 6 replies
replied to
LukeNeumann's
post
3 days ago
Would be massive! Let us know if you need any help ๐ค
Reacted to
dvilasuero's
post with ๐๐ค
3 days ago
Post
998
@Jesse-marqo
and the Marqo team are killing it on the Hub: top embedding models and datasets!
Here's how to start using their new evaluation dataset for curation and labelling:
1. Deploy Argilla on Spaces: https://huggingface.co/new-space?template=argilla%2Fargilla-template-space
2. Load Marqo/amazon-products-eval with the UI wizard.
3. Start curating!
Here's how to start using their new evaluation dataset for curation and labelling:
1. Deploy Argilla on Spaces: https://huggingface.co/new-space?template=argilla%2Fargilla-template-space
2. Load Marqo/amazon-products-eval with the UI wizard.
3. Start curating!
posted
an
update
3 days ago
Post
720
What rank are you on Hugging Face Top Yappers? ๐ฃ๏ธ
Find your rank here with this link: cfahlgren1/hub-stats
The Top 3:
- @fdaudens
- @singhsidhukuldeep
- @akhaliq
I am at #71 and need to get my numbers up! ๐
Find your rank here with this link: cfahlgren1/hub-stats
The Top 3:
- @fdaudens
- @singhsidhukuldeep
- @akhaliq
I am at #71 and need to get my numbers up! ๐
-
- 4 replies
Reacted to
nyuuzyou's
post with ๐ฅ
5 days ago
Post
908
๐ผ๏ธ Introducing Public Domain Pictures Dataset -
nyuuzyou/publicdomainpictures
Dataset highlights:
- 644,412 public domain images with comprehensive metadata from publicdomainpictures.net
- English language metadata including titles, descriptions, and keywords
- Each entry contains rich metadata including:
- Unique image ID and full-size image URLs
- Detailed titles and descriptions
- Keyword/tag collections
- Creator attribution
- Released to the public domain under Creative Commons Zero (CC0) license
Dataset highlights:
- 644,412 public domain images with comprehensive metadata from publicdomainpictures.net
- English language metadata including titles, descriptions, and keywords
- Each entry contains rich metadata including:
- Unique image ID and full-size image URLs
- Detailed titles and descriptions
- Keyword/tag collections
- Creator attribution
- Released to the public domain under Creative Commons Zero (CC0) license
- 2 replies
posted
an
update
5 days ago
Post
2827
You can clean and format datasets entirely in the browser with a few lines of SQL.
In this post, I replicate the process @mlabonne used to clean the new microsoft/orca-agentinstruct-1M-v1 dataset.
The cleaning process consists of:
- Joining the separate splits together / add split column
- Converting string messages into list of structs
- Removing empty system prompts
https://huggingface.co/blog/cfahlgren1/the-beginners-guide-to-cleaning-a-dataset
Here's his new cleaned dataset: mlabonne/orca-agentinstruct-1M-v1-cleaned
In this post, I replicate the process @mlabonne used to clean the new microsoft/orca-agentinstruct-1M-v1 dataset.
The cleaning process consists of:
- Joining the separate splits together / add split column
- Converting string messages into list of structs
- Removing empty system prompts
https://huggingface.co/blog/cfahlgren1/the-beginners-guide-to-cleaning-a-dataset
Here's his new cleaned dataset: mlabonne/orca-agentinstruct-1M-v1-cleaned
- 1 reply
There is no harm, but if I turn the notification on or off for an unapproved gated repo, an error message will appear.
Hey @John6666 , this should be fixed very soon.
Reacted to
erikkaum's
post with ๐ฅ
5 days ago
Post
1632
A while ago I started experimenting with compiling the Python interpreter to WASM.
To build a secure, fast, and lightweight sandbox for code execution โ ideal for running LLM-generated Python code.
- Send code simply as a POST request
- 1-2ms startup times
Hack away:
https://github.com/ErikKaum/runner
To build a secure, fast, and lightweight sandbox for code execution โ ideal for running LLM-generated Python code.
- Send code simply as a POST request
- 1-2ms startup times
Hack away:
https://github.com/ErikKaum/runner
posted
an
update
8 days ago
Post
2198
Why use Google Drive when you can have:
โข Free storage with generous limits๐
โข Dataset Viewer (Sorting, Filtering, FTS) ๐
โข Third Party Library Support
โข SQL Console ๐ง
โข Security ๐
โข Community, Reach, and Visibility ๐
It's a no brainer!
Check out our post on what you get instantly out of the box when you create a dataset.
https://huggingface.co/blog/researcher-dataset-sharing
โข Free storage with generous limits๐
โข Dataset Viewer (Sorting, Filtering, FTS) ๐
โข Third Party Library Support
โข SQL Console ๐ง
โข Security ๐
โข Community, Reach, and Visibility ๐
It's a no brainer!
Check out our post on what you get instantly out of the box when you create a dataset.
https://huggingface.co/blog/researcher-dataset-sharing
- 1 reply
Yeah for sure! Would be cool to see links to the leaderboards of these to see more than top 5 and see where most of the community is ๐ @maxiw
Maybe like top 100 or top 500 with sql console saved link
This is cool, here's a list of the top most commented posts!
https://huggingface.co/datasets/maxiw/hf-posts/embed/sql-console/_ppKBx4
Qwen2.5 Coder + SQL Console ๐๐ค
wdyt? @victor
Reacted to
m-ric's
post with ๐
9 days ago
Post
3653
๐ง๐ต๐ฒ ๐ป๐ฒ๐
๐ ๐ฏ๐ถ๐ด ๐๐ผ๐ฐ๐ถ๐ฎ๐น ๐ป๐ฒ๐๐๐ผ๐ฟ๐ธ ๐ถ๐ ๐ป๐ผ๐ ๐ฆ, ๐ถ๐'๐ ๐๐๐ฏ ๐ฃ๐ผ๐๐๐! [INSERT STONKS MEME WITH LASER EYES]
See below: I got 105k impressions since regularly posting Hub Posts, coming close to my 275k on Twitter!
โ๏ธ Computed with the great dataset maxiw/hf-posts
โ๏ธ Thanks to Qwen2.5-Coder-32B for showing me how to access dict attributes in a SQL request!
cc @merve who's far in front of me
See below: I got 105k impressions since regularly posting Hub Posts, coming close to my 275k on Twitter!
โ๏ธ Computed with the great dataset maxiw/hf-posts
โ๏ธ Thanks to Qwen2.5-Coder-32B for showing me how to access dict attributes in a SQL request!
cc @merve who's far in front of me
-
- 9 replies