Spaces:
Running
Running
File size: 68,593 Bytes
e137e27 87a6313 e137e27 4028499 90b43c6 e137e27 aba6489 4cc0103 aba6489 25a9fcb b16daa1 25a9fcb aba6489 25a9fcb 3d4aecc 4cc0103 aba6489 4cc0103 aba6489 4cc0103 aba6489 3d4aecc 25a9fcb aba6489 25a9fcb aba6489 25a9fcb aba6489 b9f2fad aba6489 6475c3e aba6489 6475c3e aba6489 b9f2fad 67e65d7 2dd50ea 25a9fcb 5e5aef1 aba6489 5e5aef1 2dd50ea 5e5aef1 aba6489 6475c3e aba6489 6475c3e aba6489 5e5aef1 2dd50ea 5e5aef1 aba6489 6475c3e aba6489 8825bba aba6489 5e5aef1 2dd50ea 5e5aef1 aba6489 6475c3e aba6489 6475c3e aba6489 5e5aef1 2dd50ea 5e5aef1 aba6489 8825bba aba6489 6475c3e aba6489 8825bba aba6489 5e5aef1 2dd50ea 5e5aef1 aba6489 6475c3e aba6489 8825bba aba6489 5e5aef1 2dd50ea 5e5aef1 aba6489 6475c3e aba6489 6475c3e aba6489 5e5aef1 2dd50ea 5e5aef1 aba6489 6475c3e aba6489 6475c3e aba6489 5e5aef1 2dd50ea 5e5aef1 aba6489 6475c3e aba6489 8825bba aba6489 5e5aef1 2dd50ea 5e5aef1 aba6489 6475c3e aba6489 6475c3e aba6489 5e5aef1 3ed48b8 5e5aef1 2c39f2b 5e5aef1 aba6489 6475c3e aba6489 8825bba aba6489 5e5aef1 2dd50ea 5e5aef1 2c39f2b 5e5aef1 aba6489 6475c3e aba6489 8825bba aba6489 5e5aef1 2dd50ea d4c2068 25a9fcb ee43c81 d4c2068 87a6313 aba6489 f44dec9 aba6489 e3fd33e aba6489 e3fd33e d4c2068 8a16e84 1630e9d 3d4aecc aba6489 6a336ca aba6489 0e12ce8 aba6489 1630e9d aba6489 0e12ce8 aba6489 6a336ca aba6489 1630e9d 3d4aecc 0698fac 3d4aecc 0698fac 1630e9d 24b53c0 8feeca0 b72b785 aba6489 0698fac aba6489 0e12ce8 aba6489 8feeca0 24b53c0 103b5cf aba6489 6a336ca aba6489 c9bd480 aba6489 ee43c81 583d7c5 5094bb7 aba6489 860a948 79041d0 aba6489 79041d0 aba6489 79041d0 aba6489 6a336ca 79041d0 aba6489 79041d0 aba6489 79041d0 aba6489 6a336ca aba6489 c9bd480 aba6489 5094bb7 583d7c5 5094bb7 aba6489 316946b 5094bb7 583d7c5 e3ed423 aba6489 5f4285e aba6489 2c39f2b 316946b e3ed423 583d7c5 5094bb7 aba6489 79041d0 aba6489 6a336ca aba6489 79041d0 aba6489 2c39f2b aba6489 c9bd480 aba6489 5094bb7 583d7c5 5094bb7 aba6489 6a9172e aba6489 6a9172e aba6489 c9bd480 aba6489 5094bb7 583d7c5 5094bb7 aba6489 6a9172e aba6489 913dc7b aba6489 913dc7b aba6489 c9bd480 aba6489 8a16e84 583d7c5 5094bb7 aba6489 6a9172e aba6489 6a9172e aba6489 4cc0103 aba6489 6a336ca aba6489 4cc0103 aba6489 5094bb7 583d7c5 5094bb7 aba6489 6a9172e aba6489 5094bb7 583d7c5 5094bb7 aba6489 6a9172e aba6489 9c1b63e 5672cf7 aba6489 6a9172e aba6489 0e12ce8 aba6489 6a336ca 4cc0103 9c1b63e aba6489 4cc0103 aba6489 c9bd480 aba6489 5094bb7 583d7c5 5094bb7 aba6489 6a9172e 5672cf7 583d7c5 6a9172e 6a336ca aba6489 c9bd480 aba6489 5094bb7 583d7c5 5094bb7 aba6489 6a9172e aba6489 ff67812 666337a ff67812 aba6489 6a336ca aba6489 4cc0103 aba6489 5094bb7 583d7c5 aba6489 0e12ce8 aba6489 5672cf7 6a9172e aba6489 6a336ca aba6489 4cc0103 aba6489 c9bd480 aba6489 8a16e84 583d7c5 5094bb7 aba6489 6a336ca 9c1b63e aba6489 4cc0103 aba6489 4cc0103 aba6489 4cc0103 aba6489 c9bd480 aba6489 5094bb7 24b53c0 aba6489 9f87a47 31b08ca aba6489 31b08ca e137e27 09bef6a e137e27 79041d0 e137e27 5dd5e15 e137e27 79041d0 e137e27 aba6489 9cbd894 aba6489 9cbd894 aba6489 09bef6a aba6489 09bef6a aba6489 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 624 625 626 627 628 629 630 631 632 633 634 635 636 637 638 639 640 641 642 643 644 645 646 647 648 649 650 651 652 653 654 655 656 657 658 659 660 661 662 663 664 665 666 667 668 669 670 671 672 673 674 675 676 677 678 679 680 681 682 683 684 685 686 687 688 689 690 691 692 693 694 695 696 697 698 699 700 701 702 703 704 705 706 707 708 709 710 711 712 713 714 715 716 717 718 719 720 721 722 723 724 725 726 727 728 729 730 731 732 733 734 735 736 737 738 739 740 741 742 743 744 745 746 747 748 749 750 751 752 753 754 755 756 757 758 759 760 761 762 763 764 765 766 767 768 769 770 771 772 773 774 775 776 777 778 779 780 781 782 783 784 785 786 787 788 789 790 791 792 793 794 795 796 797 798 799 800 801 802 803 804 805 806 807 808 809 810 811 812 813 814 815 816 817 818 819 820 821 822 823 824 825 826 827 828 829 830 831 832 833 834 835 836 837 838 839 840 841 842 843 844 845 846 847 848 849 850 851 852 853 854 855 856 857 858 859 860 861 862 863 864 865 866 867 868 869 870 871 872 873 874 875 876 877 878 879 880 881 882 883 884 885 886 887 888 889 890 891 892 893 894 895 896 897 898 899 900 901 902 903 904 905 906 907 908 909 910 911 912 913 914 915 916 917 918 919 920 921 922 923 924 925 926 927 928 929 930 931 932 933 934 935 936 937 938 939 940 941 942 943 944 945 946 947 948 949 950 951 952 953 954 955 956 957 958 959 960 961 962 963 964 965 966 967 968 969 970 971 972 973 974 975 976 977 978 979 980 981 982 983 984 985 986 987 988 989 990 991 992 993 994 995 996 997 998 999 1000 1001 1002 1003 1004 1005 1006 1007 1008 1009 1010 1011 1012 1013 1014 1015 1016 1017 1018 1019 1020 1021 1022 1023 1024 1025 1026 1027 1028 1029 1030 1031 1032 1033 1034 1035 1036 1037 1038 1039 1040 1041 1042 1043 1044 1045 1046 1047 1048 1049 1050 1051 1052 1053 1054 1055 1056 1057 1058 1059 1060 1061 1062 1063 1064 1065 1066 1067 1068 1069 1070 1071 1072 1073 1074 1075 1076 1077 1078 1079 1080 1081 1082 1083 1084 1085 1086 1087 1088 1089 1090 1091 1092 1093 1094 1095 1096 1097 1098 1099 1100 1101 1102 1103 1104 1105 1106 1107 1108 1109 1110 1111 1112 1113 1114 1115 1116 1117 1118 1119 1120 1121 1122 1123 1124 1125 1126 1127 1128 1129 1130 1131 1132 1133 1134 1135 1136 1137 1138 1139 1140 1141 1142 1143 1144 1145 1146 1147 1148 1149 1150 1151 1152 1153 1154 1155 1156 1157 1158 1159 1160 1161 1162 1163 1164 1165 1166 1167 1168 1169 1170 1171 1172 1173 1174 1175 1176 1177 1178 1179 1180 1181 1182 1183 1184 1185 1186 1187 1188 1189 1190 1191 1192 1193 1194 1195 1196 1197 1198 1199 1200 1201 1202 1203 1204 1205 1206 1207 1208 1209 1210 1211 1212 1213 1214 1215 1216 1217 1218 1219 1220 1221 1222 1223 1224 1225 1226 1227 1228 1229 1230 1231 1232 1233 1234 1235 1236 1237 1238 1239 1240 1241 1242 1243 1244 1245 1246 1247 1248 1249 1250 1251 1252 1253 1254 1255 1256 1257 1258 1259 1260 1261 1262 1263 1264 1265 1266 1267 1268 1269 1270 1271 1272 1273 1274 1275 1276 1277 1278 1279 1280 1281 1282 1283 1284 1285 1286 1287 1288 1289 1290 1291 1292 1293 1294 1295 1296 1297 1298 1299 1300 1301 1302 1303 1304 1305 1306 1307 1308 1309 1310 1311 1312 1313 1314 1315 1316 1317 1318 1319 1320 1321 1322 1323 1324 1325 1326 1327 1328 1329 1330 1331 1332 1333 1334 1335 1336 1337 1338 1339 1340 1341 1342 1343 1344 1345 1346 1347 1348 1349 1350 1351 1352 1353 1354 1355 1356 1357 1358 1359 1360 1361 1362 1363 1364 1365 1366 1367 1368 1369 1370 1371 1372 1373 1374 1375 1376 1377 1378 1379 1380 1381 1382 1383 1384 1385 1386 1387 1388 1389 1390 1391 1392 1393 1394 1395 1396 1397 1398 1399 1400 1401 1402 1403 1404 1405 1406 1407 1408 1409 1410 1411 1412 1413 1414 1415 1416 1417 1418 1419 1420 1421 1422 1423 1424 1425 1426 1427 1428 1429 1430 1431 1432 1433 1434 1435 1436 1437 1438 1439 1440 1441 1442 1443 1444 1445 1446 1447 1448 1449 1450 1451 1452 1453 1454 1455 1456 1457 1458 1459 1460 1461 1462 1463 1464 1465 1466 1467 1468 1469 1470 1471 1472 1473 1474 1475 1476 1477 1478 1479 1480 1481 1482 1483 1484 1485 1486 1487 1488 1489 1490 1491 1492 1493 1494 1495 1496 1497 1498 1499 1500 1501 1502 1503 1504 1505 1506 1507 1508 1509 1510 1511 1512 1513 1514 1515 1516 1517 1518 1519 1520 1521 1522 1523 1524 1525 1526 1527 1528 1529 1530 1531 1532 1533 1534 1535 1536 1537 1538 1539 1540 1541 1542 1543 1544 1545 1546 1547 1548 1549 1550 1551 1552 1553 1554 1555 1556 1557 1558 1559 1560 1561 1562 1563 1564 1565 1566 1567 1568 1569 1570 1571 1572 1573 1574 1575 1576 1577 1578 1579 1580 1581 1582 1583 1584 1585 1586 1587 1588 1589 1590 1591 1592 1593 1594 1595 1596 1597 1598 1599 1600 1601 1602 1603 1604 1605 1606 1607 1608 1609 1610 1611 1612 1613 1614 1615 1616 1617 1618 1619 1620 1621 1622 1623 1624 1625 1626 1627 1628 1629 1630 1631 1632 1633 1634 1635 1636 1637 1638 1639 1640 1641 1642 1643 1644 1645 1646 1647 1648 1649 1650 1651 1652 1653 1654 1655 1656 1657 1658 1659 1660 1661 1662 1663 1664 1665 1666 1667 1668 1669 1670 1671 1672 1673 1674 1675 1676 1677 1678 1679 1680 1681 1682 1683 1684 1685 1686 1687 1688 1689 1690 1691 1692 1693 1694 1695 1696 1697 1698 1699 1700 1701 1702 1703 1704 1705 1706 1707 1708 1709 1710 1711 1712 1713 1714 1715 1716 1717 1718 1719 1720 1721 1722 1723 1724 1725 1726 1727 1728 1729 1730 1731 1732 1733 1734 1735 1736 1737 1738 1739 1740 1741 1742 |
from fasthtml.common import *
from fasthtml.components import *
from plotly import graph_objects as go
from fh_plotly import plotly2fasthtml
import pandas as pd
import json
from data_viewer import view_data, gen_random_id
from data_viewer import DV, DV2, DVS
from rich import print
import uuid
import plotly.express as px
from fasthtml.components import D_code, D_bibliography, D_appendix, D_cite
overview = (
Div(
H2("Curated Sources Processing"),
H3("What This Section Contains"),
P(
"This section provides a complete discussion on the filtering applied to the 14 curated sources that comprise the non-Common Crawl data section of TxT360. The section is split into the following topic areas: "
),
Ul(
Li("Curated Sources Data Processing Summary", style="margin-bottom: 5px"),
Li(
"Individual Filtering Discussion for Each Source",
style="margin-bottom: 5px",
),
Li(B("Estimated Reading Time: 25 minutes"),style="margin-bottom: 5px", ),
),
),
)
curated_sources_intro = Div(
H2("Domain Specific Curated Sources"),
P(
"While massive amount of data can be crawled and obtained from the Internet, there are certain sources contain data in additional formats (e.g. PDF documents), or organized and published as official dumps (e.g. Wikipedia). We refer to these sources as curated sources. These dataset often comprises high-quality data that contain domain-specificity, such as academic publications or domain specific discussions. TxT360 was strongly influenced by The Pile",
D_cite(bibtex_key="thepile"),
" regarding both inclusion of the dataset and filtering techniques.",
),
P("These sources, such as Arxiv, Wikipedia, and Stack Exchange, provide high quality data. And as mentioned above, they are excluded from the web dataset via URL matching. Details about each of the sources are provided below. ",
),
P(
"TxT360 respects the copyright of the data sources and have not included the controversial data that was used in The Pile like YouTube and Opensubtitles, Reddit threads, and book3."
),
)
treemap_data = {
"Source": [
"ArXiv",
"PubMed Central",
"PubMed Abstract",
"S2ORC Full Text",
"S2ORC Abstract",
"PhilPapers",
"Wikipedia",
"StackExchange",
"EuroParl",
"Ubuntu IRC",
"Freelaw",
"PG19",
"USPTO",
"HackerNews",
"DM Maths",
],
"Category": [
"Papers",
"Papers",
"Papers",
"Papers",
"Papers",
"Papers",
"Internet",
"Conversational",
"Legal/Formal",
"Conversational",
"Legal/Formal",
"Books",
"Legal/Formal",
"Conversational",
"Reasoning",
],
"Count": [100, 200, 150, 120, 80, 90, 300, 250, 180, 150, 150, 250, 180, 120, 90],
"Details": [
"A repository of scientific papers in various disciplines, including computer science, physics, mathematics, and more.",
"A database of biomedical and life sciences research articles.",
"Abstracts of biomedical literature from various sources.",
"Full-text articles from the Semantic Scholar Open Research Corpus.",
"Abstracts of articles from the Semantic Scholar Open Research Corpus.",
"Papers from the PhilPapers database, a comprehensive index and bibliography of philosophy research.",
"A collaborative online encyclopedia that covers a wide range of topics.",
"A network of question-and-answer websites on various subjects, including programming, science, mathematics, and more.",
"A collection of multilingual parallel corpora of parliamentary debates from the European Parliament.",
"Chat logs from the Ubuntu Internet Relay Chat (IRC) channels.",
"Legal documents and court cases from various jurisdictions.",
"A collection of books from Project Gutenberg, a digital library of public domain works.",
"Patent documents from the United States Patent and Trademark Office.",
"User-generated news and discussion platform focused on technology and startups.",
"Deep Mind Maths dataset with generated questions.",
],
}
total_count = sum(treemap_data["Count"])
treemap_data["Percentage"] = [
count / total_count * 100 for count in treemap_data["Count"]
]
fig = px.treemap(
treemap_data,
path=["Category", "Source"],
values="Count",
hover_data=["Details", "Percentage"],
hover_name="Source",
)
treemap_chart = fig
# start individual tables showing filterin
wikipedia_filter = pd.DataFrame(
{
"Dataset": [
"Wikipedia",
],
"Lines Downloaded": [
"61614907",
],
"Percent Removed After Language Filter": [
"0.00%",
],
"Percent Removed After Min Word Count Filter": [
"1.86%",
],
"Percent Removed After Unigram Probability Filter": [
"0.00%",
],
"Percent Removed After Local Dedup": [
"0.31%",
],
"Total Percentage Remaining": [
"97.84%",
],
}
)
table_html_wikipedia = wikipedia_filter.to_html(index=False, border=0)
table_div_wikipedia = Div(NotStr(table_html_wikipedia))
freelaw_filter = pd.DataFrame(
{
"Dataset": [
"FreeLaw",
],
"Lines Downloaded": [
"75971288",
],
"Percent Removed After Language Filter": [
"3.00%",
],
"Percent Removed After Min Word Count Filter": [
"7.49%",
],
"Percent Removed After Unigram Probability Filter": [
"0.07%",
],
"Percent Removed After Local Dedup": [
"",
],
"Total Percentage Remaining": [
"%",
],
}
)
table_html_freelaw = freelaw_filter.to_html(index=False, border=0)
table_div_freelaw = Div(NotStr(table_html_freelaw))
dmm_filter = pd.DataFrame(
{
"Dataset": [
"DM Math",
],
"Lines Downloaded": [
"112559888",
],
"Percent Removed After Language Filter": [
"0.00%",
],
"Percent Removed After Min Word Count Filter": [
"0.00%",
],
"Percent Removed After Unigram Probability Filter": [
"0.00%",
],
"Percent Removed After Local Dedup": [
"0.00%",
],
"Total Percentage Remaining": [
"100.00%",
],
}
)
table_html_dmm = dmm_filter.to_html(index=False, border=0)
table_div_dmm = Div(NotStr(table_html_dmm))
uspto_filter = pd.DataFrame(
{
"Dataset": [
"USPTO",
],
"Lines Downloaded": [
"6880276",
],
"Percent Removed After Language Filter": [
"0.02%",
],
"Percent Removed After Min Word Count Filter": [
"1.88%",
],
"Percent Removed After Unigram Probability Filter": [
"0.01%",
],
"Percent Removed After Local Dedup": [
"22.94%",
],
"Total Percentage Remaining": [
"75.15%",
],
}
)
table_html_uspto = uspto_filter.to_html(index=False, border=0)
table_div_uspto = Div(NotStr(table_html_uspto))
pg19_filter = pd.DataFrame(
{
"Dataset": [
"PG-19",
],
"Lines Downloaded": [
"28752",
],
"Percent Removed After Language Filter": [
"0.24%",
],
"Percent Removed After Min Word Count Filter": [
"0.00%",
],
"Percent Removed After Unigram Probability Filter": [
"0.17%",
],
"Percent Removed After Local Dedup": [
"0.80%",
],
"Total Percentage Remaining": [
"98.78%",
],
}
)
table_html_pg19 = pg19_filter.to_html(index=False, border=0)
table_div_pg19 = Div(NotStr(table_html_pg19))
hn_filter = pd.DataFrame(
{
"Dataset": [
"HackerNews",
],
"Lines Downloaded": [
"2064931",
],
"Percent Removed After Language Filter": [
"2.62%",
],
"Percent Removed After Min Word Count Filter": [
"0.02%",
],
"Percent Removed After Unigram Probability Filter": [
"0.34%",
],
"Percent Removed After Local Dedup": [
"61.84%",
],
"Total Percentage Remaining": [
"35.18%",
],
}
)
table_html_hn = hn_filter.to_html(index=False, border=0)
table_div_hn = Div(NotStr(table_html_hn))
uirc_filter = pd.DataFrame(
{
"Dataset": [
"Ubunutu IRC",
],
"Lines Downloaded": [
"37966",
],
"Percent Removed After Language Filter": [
"38.10%",
],
"Percent Removed After Min Word Count Filter": [
"0.14%",
],
"Percent Removed After Unigram Probability Filter": [
"1.12%",
],
"Percent Removed After Local Dedup": [
"0.66%",
],
"Total Percentage Remaining": [
"59.98%",
],
}
)
table_html_uirc = uirc_filter.to_html(index=False, border=0)
table_div_uirc = Div(NotStr(table_html_uirc))
up_filter = pd.DataFrame(
{
"Dataset": [
"EuroParl",
],
"Lines Downloaded": [
"69814",
],
"Percent Removed After Language Filter": [
"0.00%",
],
"Percent Removed After Min Word Count Filter": [
"0.00%",
],
"Percent Removed After Unigram Probability Filter": [
"0.00%",
],
"Percent Removed After Local Dedup": [
"1.00%",
],
"Total Percentage Remaining": [
"99.00%",
],
}
)
table_html_up = up_filter.to_html(index=False, border=0)
table_div_up = Div(NotStr(table_html_up))
se_filter = pd.DataFrame(
{
"Dataset": [
"StackExchange",
],
"Lines Downloaded": [
"23246548",
],
"Percent Removed After Language Filter": [
"0.00%",
],
"Percent Removed After Min Word Count Filter": [
"0.00%",
],
"Percent Removed After Unigram Probability Filter": [
"0.00%",
],
"Percent Removed After Local Dedup": [
"0.00%",
],
"Total Percentage Remaining": [
"100.00%",
],
}
)
table_html_se = se_filter.to_html(index=False, border=0)
table_div_se = Div(NotStr(table_html_se))
arx_filter = pd.DataFrame(
{
"Dataset": [
"ArXiv",
],
"Lines Downloaded": [
"1911867",
],
"Percent Removed After Language Filter": [
"2.22%",
],
"Percent Removed After Min Word Count Filter": [
"5.65%",
],
"Percent Removed After Unigram Probability Filter": [
"0.07%",
],
"Percent Removed After Local Dedup": [
"0.00%",
],
"Total Percentage Remaining": [
"92.06%",
],
}
)
table_html_arx = arx_filter.to_html(index=False, border=0)
table_div_arx = Div(NotStr(table_html_arx))
s2o_filter = pd.DataFrame(
{
"Dataset": [
"S2ORC",
],
"Lines Downloaded": [
"12963563",
],
"Percent Removed After Language Filter": [
"0.00%",
],
"Percent Removed After Min Word Count Filter": [
"0.00%",
],
"Percent Removed After Unigram Probability Filter": [
"0.00%",
],
"Percent Removed After Local Dedup": [
"0.00%",
],
"Total Percentage Remaining": [
"100.00%",
],
}
)
table_html_s2o = s2o_filter.to_html(index=False, border=0)
table_div_s2o = Div(NotStr(table_html_s2o))
s2oa_filter = pd.DataFrame(
{
"Dataset": [
"S2ORC Abstract",
],
"Lines Downloaded": [
"102324176",
],
"Percent Removed After Language Filter": [
"18.04%",
],
"Percent Removed After Min Word Count Filter": [
"1.17%",
],
"Percent Removed After Unigram Probability Filter": [
"0.00%",
],
"Percent Removed After Local Dedup": [
"0.13%",
],
"Total Percentage Remaining": [
"80.66%",
],
}
)
table_html_s2oa = s2oa_filter.to_html(index=False, border=0)
table_div_s2oa = Div(NotStr(table_html_s2oa))
med_filter = pd.DataFrame(
{
"Dataset": [
"PubMed - Central",
],
"Lines Downloaded": [
"5230932",
],
"Percent Removed After Language Filter": [
"7.66%",
],
"Percent Removed After Min Word Count Filter": [
"1.29%",
],
"Percent Removed After Unigram Probability Filter": [
"0.02%",
],
"Percent Removed After Local Dedup": [
"0.00%",
],
"Total Percentage Remaining": [
"91.03%",
],
}
)
table_html_med = med_filter.to_html(index=False, border=0)
table_div_med = Div(NotStr(table_html_med))
pma_filter = pd.DataFrame(
{
"Dataset": [
"PubMed - Abstract",
],
"Lines Downloaded": [
"25787474",
],
"Percent Removed After Language Filter": [
"0.01%",
],
"Percent Removed After Min Word Count Filter": [
"0.14%",
],
"Percent Removed After Unigram Probability Filter": [
"0.00%",
],
"Percent Removed After Local Dedup": [
"0.00%",
],
"Total Percentage Remaining": [
"98.85%",
],
}
)
table_html_pma = pma_filter.to_html(index=False, border=0)
table_div_pma = Div(NotStr(table_html_pma))
phil_filter = pd.DataFrame(
{
"Dataset": [
"Phil Papers",
],
"Lines Downloaded": [
"49389",
],
"Percent Removed After Language Filter": [
"20.68%",
],
"Percent Removed After Min Word Count Filter": [
"0.00%",
],
"Percent Removed After Unigram Probability Filter": [
"0.12%",
],
"Percent Removed After Local Dedup": [
"0.00%",
],
"Total Percentage Remaining": [
"79.20%",
],
}
)
table_html_phil = phil_filter.to_html(index=False, border=0)
table_div_phil = Div(NotStr(table_html_phil))
## end individual tables showing filterin
## start filtered examples
wiki_examples = DV("data/curated_samples/wiki.json", 0, "Wikipedia")
freelaw_examples = DV2(
"data/curated_samples/freelaw_raw.json",
"data/curated_samples/freelaw_extract.json",
2,
)
se_examples = DV2(
"data/curated_samples/stackexchange_raw.json",
"data/curated_samples/stackexchange_extract.json",
3,
)
phil_examples = DV("data/curated_samples/philpapers_raw.json", 2, "PhilPapers")
arx_examples = DV2(
"data/curated_samples/arxiv_raw.json", "data/curated_samples/arxiv_extract.json", 3
)
s2o_examples = DV("data/curated_samples/s2orc_raw.json", 0, "S2ORC")
s2oa_examples = DV("data/curated_samples/s2orc_abstract_raw.json", 0, "S2ORC Abstract")
pubmed_examples = DV2(
"data/curated_samples/pubmed_raw.json",
"data/curated_samples/pubmed_extract.json",
3,
)
dmm_examples = DV2(
"data/curated_samples/dm_maths_raw.json",
"data/curated_samples/dm_maths_extract.json",
3,
)
pg19_examples = DV("data/curated_samples/pg19_raw.json", 0, "PG19")
eu_examples = DV("data/curated_samples/europarl_raw.json", 0, "Europarl")
## end filtered examples
data_preprocessing_div = Div(
H2("Filtering Steps and Definitions"),
P(
"Data preprocessing is a crucial step in the data science pipeline. It involves cleaning and transforming raw data into a format that is suitable for analysis. This process includes handling missing values, normalizing data, encoding categorical variables, and more."
),
P(
"The ",
B("Language Filter"),
" removes documents in unwanted languages. This step improves data quality by removing irrelevant documents.",
),
P(
"The ",
B("Minimum Word Count Filter"),
" sets a threshold for required words within a document. This step filters out low-quality or incomplete documents. However, this step may remove documents that contain valuable information so a proper analysis is important for each data source.",
),
P(
"The ",
B("Unigram Log Probability Filter"),
" calculates the log probability of each unigram to measure the significance of individual words. This step quantifies the importance of individual words but may not capture the semantic meaning of words. To calculate the average log word probability, we use word frequencies extracted from the ",
A("1T Web-gram corpus", href="https://catalog.ldc.upenn.edu/LDC2006T13"),
". Specifically, we use the list available created by ",
A(
"Rachel Tatman",
href="https://www.kaggle.com/datasets/rtatman/english-word-frequency",
),
".",
),
H3("Data Processing for S2ORC"),
P(
"The formatting of the S2ORC dataset required special filters to be applied. These filters were not applied to the other data sources."
),
P(
"The ",
B("Title and Abstract Filter"),
" extracts information from the title and abstract. This step provides additional information for analysis but may introduce bias in the analysis.",
),
P(
"The ",
B("Majority Language Filter"),
" identifies the majority language in the dataset. This step displays the distribution of languages in the dataset to enable language-specific analysis and insights.",
),
P(
"The ",
B("Paragraph Count Filter"),
" counts the number of paragraphs in each document. This step helps to analyze the structure and length of documents which can be a useful heuristic for document complexity.",
),
P(
"The ",
B("Frequency Filter"),
" calculates the frequency of each word in the dataset. This step serves to identify important words and topics in the dataset but may be sensitive to noise and outliers.",
),
)
def diff2_stacked_bar():
# Data for the stacked bar chart
data = {
"Filter": [
"Downloaded Lines",
"Language Filter",
"Min Word Count",
"Unigram Log Probability",
],
"Wikipedia": [61614907, 61614907, 60468491, 60468491],
"Freelaw": [75971288, 73690766, 68171834, 68123174],
"DM Maths": [112559888, 112559888, 112559888, 112559888],
"USPTO": [6880276, 6878964, 6749922, 6749389],
"PG19": [28752, 28683, 28682, 28632],
"Hackernews": [2064931, 2010802, 2010488, 2003636],
"Ubuntu IRC": [37966, 23501, 23468, 23205],
"Europarl": [69814, 69814, 69814, 69814],
"StackExchange": [23246548, 23246548, 23246352, 23246352],
"Arxiv": [1911867, 1869441, 1763840, 1762661],
"S2ORC": [12963563, 12963563, 12963563, 12963563],
"S2ORC Abstract": [102324176, 83867601, 82889293, 82777912],
"Pubmed Central": [5230932, 4830486, 4768310, 4767474],
"Pubmed Abstract": [25787474, 25784374, 25747955, 25746724],
"Phil Papers": [49389, 39175, 39175, 39128],
}
df = pd.DataFrame(data)
fig = go.Figure()
for dataset in df.columns[1:]:
fig.add_trace(go.Bar(name=dataset, x=df["Filter"], y=df[dataset]))
fig.update_layout(
barmode="stack",
title="Document Reduction by Filter for Each Dataset",
xaxis_title="Filter",
yaxis_title="Number of Lines",
legend_title="Dataset",
height=600,
)
return fig
filtering_process = Div(
Section(
H2("Filtering Discussion on All Curated Sources"),
P(
"Below is a detail recount of how each dataset was extracted and filtered. If specific challenges were found with a dataset, they are included and discussed to the best of our abilities. The figure below provides a global view of the document filtering results. ~8% of documents were removed during these three steps."
),
plotly2fasthtml(diff2_stacked_bar()),
H3(
"This section continues below with the specific filtering steps taken for all 14 curated datasets."
),
),
Section(
Div(
H3("Wikipedia"),
P(
"Wikipedia is an encyclopedia form of high-quality text data used for language modeling. We have included filtered and deduplicated versions of complete Wikipedia data directly provided by the Wikipedia Foundation for more than 350 languages."
),
P(
B("Download and Extraction: "),
"The Wikimedia dataset was downloaded from the official snapshot on Huggingface: ",
A(
"https://huggingface.co/datasets/wikimedia/wikipedia/tree/main",
href="https://huggingface.co/datasets/wikimedia/wikipedia/tree/main",
),
". The",
D_code("huggingface dataset.to_json", language="python"),
" function was used to convert the original parqet format to the jsonl format.",
),
P(
B("Filtering: "),
"Manual inspection of the dataset demonstrated high quality content. Only one filter was used to remove articles with few words. Based normal sentence constructs, the article was kept if it contained 10 or more words. Any article with fewer than 10 words was removed.",
),
table_div_wikipedia,
Details(
Summary("Wikipedia Filtering Examples"),
Div(
wiki_examples,
style="background-color: white; padding: 15px; margin-top: 10px; margin-bottom: 10px; border-radius: 8px; border: none; ", # Styling for the DV2 part
),
style="""
background-color: #FFFAEA; /* Light yellow background */
padding: 15px;
border-radius: 12px;
margin-bottom: 15px
""",
),
),
),
Section(
Div(
H3("ArXiv"),
P(
"The ArXiv dataset is a vast collection of preprint research papers primarily in Mathematics, Computer Science, and Physics. Established in 1991, it offers high-quality text and mathematical knowledge, making it an invaluable resource for academic and scientific research. ArXiv papers are typically written in LaTeX, a popular typesetting system for these fields. We have extracted the information from latex and converted it into a text format."
),
P(
B("Download and Extraction: "),
"All the data was downloaded in original latex format from ArXiv official S3 repo: ",
A("s3://arxiv/src", href="s3://arxiv/src"),
". We aim to encode the downloaded data in UTF-8 format, and when necessary, utilize the chardet library to infer the appropriate encoding. After that, we use ",
A("Pandoc", href="https://pandoc.org/"),
" to extract information from the latex files into markdown format. The command we use is",
D_code(
"pandoc <raw_tex_path> -s -o <output_markdown_path> -f latex+raw_tex -t markdown_mmd [--lua-filter <lua_filter_path>]",
language="bash",
),
". Finally, all markdowns were combined to create jsonl files.",
),
P(B("Unique Data Preparation Challenges: ")),
P(
"When converting LaTeX files into Markdown using Pandoc, it is crucial to account for different data formats to minimize information loss while also filtering out noisy content in LaTeX. Below, we outline our considerations and methods for handling various data types during this conversion process:"
),
Ul(
Li(
B("Tables: "),
"The process for handling tables follows three main approaches. First, tables compatible with Pandoc’s built-in formats are directly converted into standard Markdown tables. Notably, LaTeX’s '\\multicolumn' and '\\multirow' commands can be successfully translated into valid Markdown tables. Second, tables unsupported by Pandoc’s native functionality, such as deluxetable or other complex LaTeX types, are preserved in their original LaTeX format to maintain the integrity of complex structures. Third, only a few remaining tables have been converted to HTML web tables.",
style="margin-bottom: -3px",
),
Li(
B("Mathematical Expressions: "),
"Inline mathematical expressions are rendered in Markdown. More complex equations remain unchanged, e.g., presented as '\\begin{aligned}' blocks, to ensure accuracy and readability.",
style="margin-bottom: -3px",
),
Li(
B("Figures: "),
"All figures are removed during the conversion process. Placeholder figures might not contribute to the paper’s data quality and, as such, have been omitted to streamline the output.",
style="margin-bottom: -3px",
),
Li(
B("Section Headers: "),
"Section headers are converted into markdown format, using leading '#' symbols to represent the heading levels.",
style="margin-bottom: -3px",
),
Li(
B("References: "),
"References are removed. Although they may be informative, references often introduce formatting inconsistencies or add little value compared to the core content of the paper.",
style="margin-bottom: -3px",
),
),
P(
B(" Filters Applied: "),
"multiple filters are used here after manually verifying output of all the filters as suggested by peS2o dataset",
D_cite(bibtex_key="peS2o"),
),
Ul(
Li(
"Language Filter: any language other than English are discarded",
style="margin-bottom: -3px",
),
Li(
"Minimum Word Count Filter: less than 500 words (not inclusive) are discarded",
style="margin-bottom: -3px",
),
Li(
"Unigram Log Probability Filter Threshold: -20",
style="margin-bottom: -3px",
),
Li(
"Note: the Frequency Filter was calculated but not applied. The most frequent word in the paper consists of alpha characters only, and it appears in less than 7.5% of the document. Words are obtained by splitting the text on whitespace.",
style="margin-bottom: -3px",
),
),
table_div_arx,
Details(
Summary("ArXiv Filtering Examples"),
Div(
arx_examples,
style="background-color: white; padding: 15px; margin-top: 10px; margin-bottom: 10px; border-radius: 8px; border: none; ", # Styling for the DV2 part
),
style="""
background-color: #FFFAEA; /* Light yellow background */
padding: 15px;
border-radius: 12px;
margin-bottom: 15px
""",
),
),
),
Section(
Div(
H3("S2ORC"),
P(
"The Semantic Scholar Open Research Corpus (S2ORC) is a comprehensive dataset designed for natural language processing (NLP) and text-mining research over scientific papers. It includes rich metadata, and abstract and full-text content for millions of academic papers across various disciplines. This dataset is further divided into two components, S2ORC abstract and S2ORC full text."
),
H4(""),
P(
B("Download and Extraction: "),
"S2ORC was downloaded directly in zip format using S2ORC api key and a get() request: ",
D_code("response = urllib.request.urlopen(url)", language="python"),
),
P(
B("Filters Applied: "),
"Multiple filters are used here after manually verifying output of all the filters as suggested by peS2o dataset",
),
Ul(
Li(
"Title and Abstract Filter: must have title and abstract",
style="margin-bottom: -3px",
),
Li(
"Language Filter: The paper must be in English. To determine the language of each document, we use the pycld3 library. We run pycld3 on the first 2000 characters of each paragraph in the paper. The language of the paper is the most common language of the paragraphs.",
style="margin-bottom: -3px",
),
Li(
"Word Count Filter: less than 500 words (not inclusive) are discarded",
style="margin-bottom: -3px",
),
Li(
"Paragraph Count Filter: The paper must have at least 5 paragraphs after removing paragraphs with less than -20 average log world probability",
style="margin-bottom: -3px",
),
Li(
"Frequency Filter: The most frequent word in the paper consists of alpha characters only, and it appears in less than 7.5% of the document. Words are obtained by splitting the text on whitespace.",
style="margin-bottom: -3px",
),
),
table_div_s2o,
# Details(
# Summary("S2ORC Filtering Examples -- need to update"),
# Div(
# P("examples are missing"),
# style="background-color: white; padding: 15px; margin-top: 10px; margin-bottom: 10px; border-radius: 8px; border: none; ", # Styling for the DV2 part
# ),
# style="""
# background-color: #FFFAEA; /* Light yellow background */
# padding: 15px;
# border-radius: 12px;
# margin-bottom: 15px
# """,
# ),
),
),
Section(
Div(
H3("S2ORC Abstract"),
P(
"The Semantic Scholar Open Research Corpus (S2ORC) is a comprehensive dataset designed for natural language processing (NLP) and text-mining research over scientific papers. It includes rich metadata, and abstract and full-text content for millions of academic papers across various disciplines. This dataset is further divided into two components, S2ORC abstract and S2ORC full text."
),
P(
B("Download and Extraction: "),
"S2ORC was downloaded directly in zip format using S2ORC api key and a get() request: ",
D_code("response = urllib.request.urlopen(url)", language="python"),
),
P(
B("Filters Applied: "),
"multiple filters are used here after manually verifying output of all the filters as suggested by peS2o dataset. The frequency filter was not used as suggested by peS2o because it was removing good samples as inspected manually",
),
Ul(
Li(
"Title and Abstract Filter: must have title and abstract",
style="margin-bottom: -3px",
),
Li(
"Majority Language Filter: abstract must be in English",
style="margin-bottom: -3px",
),
Li(
"Minimum Word Count Filter: less than 20 (not inclusive) are discarded",
style="margin-bottom: -3px",
),
Li(
"Unigram Log Probability Threshold: -20",
style="margin-bottom: -3px",
),
Li(
"Note: Frequency Filter: The most frequent word in the paper consists of alpha characters only, and it appears in less than 7.5% of the document. Words are obtained by splitting the text on whitespace.",
style="margin-bottom: -3px",
),
),
table_div_s2oa,
#Details(
# Summary("S2ORC Abstract Filtering Examples "),
# Div(
# P("examples are missing"),
# style="background-color: white; padding: 15px; margin-top: 10px; margin-bottom: 10px; border-radius: 8px; border: none; ", # Styling for the DV2 part
# ),
# style="""
# background-color: #FFFAEA; /* Light yellow background */
# padding: 15px;
# border-radius: 12px;
# margin-bottom: 15px
# """,
# ),
)
),
Section(
Div(
H3("PubMed Central and PubMed Abstract"),
P(
B("Download and Extraction: "),
"All files were downloaded from",
A(
"ttps://ftp.ncbi.nlm.nih.gov/pub/pmc/oa_package/",
href="ttps://ftp.ncbi.nlm.nih.gov/pub/pmc/oa_package/",
),
". PubMed Central (PMC) files are downloaded in an xml.tar format. The tar files are opened and converted to markdown format using pandoc",
D_code(
"pandoc <raw_xml_path> -s -o <output_markdown_path> -f jats -t markdown_mmd [--lua-filter <lua_filter_path>]",
language="bash",
),
". The markdown files are combined to create jsonl files. PubMed Abstract (PMA) files were downloaded in xml. The BeautifulSoup library was used to extract the abstract, title, and PMID. All files were stored in jsonl format.",
),
P(B("Unique Data Preparation Challenges: ")),
Ul(
Li(
"We tried similar attempts on PMC as we did on ArXiv. The resulted markdown might have slight difference due to the different structure of the XML files.",
style="margin-bottom: -3px",
),
),
P(
B("Filters Applied: "),
"Multiple filters are used here after manually verifying output of all the filters as suggested by peS2o dataset.",
),
Ul(
Li(
"Minimum Word Count Filter: PMC documents with less than 100 words (not inclusive) are discarded; PMA documents less than 20 words are discarded",
style="margin-bottom: -3px",
),
Li("Language Filter: English only", style="margin-bottom: -3px"),
Li(
"Frequency Filter: The most frequent word in the paper consists of alpha characters only, and it appears in less than 7.5% of the document. Words are obtained by splitting the text on whitespace. This filter was not used for PMA",
style="margin-bottom: -3px",
),
Li(
"Unigram Log Probability Threshold: -20",
style="margin-bottom: -3px",
),
),
table_div_med,
table_div_pma,
Details(
Summary("PubMed Filtering Examples"),
Div(
pubmed_examples,
style="background-color: white; padding: 15px; margin-top: 10px; margin-bottom: 10px; border-radius: 8px; border: none; ", # Styling for the DV2 part
),
style="""
background-color: #FFFAEA; /* Light yellow background */
padding: 15px;
border-radius: 12px;
margin-bottom: 15px
""",
),
),
),
Section(
Div(
H3("Phil Papers"),
P(
"Papers from the PhilPapers database, a comprehensive index and bibliography of philosophy research maintained by the Center for Digital Philosophy at the University of Western Ontario."
),
P(
B("Download and Extraction: "),
"Original PDF files download from ",
A(
"https://philarchive.org/oai.pl",
href="https://philarchive.org/oai.pl",
),
". All available PDF's were downloaded. Each PDF was converted to text using java",
D_code(
"-jar ../philpapers_resources/src/pdfbox-app-2.0.21.jar ExtractText {f0} {FOUT.name}",
language="python",
),
". After converting to text formatting, a language was detected and added using the langdetect (citation needed) library.",
),
P(B("Filters Applied: ")),
Ul(
Li(
P(
"Hyphenation Removal:",
D_code("end-of", language="python"),
" becomes ",
D_code("end of", language="python"),
),
style="margin-bottom: -3px",
),
Li(
P(
"Newline Filtering:",
D_code("This is/na sentence.", language="python"),
" becomes ",
D_code("This is a sentence.", language="python"),
),
style="margin-bottom: -3px",
),
Li(
P(
"Header/Footer Filtering:",
D_code("(c) 2023 Company Name.", language="python"),
" is removed ",
),
style="margin-bottom: -3px",
),
Li(
P(
"Double Whitespace Filtering:",
D_code("This is a test.", language="python"),
" becomes ",
D_code("This is a test.", language="python"),
),
style="margin-bottom: -3px",
),
Li(
P(
"Mean Line Length Check: ",
"removes paragraphs with an average line length of < 2.0",
),
style="margin-bottom: -3px",
),
Li(
P(
"CID Percentage Filter: ",
"removes LaTex heavy paragraphs that contain over 10% “CID” font artifacts.",
),
style="margin-bottom: -3px",
),
Li(
P(
"Letterness Filter: ",
"discards paragraphs with a low proportion of letters",
),
style="margin-bottom: -3px",
),
Li(
P(
"Removing Leading/Trailing Numbers: ",
"removes numbers at the start or end of paragraphs. ",
D_code("1 This is a sentence.", language="python"),
" becomes ",
D_code("This is a sentence.", language="python"),
),
style="margin-bottom: -3px",
),
Li(
P("Fixing Unicode Issues: ", "fixes Unicode issues."),
style="margin-bottom: -3px",
),
Li(
P(
"Combining Diacritics Correction: ",
D_code("a'", language="python"),
" becomes ",
D_code("å", language="python"),
),
style="margin-bottom: -3px",
),
Li(
P(
"Unigram Log Probability: ",
"the document must have higher than -20 average unigram log probability.",
),
style="margin-bottom: -3px",
),
),
table_div_phil,
Details(
Summary("Phil Papers Filtering Examples"),
Div(
phil_examples,
style="background-color: white; padding: 15px; margin-top: 10px; margin-bottom: 10px; border-radius: 8px; border: none; ", # Styling for the DV2 part
),
style="""
background-color: #FFFAEA; /* Light yellow background */
padding: 15px;
border-radius: 12px;
margin-bottom: 15px
""",
),
),
),
Section(
Div(
H3("EuroParl"),
P(
"A collection of multilingual parallel corpora of parliamentary debates from the European Parliament. This is a high-quality legacy dataset earlier used for translation tasks."
),
P(
B("Download and Extraction: "),
"Original dataset was downloaded from ",
A(
"http://www.statmt.org/europarl/v7/europarl.tgz",
href="http://www.statmt.org/europarl/v7/europarl.tgz",
),
". The files were converted to jsonl lines for filtering.",
),
P(
B("Filters Applied: "),
"EuroParl was initially filtered during the download process. Documents with fewer than 200 characters were removed. The documents also contained HTML tags which were removed.",
),
D_code(
"""
Raw single line in data: <P> Hi I am speaker
After tag removal: P Hi I am speaker
We remove everything that starts with ["P", "BRK", "CHAPTER", "/P"]
and only keep tagnae == SPEAKER
because line starting with <SPEAKER> TEXT TEXT ....... has the relevant text
""",
style="block",
language="python",
),
D_code(
"""
def process_tag(original_tag):
tag = original_tag.strip(">").strip("<")
# Skip empty tags
if not tag:
return None
tagname = tag.split()[0]
# Skip paragraph, break, and chapter tags
if tagname in ["P", "BRK", "CHAPTER", "/P"]:
return None
# For speaker tags, return the name
if tagname == "SPEAKER":
soup = bs4.BeautifulSoup(original_tag, "html.parser")
name = soup.speaker["name"]
return name
# Raise a error here if there is a tag we don't know
raise ValueError(f"Unknown tag {tag}")
""",
style="block",
language="python",
),
table_div_up,
Details(
Summary("EuroParl Filtering Examples"),
Div(
eu_examples,
style="background-color: white; padding: 15px; margin-top: 10px; margin-bottom: 10px; border-radius: 8px; border: none; ", # Styling for the DV2 part
),
style="""
background-color: #FFFAEA; /* Light yellow background */
padding: 15px;
border-radius: 12px;
margin-bottom: 15px
""",
),
),
),
Section(
Div(
H3("HackerNews"),
P(
"A dialog-based dataset where user comments on the links as the head post aggregated by Y Combinator."
),
P(
B("Download and Extraction: "),
"The dataset was downloaded from the HackerNews repo here: ",
A(
"https://hacker-news.firebaseio.com/v0/item/",
href="https://hacker-news.firebaseio.com/v0/item/",
),
". The dataset was parsed using the Story ID. In this dataset each post is a story, and each reply is considered subsequent story. Story IDs were considered between ID 1 to 37500000. The URL for all Story IDs was pinged. If that ID returned an error, the ID was removed. Each request was given a 2 second wait to account for network time.",
),
P(
"The HackerNews dataset contains a vast amount of stories and is known for lively discussions. Due to the number of replies a story may contain, only longest comment thread for each story was sampled past level 3. All stories included the title (1st level) and all direct replies (2nd level). We may consider relax this constrain and extract more data."
),
P(B("Unique Data Preparation Challenges: ")),
Ul(
Li(
"The converesation and forum style structure can be a very helpful signal for language model training. During processing the dataset, we try to encode such structure but without introducing too much noise. We choose to use an",
D_code("<AUTHOR>", language="html"),
" tag to encode the main thread text by the original poster, and use a ",
D_code("<COMMENT>", language="html"),
" tag to encode the replies. We initially choose ",
D_code("<P>", language="html"),
" as a tag since it is used by some instruction tuning dataset, but realize the ",
D_code("<P>", language="html"),
" tag can easily conflict with the original text.",
style="margin-bottom: -3px",
),
Li(
"As discussed above, the comment heirarchies required a thoughful approach to extracting meaningful data. ",
style="margin-bottom: -3px",
),
Li(
"In the comment thread heirarchy, relationships had to be assigned to between the comments, sub-comments, and original story ID. ",
style="margin-bottom: -3px",
),
),
P(B("Filters Applied: ")),
Ul(
Li("Language Filter: English", style="margin-bottom: -3px"),
Li("Minimum Word Count Filter: 10", style="margin-bottom: -3px"),
Li(
"Unigram Log Probability Threshold: -20",
style="margin-bottom: -3px",
),
),
table_div_hn,
),
),
Section(
Div(
H3("USPTO"),
P("Patent documents from the United States Patent and Trademark Office."),
P(
B("Download and Extraction: "),
"Data was downloaded and extracted using tags from ",
A(
"https://bulkdata.uspto.gov/data/patent/grant/redbook/fulltext/",
href="https://bulkdata.uspto.gov/data/patent/grant/redbook/fulltext/",
),
". There were three different formats that needed three different functions to download and extract the data based on year:",
I("Pre_2002"),
", ",
I("2002_to_2004"),
" and",
I("post_2004"),
". We used the exact code used in The Pile (citation needed).",
),
P(B("Filters Applied: ")),
Ul(
Li("Language Filter: English", style="margin-bottom: -3px"),
Li("Minimum Word Count Filter: 50", style="margin-bottom: -3px"),
Li("Unigram Log Probability", style="margin-bottom: -3px"),
),
table_div_uspto,
),
),
Section(
Div(
H3("FreeLaw"),
P(
"Legal documents and court cases from various jurisdictions provided by US-registered non-profit firm Free Law Project. We have included data from CourtListener which included millions of legal opinions from federal and state courts."
),
P(
B("Download and Extraction"),
"The dataset was downloaded from: ",
A(
"https://storage.courtlistener.com/bulk-data/",
href="https://storage.courtlistener.com/bulk-data/",
),
". There are 19 CSV files which contain overlapping content. CSV files can contain content in multiple columns requiring a holistic extraction approach. Text was extracted from the following using html2text function. The block below shows how each text type was extracted.",
),
D_code(
"""
("html", html2text), ("html_lawbox", html2text),
("html_columbia", html2text), ("html_anon_2020", html2text),
("html_with_citations", html2text), ("xml_harvard", html2text),
plain_text
""",
language="python",
),
P(
"All content was downloaded leading to high number of documents filtered during local deduplication. Following The Pile, priority was given to plain_text first, followed by the columns in the table in reverse order."
),
P(B("Unique Data Preparation Challenges: ")),
P("The Freelaw text uses a lot of whitespaces and newlines to format the document visually. These lines are not necessary for language model learning and sometimes have confusing semantic meanings. We attempt to unify how whitespaces appear in this dataset with the following heuristics."),
Ul(
Li(
"Consecutive whitespaces and tabs were found. Consecutive Whitespaces and tabes were reduce to one, single whitespace.",
style="margin-bottom: -3px",
),
Li(
"Whitespaces were found between new lines with no addition text. These whitespaces were removed.",
style="margin-bottom: -3px",
),
Li(
"Consecutive new lines were found in some documents without leading to a new paragraph. All consecutive newline to a single new line.",
style="margin-bottom: -3px",
),
Li(
"Converted all single new lines to whitespace. If whitespace was found after a new line with no text, the whitespace was removed. All leading and trailing whitespace was removed.",
style="margin-bottom: -3px",
),
Li(
"All form feed (",
D_code("\\f", language="bash"),
")characters were removed.", style="margin-bottom: -3px"
),
),
P(B("Filters Applied: ")),
Ul(
Li("Language Filter: English", style="margin-bottom: -3px"),
Li("Minimum Word Count Filter: 50", style="margin-bottom: -3px"),
Li("Unigram Log Probability", style="margin-bottom: -3px"),
),
P(
"Note: Local deduplication within FreeLaw itself removed 90%+ of the dataset as duplicate."
),
table_div_freelaw,
Details(
Summary("FreeLaw Filtering Examples"),
Div(
freelaw_examples,
style="background-color: white; padding: 15px; margin-top: 10px; margin-bottom: 10px; border-radius: 8px; border: none; ", # Styling for the DV2 part
),
style="""
background-color: #FFFAEA; /* Light yellow background */
padding: 15px;
border-radius: 12px;
margin-bottom: 15px
""",
),
),
),
Section(
Div(
H3("StackExchange"),
P(
"A network of question-and-answer websites on various subjects, including programming, science, mathematics, and more. This is one of the largest publicly available repositories for question-answer pairs. We have included comments also to include an overall discussion on each post."
),
P(
B("Download and Extraction: "),
"The archive dataset was used to download all data from StackExchange and 364 StackExchange's sub URLs including: ",
A("math.stackexchange.com", href="math.stackexchange.com"),
". Raw data was extracted an XML format and only two files Posts.xml and Comments.xml were considered. To match the StackExchange hierarchy, each file was parsed using post_id to connect questions to answers and then to comments. We will include the full list of sub URLs in when the code is released.",
),
D_code(
"""
1. Questions:
2. Comment1:
3. Comment2:
4. Answer1:
5. Comment1:
6. Comment2:
7. Answer2:
8. Comment1:
9. Comment2:""",
block="block",
language="python",
),
P(B("Unique Data Preparation Challenges: ")),
Ul(
Li(
"Handling code block was a required finding the specific blocks and exacting the details in one snippet.",
style="margin-bottom: -3px",
),
Li(
"Question and Answer formatting had to be rewritten to match the question and the anwer.",
style="margin-bottom: -3px",
),
Li(
"Occasionally a title was not included at the beginning of a question. For consistent formatting, a title was added.",
style="margin-bottom: -3px",
),
),
P(B("Filters Applied: ")),
Ul(
Li("Minimum Word Count Filter: 10", style="margin-bottom: -3px"),
),
table_div_se,
Details(
Summary("StackExchange Filtering Examples"),
Div(
se_examples,
style="background-color: white; padding: 15px; margin-top: 10px; margin-bottom: 10px; border-radius: 8px; border: none; ", # Styling for the DV2 part
),
style="""
background-color: #FFFAEA; /* Light yellow background */
padding: 15px;
border-radius: 12px;
margin-bottom: 15px
""",
),
),
),
Section(
Div(
H3("Ubuntu IRC"),
P(
"Chat logs from the Ubuntu Internet Relay Chat (IRC) channels on the Freenode IRC chat server. This data is also another form of dialog dataset on niche topics."
),
P(
B("Download and Extraction: "),
"The dataset was downloaded from: ",
A(
"https://irclogs.ubuntu.com/{date.year}/{date.month:02d}/{date.day:02d}/",
href="https://irclogs.ubuntu.com/{date.year}/{date.month:02d}/{date.day:02d}/",
),
" based on the year.",
),
P("During extraction, the logs were cleaned using following functions:"),
D_code(
"""
def exclude_system(x):
return '\n'.join(line for line in x.split('\n') if not line.startswith('==='))
def exclude_select_system(x):
return '\n'.join(line for line in x.split('\n') if not (line.startswith('===')
and any(term in line for term in
['has joined #', 'has left #', 'Topic for #', "Topic (#", "is now known as"]) ))
def clean(x):
return '\n'.join('* ' + line[4:] if line.startswith('===') else line[8:] for line in x.split('\n'))
""",
block="block",
language="python",
),
P(B("Unique Data Preparation Challenges: ")),
Ul(
Li(
"Similar to the HackerNews challenges, we had to map comments and sub-comments to the original question.",
style="margin-bottom: -3px",
),
Li(
"The dataset comes with the usernames of post authors. We attempt to replace them with strings such as <USER1> to remove the PII. This step might also reduce the language model's effort to memorizing the user names."
),
),
P(B("Filters Applied: ")),
Ul(
Li("Language Filter: English", style="margin-bottom: -3px"),
Li("Minimum Word Count Filter: 10", style="margin-bottom: -3px"),
Li("Unigram Log Probability", style="margin-bottom: -3px"),
),
table_div_uirc,
),
),
Section(
Div(
H3("DM Math"),
P(
"DeepMind Math dataset with generated questions from various topics like algebra, calculus, geometry, etc. Maths data is included to improve model reasoning abilities in the downstream tasks."
),
P(
B("Download and Extraction: "),
"The dataset was downloaded directly from the Huggingface repo: ",
A(
"https://huggingface.co/datasets/deepmind/math_dataset",
href="https://huggingface.co/datasets/deepmind/math_dataset",
),
". The data was converted to the jsonl format where lines is represented as:",
),
D_code(
"""
Question: TEXT
Answer: TEXT""",
block="block",
language="python",
),
P(B("Unique Data Preparation Challenges: ")),
Ul(
Li(
"In one of our versions, we save the string as a byte string instead of raw text, introducing addition byte indicators at the string level",
style="margin-bottom: -3px",
),
Li('No space before keyword "Answer:"', style="margin-bottom: -3px"),
),
P(B("Filters Applied: ")),
Ul(
Li("No filtering was applied to DM Math", style="margin-bottom: -3px"),
),
table_div_dmm,
Details(
Summary("DM Math Filtering Examples"),
Div(
dmm_examples,
style="background-color: white; padding: 15px; margin-top: 10px; margin-bottom: 10px; border-radius: 8px; border: none; ", # Styling for the DV2 part
),
style="""
background-color: #FFFAEA; /* Light yellow background */
padding: 15px;
border-radius: 12px;
margin-bottom: 15px
""",
),
),
),
Section(
Div(
H3("PG-19"),
P(
"A collection of books from Project Gutenberg, a digital library of public domain works. This contains all the books that were published before 1919."
),
P(
B("Download and Extraction: "),
"The dataset was downloaded directly from Huggingface: ",
A(
"https://huggingface.co/datasets/deepmind/pg19",
href="https://huggingface.co/datasets/deepmind/pg19",
),
".",
),
P(B("Unique Data Preparation Challenges: ")),
Ul(
Li(
"The original books uses a lot of witespaces to format the text, similar to the case of FreeLaw. Sometimes, 10+ consecutive whitespaces were found. These whitespaces were reduce to one, single whitespace.",
style="margin-bottom: -3px",
),
Li(
"For similar reasons, consecutive new lines were found in some documents. All consecutive news over two were were reduce to two new lines.",
style="margin-bottom: -3px",
),
Li(
"The books are formmated with end-of-line hyphenation and break a single words into two lines. Hence a regular word such as ",
D_code("text", language="bash"),
" could become ",
D_code("te-\\nxt", language="bash"),
". We detect the combination of ",
D_code("-\\n", language="bash"),
" and remove them to the origin word heuristically.",
style="margin-bottom: -3px",
),
Li(
"Text delimiters such as * * * * * * * * were used to indicate structures like sections. We removed such known delimiters and replaced them with proper whitespaces and new lines. For others, we make sure there are no additional leading or trailing whitepsaces.",
style="margin-bottom: -3px",
),
),
P(B("Filters Applied:")),
Ul(
Li("Language Filter: English", style="margin-bottom: -3px"),
Li("Minimum Word Count Filter: 20", style="margin-bottom: -3px"),
Li("Unigram Log Probability: ", "-20", style="margin-bottom: -3px"),
),
table_div_pg19,
Details(
Summary("PG-19 Filtering Examples"),
Div(
pg19_examples,
style="background-color: white; padding: 15px; margin-top: 10px; margin-bottom: 10px; border-radius: 8px; border: none; ", # Styling for the DV2 part
),
style="""
background-color: #FFFAEA; /* Light yellow background */
padding: 15px;
border-radius: 12px;
margin-bottom: 15px
""",
),
),
),
)
local_dedup_text = P(
"Each curated data source has been prepared using its specific rules and has been locally deduped using min-hash near deduplication. Details about the dataset are shown below in the table:"
)
data_pipeline_table = pd.DataFrame(
{
"Data Source": [
"Papers",
"Wikipedia",
"StackExchange",
"Europarl",
"Ubuntu IRC",
"HackerNews",
"PG-19",
"USPTO",
"Freelaw",
"DM Math",
],
"Percent Filtered": [
"15%",
"21%",
"<0.1%",
"1%",
"0.4%",
"60%",
"0.8%",
"22.5%",
"94%",
"0",
],
"Unique Document Percentage": [
"75.99%",
"91.91%",
"98.02%",
"98.87%",
"100%",
"99.91%",
"31.81%",
"99.94%",
"91.01%",
"0",
],
"2 - 5 Duplicates": [
"19.4%",
"4.7%",
"1.27%",
"0.94%",
"0",
"0.05%",
"20.03%",
"0.05%",
"6,87%",
"0",
],
"6 - 10 Duplicates": [
"2.89%",
"1.58%",
"0.35%",
"0.09%",
"0",
"0.02%",
"24.27%",
"0.01%",
"1.07%",
"0",
],
"11 - 100 Duplicates": [
"1.17%",
"1.76%",
"0.35%",
"0.1",
"0",
"0.02%",
"22.26%",
"0.01%",
"1.05%",
"0",
],
"101 - 1000 Duplicates": [
"0.01%",
"0.05%",
"0.01%",
"0",
"0",
"<0.01%",
"1.58%",
"<0.01%",
"0.01%",
"0",
],
"1001+ Duplicates": [
"<0.01%",
"<0.01%",
"<0.01%",
"0",
"0",
"<0.01%",
"0.06%",
"0",
"0",
"0",
],
}
)
table_html_data_pipe = data_pipeline_table.to_html(index=False, border=0)
table_div_data_pipe = Div(NotStr(table_html_data_pipe), style="margin: 40px;")
def curated():
data_preparation_steps = pd.DataFrame(
{
"Method": [
"HTTP/FTP dumps",
"Web crawling",
"Archive snapshot",
"Generated",
"Curated",
],
"Description": [
"Acquiring data from HTTP/FTP dumps",
"Crawling websites to extract data",
"Working with archive dumps",
"Generating synthetic data",
"High quality curated data",
],
"Source": [
"Freelaw | Wikipedia | PhilPapers | Arxiv | S2ORC | Pubmeds",
"USPTO | Hackernews | Ubuntu IRC",
"StackExchange",
"DM Maths",
"PG19 | Europarl",
],
}
)
table_html = data_preparation_steps.to_html(index=False, border=0)
table_div = Div(NotStr(table_html), style="margin: 40px;")
text = P(
"""This initial stage serves as the foundation for the entire
process. Here, we focus on acquiring and extracting the raw data, which can
come from various sources such as crawling websites, using HTTP/FTP dumps,
or working with archive dumps. For instance, to download and prepare a
dataset, we can specific downloaders based on the data source. Each dataset
might have its own downloader script which can be updated in real time to
handle changes in the data source. Here is a general outline of the data
preparation process: It is worth noting that some pipelines might require
invoking additional functions or scripts to handle specific data sources or
formats. These helper scripts can be located within specific directories
or modules dedicated to the dataset."""
)
return Div(
Section(
overview,
id="section31",
),
Section(
curated_sources_intro,
plotly2fasthtml(treemap_chart),
# id="section31",
),
Section(
data_preprocessing_div,
id="section32",
),
Section(
filtering_process,
id="section33",
),
id="inner-text",
)
|