multilingual-cpv-sector-classifier

This model is a fine-tuned version of bert-base-multilingual-cased on the Tenders Economic Daily Public Procurement Data. It achieves the following results on the evaluation set:

• F1 Score: 0.686

Model description

The model takes procurement descriptions written in any of 104 languages and classifies them into 45 sector classes represented by CPV(Common Procurement Vocabulary) code descriptions as listed below.

Common Procurement Vocabulary
Administration, defence and social security services. 👮‍♀️
Agricultural machinery. 🚜
Agricultural, farming, fishing, forestry and related products. 🌾
Agricultural, forestry, horticultural, aquacultural and apicultural services. 👨🏿‍🌾
Architectural, construction, engineering and inspection services. 👷‍♂️
Business services: law, marketing, consulting, recruitment, printing and security. 👩‍💼
Chemical products. 🧪
Clothing, footwear, luggage articles and accessories. 👖
Collected and purified water. 🌊
Construction structures and materials; auxiliary products to construction (excepts electric apparatus). 🧱
Construction work. 🏗️
Education and training services. 👩🏿‍🏫
Electrical machinery, apparatus, equipment and consumables; Lighting. ⚡
Financial and insurance services. 👨‍💼
Food, beverages, tobacco and related products. 🍽️
Furniture (incl. office furniture), furnishings, domestic appliances (excl. lighting) and cleaning products. 🗄️
Health and social work services. 👨🏽‍⚕️
Hotel, restaurant and retail trade services. 🏨
IT services: consulting, software development, Internet and support. 🖥️
Industrial machinery. 🏭
Installation services (except software). 🛠️
Laboratory, optical and precision equipments (excl. glasses). 🔬
Leather and textile fabrics, plastic and rubber materials. 🧵
Machinery for mining, quarrying, construction equipment. ⛏️
Medical equipments, pharmaceuticals and personal care products. 💉
Mining, basic metals and related products. ⚙️
Musical instruments, sport goods, games, toys, handicraft, art materials and accessories. 🎸
Office and computing machinery, equipment and supplies except furniture and software packages. 🖨️
Other community, social and personal services. 🧑🏽‍🤝‍🧑🏽
Petroleum products, fuel, electricity and other sources of energy. 🔋
Postal and telecommunications services. 📶
Printed matter and related products. 📰
Public utilities. ⛲
Radio, television, communication, telecommunication and related equipment. 📡
Real estate services. 🏠
Recreational, cultural and sporting services. 🚴
Repair and maintenance services. 🔧
Research and development services and related consultancy services. 👩‍🔬
Security, fire-fighting, police and defence equipment. 🧯
Services related to the oil and gas industry. ⛽
Sewage-, refuse-, cleaning-, and environmental services. 🧹
Software package and information systems. 🔣
Supporting and auxiliary transport services; travel agencies services. 🚃
Transport equipment and auxiliary products to transportation. 🚌
Transport services (excl. Waste transport). 💺

Intended uses & limitations

• Input description should be written in any of the 104 languages that MBERT supports.
• The model is just evaluated in 22 languages. Thus there is no information about the performances in other languages.
• The domain is also restricted by the awarded procurement notice descriptions in European Union. Evaluating on whole document texts might change the performance.

Training and evaluation data

• The whole data consists of 744,360 rows. Shuffled and split into train and validation sets by using 80%/20% manner.
• Each description represents a unique contract notice description awarded between 2011 and 2018.
• Both training and validation data have contract notice descriptions written in 22 European Languages. (Malta and Irish are extracted due to scarcity compared to whole data)

Training procedure

The training procedure has been completed on Google Cloud V3-8 TPUs. Thanks Google for giving the access to Cloud TPUs

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 2e-05
• num_epochs: 3
• gradient_accumulation_steps: 8
• batch_size_per_device: 4
• total_train_batch_size: 32

Training results

Epoch	Step	F1 Score
1	18,609	0.630
2	37,218	0.674
3	55,827	0.686

Language	F1 Score	Test Size
PL	0.759	13950
RO	0.736	3522
SK	0.719	1122
LT	0.687	2424
HU	0.681	1879
BG	0.675	2459
CS	0.668	2694
LV	0.664	836
DE	0.645	35354
FI	0.644	1898
ES	0.643	7483
PT	0.631	874
EN	0.631	16615
HR	0.626	865
IT	0.626	8035
NL	0.624	5640
EL	0.623	1724
SL	0.615	482
SV	0.607	3326
DA	0.603	1925
FR	0.601	33113
ET	0.572	458