KBTG-Labs/THaLLE-RLU-0.1-7B

THaLLE-RLU: Text Hyperlocally Augmented Large Language - Extension Regulation Language Understanding

Training details

This model is a fine-tuned Qwen2.5-7B-Instruct on the regulatory domain. With more than 40,000 data collected using LoRA.

Usage

Requirements

Since KBTG-Labs/THaLLE-RLU-0.1-7B is a fine-tuned version of Qwen2.5-7B-Instruct, we recommend using the latest transformers library. (For reference, we used transformers==4.45.1 during fine-tuning.)

Inference

script python THaLLE-RLU-0_1-7B_inference.py for this model's evaluation result. This Python script performs inference on multiple predefined tasks, utilizing validation datasets and model weights from Hugging Face (HF) repositories. The primary purpose of this script is to streamline the inference process for each task in a challenge environment, ensuring consistent and accurate model evaluations.

Instructions for Running the Script

1.Setup Validation Files: Place each validation file in the specified path according to task requirements.
2.Specify Save Directory: Define a directory to save the results of each task. Each task’s output will be saved as both CSV and JSON files for easy evaluation and future reference.
3.Model Weights: Ensure that the Hugging Face model weights are accessible (through a pre-trained model in the HF repo). Update the model_path variable to point to the correct repository if necessary.

Script Components

The script is divided into several key sections to facilitate understanding and modification.

1.Model and Inference Setup

• model_path: Update this variable to point to your Hugging Face model repository.
• Model Initialization: The model and tokenizer are loaded using Hugging Face’s from_pretrained method, allowing you to directly use a pre-trained model without additional setup.
• QwenInference Class: This class handles inference generation for each task by setting system and user prompts specific to the requirements of each challenge task.

2. Task Classes

Each task is encapsulated in its own class (e.g., AbbreviationTask, DefinitionTask) that inherits from a base Task class. Each task class performs the following:

• Input Formatting: Formats inputs using specific prompts, ensuring consistent structure.
• Inference Execution: Runs inference on each question in the validation set.
• Result Saving: Saves the inference outputs in both CSV and JSON formats to the specified directory.

3. The Task Base Class

• The base Task class provides methods to initialize paths, run inferences, and save results.
• save_results Method: Saves inference results to both CSV and JSON files for easy evaluation.

4. Main Function:

• Purpose: The main function configures and executes each task sequentially, using the respective validation files.
• Task Configuration: Each task is defined with a specific validation file path, output save path, system prompt, input variable for inference, and input template, as specified on the COLING-2025 Regulations Challenge website under the Task Definition.
• Execution: The script iterates over each task, creating instances of each task class and running their inference.

How to Run the Script

1. Prepare Validation Files and Output Directory:

• Validation Files: Place all required validation files in a specified directory, as indicated in the main function of the THaLLE-RLU-0_1-7B_inference.py script.
• Output Directory: Specify an output directory where the results for each task will be saved, configured within the main function.

Example Usage:

The filenames and column headers used in the example below are consistent with those in the validation CSV files provided on GitHub. Here’s how to configure each task and file in your script

tasks = [
        # Abbreviation Tasks
        {
            "class": AbbreviationTask,
            "file_path": "val_data/Task1-Abbreviation-acronym-validation.csv",
            "col_question": "abbreviations",
            "col_label": "answer",
            "csv_output_path": save_path_csv+"/abbreviation_task_acronym.csv",
            "json_output_path": save_path_json+"/abbreviation_task_acronym.json",
            "params": {"task_type": "abbreviation"}
        },
        {
            "class": AbbreviationTask,
            "file_path": "val_data/Task1-Abbreivation-tickers-validation.csv",
            "col_question": "Ticker",
            "col_label": "Company Name",
            "csv_output_path": save_path_csv+"/abbreviation_task_ticker.csv",
            "json_output_path": save_path_json+"/abbreviation_task_ticker.json",
            "params": {"task_type": "abbreviation"}
        },
        # Definition Task
        {
            "class": DefinitionTask,
            "file_path": "val_data/Task2-Definition-Federal_Reserve_Regulations-validation.csv",
            "col_question": "term",
            "col_label": "answer",
            "csv_output_path": save_path_csv+"/definition_task.csv",
            "json_output_path": save_path_json+"/definition_task.json",
            "params": {"task_type": "definition"}
        },
        # Additional task configurations...
]

Special Case: CFA Task Format

For the CFA task, which has not been released on GitHub, the column naming convention follows the example dataset shared on the COLING-2025 Regulations Challenge website under Dataset session. This format is applied as follows:

Context	Question	Choice 1	Choice 2	Choice 3	Answer
Maria Harris is a CFA Level 3 candidate and portfolio manager for Islandwide Hedge Fund...	Which action by Park violated Standard...?	A. Increasing allocation to the problem client.	B. Decreased allocation to the brother-in-law and other firm clients.	C. Both actions are violations.	C. Both actions are violations.

Here is an example for setting up the CFA exam task:

tasks = [
    # Other task configurations...
    {
        "class": CertificateQuestionTask,
        "file_path": "val_data/Task6-CFA-Level1.csv",
        "col_question": "Question",
        "col_label": "Answer",
        "csv_output_path": save_path_csv + "/certificate_task_cfa_level1.csv",
        "json_output_path": save_path_json + "/certificate_task_cfa_level1.json",
        "params": {"task_type": "CFA", "level": 1}
    },
     # Additional tasks...
]

2. Run the Script:

• Execute the script with Python by running: $ python THaLLE-RLU-0_1-7B_inference.py
• The script will automatically load the model, process each validation file, and save the inference results.

3. Results:

• The results for each task will be saved in the specified output directory as both CSV and JSON files.
• Each file contains the model's response for each question in the validation file, enabling easy comparison with ground-truth labels for challenge evaluation.

Key Notes

• Model Requirements: Ensure the model weights are available in the HF repo. Update the model_path if needed.
• Task Flexibility: This script is modular, so you can easily add or remove tasks by modifying the tasks configuration in the main function.
• Inference Customization: The script allows easy customization of prompts for each task in their respective classes. You can adjust these prompts based on any updated challenge requirements.

Note: The THaLLE-RLU-0_1-7B_inference.py was used exclusively for the competition, so the code may not be optimized or polished.