End of training

.hydra/config.yaml

@@ -3,6 +3,9 @@ model:
   model_type: codegen
   pretrained_model_name: Salesforce/codegen-350M-mono
   max_length: 1024
+  model_config_params:
+    sensor_loc_type: locs_from_token
+    sensor_token: ' omit'
 hparams:
   learning_rate: 2.0e-05
   weight_decay: 0.02

.hydra/hydra.yaml

@@ -142,8 +142,8 @@ hydra:
     name: train
     chdir: null
     override_dirname: model.dataset_name=redwoodresearch/diamonds-seed4
-    id: '746835'
-    num: 0
+    id: '748836_3'
+    num: 3
     config_name: codegen_diamonds_slurm
     env_set: {}
     env_copy: []
@@ -166,7 +166,7 @@ hydra:
     - path: ''
       schema: structured
       provider: schema
-    output_dir: /nas/ucb/oliveradk/measurement-pred/multirun/2024-12-16/18-52-16/0
+    output_dir: /nas/ucb/oliveradk/measurement-pred/multirun/2024-12-19/09-54-27/3
     choices:
       hparams: hparams
       model: codegen_diamonds

README.md

@@ -17,16 +17,16 @@ should probably proofread and complete it, then remove this comment. -->
 
 This model is a fine-tuned version of [Salesforce/codegen-350M-mono](https://huggingface.co/Salesforce/codegen-350M-mono) on an unknown dataset.
 It achieves the following results on the evaluation set:
-- Loss: 0.3745
-- Accuracy: 0.9126
-- Accuracy Sensor 0: 0.9165
-- Auroc Sensor 0: 0.9601
-- Accuracy Sensor 1: 0.9099
-- Auroc Sensor 1: 0.9647
-- Accuracy Sensor 2: 0.9342
-- Auroc Sensor 2: 0.9771
-- Accuracy Aggregated: 0.8898
-- Auroc Aggregated: 0.9613
+- Loss: 0.3733
+- Accuracy: 0.9086
+- Accuracy Sensor 0: 0.9144
+- Auroc Sensor 0: 0.9506
+- Accuracy Sensor 1: 0.9050
+- Auroc Sensor 1: 0.9584
+- Accuracy Sensor 2: 0.9332
+- Auroc Sensor 2: 0.9753
+- Accuracy Aggregated: 0.8820
+- Auroc Aggregated: 0.9557
 
 ## Model description
 
@@ -61,11 +61,11 @@ The following hyperparameters were used during training:
 
 | Training Loss | Epoch  | Step | Validation Loss | Accuracy | Accuracy Sensor 0 | Auroc Sensor 0 | Accuracy Sensor 1 | Auroc Sensor 1 | Accuracy Sensor 2 | Auroc Sensor 2 | Accuracy Aggregated | Auroc Aggregated |
 |:-------------:|:------:|:----:|:---------------:|:--------:|:-----------------:|:--------------:|:-----------------:|:--------------:|:-----------------:|:--------------:|:-------------------:|:----------------:|
-| 0.2756        | 0.9997 | 781  | 0.3221          | 0.8643   | 0.8659            | 0.9177         | 0.8499            | 0.9112         | 0.9025            | 0.9476         | 0.8388              | 0.9090           |
-| 0.1793        | 1.9994 | 1562 | 0.2547          | 0.8960   | 0.9032            | 0.9461         | 0.8847            | 0.9450         | 0.9345            | 0.9710         | 0.8617              | 0.9433           |
-| 0.1281        | 2.9990 | 2343 | 0.2960          | 0.8797   | 0.8882            | 0.9563         | 0.8726            | 0.9584         | 0.9133            | 0.9719         | 0.8447              | 0.9553           |
-| 0.0685        | 4.0    | 3125 | 0.3088          | 0.9049   | 0.9163            | 0.9597         | 0.9014            | 0.9638         | 0.9259            | 0.9765         | 0.8761              | 0.9609           |
-| 0.0342        | 4.9984 | 3905 | 0.3745          | 0.9126   | 0.9165            | 0.9601         | 0.9099            | 0.9647         | 0.9342            | 0.9771         | 0.8898              | 0.9613           |
+| 0.3029        | 0.9997 | 781  | 0.3411          | 0.8441   | 0.8553            | 0.9103         | 0.8390            | 0.9066         | 0.8633            | 0.9334         | 0.8188              | 0.8975           |
+| 0.2003        | 1.9994 | 1562 | 0.2859          | 0.8852   | 0.8929            | 0.9380         | 0.8778            | 0.9380         | 0.9319            | 0.9638         | 0.8384              | 0.9361           |
+| 0.1366        | 2.9990 | 2343 | 0.2701          | 0.8945   | 0.9041            | 0.9549         | 0.8902            | 0.9570         | 0.9245            | 0.9755         | 0.8591              | 0.9539           |
+| 0.0812        | 4.0    | 3125 | 0.2992          | 0.9046   | 0.9166            | 0.9542         | 0.8947            | 0.9585         | 0.9339            | 0.9765         | 0.8730              | 0.9567           |
+| 0.0381        | 4.9984 | 3905 | 0.3733          | 0.9086   | 0.9144            | 0.9506         | 0.9050            | 0.9584         | 0.9332            | 0.9753         | 0.8820              | 0.9557           |
 
 
 ### Framework versions

config.json

@@ -48,7 +48,6 @@
   "tokenizer_class": "GPT2Tokenizer",
   "torch_dtype": "float32",
   "transformers_version": "4.41.0",
-  "use_aggregated": true,
   "use_cache": false,
   "vocab_size": 51200
 }

configuration_measurement_pred.py

@@ -7,7 +7,6 @@ class MeasurementPredictorConfig(PretrainedConfig):
         sensor_token=" omit",
         sensor_loc_type="locs_from_token",
         n_sensors=3,
-        use_aggregated=True,
         sensors_weight = 0.7,
         aggregate_weight=0.3,
         **kwargs
@@ -15,7 +14,6 @@ class MeasurementPredictorConfig(PretrainedConfig):
         self.sensor_token = sensor_token 
         self.sensor_loc_type = sensor_loc_type
         self.n_sensors = n_sensors
-        self.use_aggregated = use_aggregated
         self.sensors_weight = sensors_weight
         self.aggregate_weight = aggregate_weight
         super().__init__(**kwargs)

logs/events.out.tfevents.1734630919.gan.ist.berkeley.edu.947855.0

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0c9782edca7024c8df399f33dc66c2e68f51006bfab574e4d41fcd0ec4e1fd27
+size 16043

model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:aeb3660f44c9bfb4d0af2162ca48b4d624f0d48f0335c52c0d91f5979689661d
+oid sha256:00dde8aa831a4318c073eac0225f9e719b9f67bd1dd9db30911b365594367ad0
 size 1216963976

modeling_code_gen_measurement_pred.py

@@ -1,5 +1,5 @@
 from transformers.models.codegen import CodeGenPreTrainedModel, CodeGenModel
-
+from transformers import PreTrainedTokenizerBase
 from .modeling_measurement_pred import MeasurementPredictorMixin
 from .configuration_code_gen_measuremet_pred import CodeGenMeasurementPredictorConfig
 
@@ -11,3 +11,9 @@ class CodeGenMeasurementPredictor(CodeGenPreTrainedModel, MeasurementPredictorMi
         super().__init__(config)
         self.transformer = CodeGenModel(config)
         self.post_init()
+    
+    def set_pad_token(self, tokenizer: PreTrainedTokenizerBase):
+        pad_token = ' .'
+        pad_token_id = tokenizer.encode(pad_token)[0]
+        tokenizer.pad_token = pad_token
+        tokenizer.pad_token_id = pad_token_id

modeling_measurement_pred.py

@@ -1,4 +1,5 @@
 from typing import Optional, Tuple, Union
+from abc import abstractmethod
 
 import torch
 from torch.nn import BCEWithLogitsLoss
@@ -20,16 +21,18 @@ class MeasurementPredictorMixin(PreTrainedModel):
         self.sensor_probes = torch.nn.ModuleList([
             torch.nn.Linear(config.emb_dim, 1) for _ in range(config.n_sensors)
         ])
-        self.use_aggregated = config.use_aggregated
-        if config.use_aggregated:
-            self.aggregate_probe = torch.nn.Linear(config.emb_dim, 1)
+        self.aggregate_probe = torch.nn.Linear(config.emb_dim, 1)
         self.sensors_weight = config.sensors_weight
         self.aggregate_weight = config.aggregate_weight
 
-        self.get_sensor_locs: SensorLocFinder = None 
+        self.find_sensor_locs: SensorLocFinder = None 
+    
+    @abstractmethod
+    def set_pad_token(self, tokenizer: PreTrainedTokenizerBase):
+        pass
     
     def init_sensor_loc_finder(self, tokenizer: PreTrainedTokenizerBase):
-        self.get_sensor_locs = SENSOR_LOC_REGISTRY[self.sensor_loc_type](
+        self.find_sensor_locs = SENSOR_LOC_REGISTRY[self.sensor_loc_type](
             tokenizer, sensor_token=self.sensor_token, n_sensors=self.n_sensors
         )
 
@@ -67,28 +70,27 @@ class MeasurementPredictorMixin(PreTrainedModel):
             output_hidden_states=output_hidden_states,
             return_dict=return_dict,
         )
-        sensor_locs = self.get_sensor_locs(input_ids)
+        # get sensor embeddings (including aggregate)
+        sensor_locs = self.find_sensor_locs(input_ids)
         sensor_embs = base_model_output.last_hidden_state.gather(
             1, sensor_locs.unsqueeze(-1).expand(-1, -1, self.config.emb_dim)
         )
-        assert sensor_embs.shape == (input_ids.shape[0], self.n_sensors, self.config.emb_dim), f"{sensor_embs.shape} != {(input_ids.shape[0], self.n_sensors, self.config.emb_dim)}"
+        assert sensor_embs.shape == (input_ids.shape[0], self.n_sensors + 1, self.config.emb_dim), sensor_embs.shape
+        
+        # get sensor and aggregate logits
         sensor_logits = torch.concat([self.sensor_probes[i](sensor_embs[:, i, :]) 
                                for i in range(self.n_sensors)], dim=-1)
-        logits = sensor_logits
+        aggregate_logits = self.aggregate_probe(sensor_embs[:, -1, :])
+        logits = torch.concat([sensor_logits, aggregate_logits], dim=-1)
 
-        if self.use_aggregated:
-            last_emb = base_model_output.last_hidden_state[:, -1, :]
-            aggregate_logits = self.aggregate_probe(last_emb)
-            logits = torch.concat([logits, aggregate_logits], dim=-1)
-        
+        # compute loss
         loss = None
         if labels is not None:
             loss_fct = BCEWithLogitsLoss()
-            sensor_loss = loss_fct(sensor_logits, labels[:, :self.n_sensors]) * self.sensors_weight
+            sensor_loss = loss_fct(sensor_logits[:, :self.n_sensors], labels[:, :self.n_sensors]) * self.sensors_weight
             loss = sensor_loss
-            if self.use_aggregated: #TOOD: should be use aggregate
-                aggregate_loss = loss_fct(aggregate_logits, labels[:, -1:]) * self.aggregate_weight
-                loss += aggregate_loss
+            aggregate_loss = loss_fct(aggregate_logits, labels[:, -1:]) * self.aggregate_weight
+            loss += aggregate_loss
 
         if not return_dict:
             output = (logits, ) + base_model_output[1:]

sensor_loc_stories.py

@@ -26,6 +26,8 @@ class StoriesSensorLocFinder(SensorLocFinder):
             torch.argmax(eqs.to(torch.uint8), dim=-2),
             input_ids.shape[-1] - 3,
         ).clamp(max=input_ids.shape[-1] - 3)
+        aggregate_sensor_loc = locs[:, -1].unsqueeze(1)
+        locs = torch.cat([locs, aggregate_sensor_loc], dim=1)
         return locs
 
     

sensor_locs_from_token.py

@@ -13,4 +13,6 @@ class SensorLocFinderFromToken(SensorLocFinder):
     def find_sensor_locs(self, input_ids: torch.Tensor) -> torch.Tensor:
         flat_sensor_token_idxs = (input_ids == self.sensor_token_id).nonzero(as_tuple=True)[1]
         sensor_token_idxs = flat_sensor_token_idxs.view(-1, self.n_sensors)
+        aggregate_sensor_token_idx = sensor_token_idxs[:, -1].unsqueeze(1)
+        sensor_token_idxs = torch.cat([sensor_token_idxs, aggregate_sensor_token_idx], dim=1)
         return sensor_token_idxs

special_tokens_map.json

@@ -13,7 +13,7 @@
     "rstrip": false,
     "single_word": false
   },
-  "pad_token": "<|endoftext|>",
+  "pad_token": "Ġ.",
   "unk_token": {
     "content": "<|endoftext|>",
     "lstrip": false,

tokenizer.json

@@ -12,9 +12,9 @@
     },
     "direction": "Left",
     "pad_to_multiple_of": null,
-    "pad_id": 50256,
+    "pad_id": 764,
     "pad_type_id": 0,
-    "pad_token": "<|endoftext|>"
+    "pad_token": "Ġ."
   },
   "added_tokens": [
     {

tokenizer_config.json

@@ -318,7 +318,7 @@
   "clean_up_tokenization_spaces": true,
   "eos_token": "<|endoftext|>",
   "model_max_length": 2048,
-  "pad_token": "<|endoftext|>",
+  "pad_token": "Ġ.",
   "padding_side": "left",
   "return_token_type_ids": false,
   "tokenizer_class": "CodeGenTokenizer",

train.log

@@ -1 +1 @@
-[2024-12-16 18:53:17,277][accelerate.utils.other][WARNING] - Detected kernel version 5.4.0, which is below the recommended minimum of 5.5.0; this can cause the process to hang. It is recommended to upgrade the kernel to the minimum version or higher.
+[2024-12-19 09:55:18,619][accelerate.utils.other][WARNING] - Detected kernel version 5.4.0, which is below the recommended minimum of 5.5.0; this can cause the process to hang. It is recommended to upgrade the kernel to the minimum version or higher.