End of training

.hydra/config.yaml

@@ -1,8 +1,11 @@
 model:
-  dataset_name: redwoodresearch/diamonds-seed1
+  dataset_name: redwoodresearch/diamonds-seed2
   model_type: codegen
   pretrained_model_name: Salesforce/codegen-350M-mono
   max_length: 1024
+  model_config_params:
+    sensor_loc_type: locs_from_token
+    sensor_token: ' omit'
 hparams:
   learning_rate: 2.0e-05
   weight_decay: 0.02

.hydra/hydra.yaml

@@ -137,13 +137,13 @@ hydra:
     hydra:
     - hydra.mode=MULTIRUN
     task:
-    - model.dataset_name=redwoodresearch/diamonds-seed1
+    - model.dataset_name=redwoodresearch/diamonds-seed2
   job:
     name: train
     chdir: null
-    override_dirname: model.dataset_name=redwoodresearch/diamonds-seed1
-    id: '747438'
-    num: 0
+    override_dirname: model.dataset_name=redwoodresearch/diamonds-seed2
+    id: '748836_1'
+    num: 1
     config_name: codegen_diamonds_slurm
     env_set: {}
     env_copy: []
@@ -166,7 +166,7 @@ hydra:
     - path: ''
       schema: structured
       provider: schema
-    output_dir: /nas/ucb/oliveradk/measurement-pred/multirun/2024-12-17/07-26-22/0
+    output_dir: /nas/ucb/oliveradk/measurement-pred/multirun/2024-12-19/09-54-27/1
     choices:
       hparams: hparams
       model: codegen_diamonds

.hydra/overrides.yaml

@@ -1 +1 @@
-- model.dataset_name=redwoodresearch/diamonds-seed1
+- model.dataset_name=redwoodresearch/diamonds-seed2

README.md

@@ -17,16 +17,16 @@ should probably proofread and complete it, then remove this comment. -->
 
 This model is a fine-tuned version of [Salesforce/codegen-350M-mono](https://huggingface.co/Salesforce/codegen-350M-mono) on an unknown dataset.
 It achieves the following results on the evaluation set:
-- Loss: 0.4023
-- Accuracy: 0.9108
-- Accuracy Sensor 0: 0.9220
-- Auroc Sensor 0: 0.9580
-- Accuracy Sensor 1: 0.9109
-- Auroc Sensor 1: 0.9645
-- Accuracy Sensor 2: 0.9260
-- Auroc Sensor 2: 0.9611
-- Accuracy Aggregated: 0.8845
-- Auroc Aggregated: 0.9532
+- Loss: 0.4189
+- Accuracy: 0.9210
+- Accuracy Sensor 0: 0.9298
+- Auroc Sensor 0: 0.9628
+- Accuracy Sensor 1: 0.9259
+- Auroc Sensor 1: 0.9711
+- Accuracy Sensor 2: 0.9266
+- Auroc Sensor 2: 0.9619
+- Accuracy Aggregated: 0.9019
+- Auroc Aggregated: 0.9592
 
 ## Model description
 
@@ -61,11 +61,11 @@ The following hyperparameters were used during training:
 
 | Training Loss | Epoch  | Step | Validation Loss | Accuracy | Accuracy Sensor 0 | Auroc Sensor 0 | Accuracy Sensor 1 | Auroc Sensor 1 | Accuracy Sensor 2 | Auroc Sensor 2 | Accuracy Aggregated | Auroc Aggregated |
 |:-------------:|:------:|:----:|:---------------:|:--------:|:-----------------:|:--------------:|:-----------------:|:--------------:|:-----------------:|:--------------:|:-------------------:|:----------------:|
-| 0.3009        | 0.9997 | 781  | 0.4552          | 0.8074   | 0.8220            | 0.9041         | 0.8092            | 0.9255         | 0.8372            | 0.9304         | 0.7610              | 0.9026           |
-| 0.1989        | 1.9994 | 1562 | 0.3633          | 0.8595   | 0.8835            | 0.9425         | 0.8544            | 0.9520         | 0.8757            | 0.9517         | 0.8244              | 0.9351           |
-| 0.1335        | 2.9990 | 2343 | 0.3032          | 0.8924   | 0.8985            | 0.9529         | 0.8877            | 0.9608         | 0.9246            | 0.9573         | 0.8588              | 0.9463           |
-| 0.093         | 4.0    | 3125 | 0.3016          | 0.9138   | 0.9203            | 0.9581         | 0.9131            | 0.9651         | 0.9304            | 0.9609         | 0.8914              | 0.9529           |
-| 0.0432        | 4.9984 | 3905 | 0.4023          | 0.9108   | 0.9220            | 0.9580         | 0.9109            | 0.9645         | 0.9260            | 0.9611         | 0.8845              | 0.9532           |
+| 0.2961        | 0.9997 | 781  | 0.4800          | 0.7906   | 0.8122            | 0.9078         | 0.7952            | 0.9255         | 0.8160            | 0.9280         | 0.7391              | 0.8990           |
+| 0.1901        | 1.9994 | 1562 | 0.3107          | 0.8847   | 0.9115            | 0.9491         | 0.8649            | 0.9604         | 0.8951            | 0.9532         | 0.8674              | 0.9397           |
+| 0.1154        | 2.9990 | 2343 | 0.3076          | 0.9009   | 0.9154            | 0.9575         | 0.8946            | 0.9656         | 0.9255            | 0.9576         | 0.8682              | 0.9492           |
+| 0.0708        | 4.0    | 3125 | 0.3162          | 0.9207   | 0.9297            | 0.9621         | 0.9245            | 0.9710         | 0.9285            | 0.9619         | 0.9001              | 0.9587           |
+| 0.0314        | 4.9984 | 3905 | 0.4189          | 0.9210   | 0.9298            | 0.9628         | 0.9259            | 0.9711         | 0.9266            | 0.9619         | 0.9019              | 0.9592           |
 
 
 ### Framework versions

config.json

@@ -48,7 +48,6 @@
   "tokenizer_class": "GPT2Tokenizer",
   "torch_dtype": "float32",
   "transformers_version": "4.41.0",
-  "use_aggregated": true,
   "use_cache": false,
   "vocab_size": 51200
 }

configuration_measurement_pred.py

@@ -7,7 +7,6 @@ class MeasurementPredictorConfig(PretrainedConfig):
         sensor_token=" omit",
         sensor_loc_type="locs_from_token",
         n_sensors=3,
-        use_aggregated=True,
         sensors_weight = 0.7,
         aggregate_weight=0.3,
         **kwargs
@@ -15,7 +14,6 @@ class MeasurementPredictorConfig(PretrainedConfig):
         self.sensor_token = sensor_token 
         self.sensor_loc_type = sensor_loc_type
         self.n_sensors = n_sensors
-        self.use_aggregated = use_aggregated
         self.sensors_weight = sensors_weight
         self.aggregate_weight = aggregate_weight
         super().__init__(**kwargs)

logs/events.out.tfevents.1734630919.gail.ist.berkeley.edu.140348.0

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b4c6247b038ae42d4509821b4d01319d171d792d10a51710ab6d0134bc370bed
+size 16043

model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:57651a18b0a367e6c0ffaa97666d103bcf42708a6d8abab94d3cf7704204ad7d
+oid sha256:c6504413594e6bb22cce3c47736265b71f62a4a3d39ab10a3489af264340bce6
 size 1216963976

modeling_code_gen_measurement_pred.py

@@ -1,5 +1,5 @@
 from transformers.models.codegen import CodeGenPreTrainedModel, CodeGenModel
-
+from transformers import PreTrainedTokenizerBase
 from .modeling_measurement_pred import MeasurementPredictorMixin
 from .configuration_code_gen_measuremet_pred import CodeGenMeasurementPredictorConfig
 
@@ -11,3 +11,9 @@ class CodeGenMeasurementPredictor(CodeGenPreTrainedModel, MeasurementPredictorMi
         super().__init__(config)
         self.transformer = CodeGenModel(config)
         self.post_init()
+    
+    def set_pad_token(self, tokenizer: PreTrainedTokenizerBase):
+        pad_token = ' .'
+        pad_token_id = tokenizer.encode(pad_token)[0]
+        tokenizer.pad_token = pad_token
+        tokenizer.pad_token_id = pad_token_id

modeling_measurement_pred.py

@@ -1,4 +1,5 @@
 from typing import Optional, Tuple, Union
+from abc import abstractmethod
 
 import torch
 from torch.nn import BCEWithLogitsLoss
@@ -20,16 +21,18 @@ class MeasurementPredictorMixin(PreTrainedModel):
         self.sensor_probes = torch.nn.ModuleList([
             torch.nn.Linear(config.emb_dim, 1) for _ in range(config.n_sensors)
         ])
-        self.use_aggregated = config.use_aggregated
-        if config.use_aggregated:
-            self.aggregate_probe = torch.nn.Linear(config.emb_dim, 1)
+        self.aggregate_probe = torch.nn.Linear(config.emb_dim, 1)
         self.sensors_weight = config.sensors_weight
         self.aggregate_weight = config.aggregate_weight
 
-        self.get_sensor_locs: SensorLocFinder = None 
+        self.find_sensor_locs: SensorLocFinder = None 
+    
+    @abstractmethod
+    def set_pad_token(self, tokenizer: PreTrainedTokenizerBase):
+        pass
     
     def init_sensor_loc_finder(self, tokenizer: PreTrainedTokenizerBase):
-        self.get_sensor_locs = SENSOR_LOC_REGISTRY[self.sensor_loc_type](
+        self.find_sensor_locs = SENSOR_LOC_REGISTRY[self.sensor_loc_type](
             tokenizer, sensor_token=self.sensor_token, n_sensors=self.n_sensors
         )
 
@@ -67,28 +70,27 @@ class MeasurementPredictorMixin(PreTrainedModel):
             output_hidden_states=output_hidden_states,
             return_dict=return_dict,
         )
-        sensor_locs = self.get_sensor_locs(input_ids)
+        # get sensor embeddings (including aggregate)
+        sensor_locs = self.find_sensor_locs(input_ids)
         sensor_embs = base_model_output.last_hidden_state.gather(
             1, sensor_locs.unsqueeze(-1).expand(-1, -1, self.config.emb_dim)
         )
-        assert sensor_embs.shape == (input_ids.shape[0], self.n_sensors, self.config.emb_dim), f"{sensor_embs.shape} != {(input_ids.shape[0], self.n_sensors, self.config.emb_dim)}"
+        assert sensor_embs.shape == (input_ids.shape[0], self.n_sensors + 1, self.config.emb_dim), sensor_embs.shape
+        
+        # get sensor and aggregate logits
         sensor_logits = torch.concat([self.sensor_probes[i](sensor_embs[:, i, :]) 
                                for i in range(self.n_sensors)], dim=-1)
-        logits = sensor_logits
+        aggregate_logits = self.aggregate_probe(sensor_embs[:, -1, :])
+        logits = torch.concat([sensor_logits, aggregate_logits], dim=-1)
 
-        if self.use_aggregated:
-            last_emb = base_model_output.last_hidden_state[:, -1, :]
-            aggregate_logits = self.aggregate_probe(last_emb)
-            logits = torch.concat([logits, aggregate_logits], dim=-1)
-        
+        # compute loss
         loss = None
         if labels is not None:
             loss_fct = BCEWithLogitsLoss()
-            sensor_loss = loss_fct(sensor_logits, labels[:, :self.n_sensors]) * self.sensors_weight
+            sensor_loss = loss_fct(sensor_logits[:, :self.n_sensors], labels[:, :self.n_sensors]) * self.sensors_weight
             loss = sensor_loss
-            if self.use_aggregated: #TOOD: should be use aggregate
-                aggregate_loss = loss_fct(aggregate_logits, labels[:, -1:]) * self.aggregate_weight
-                loss += aggregate_loss
+            aggregate_loss = loss_fct(aggregate_logits, labels[:, -1:]) * self.aggregate_weight
+            loss += aggregate_loss
 
         if not return_dict:
             output = (logits, ) + base_model_output[1:]

sensor_loc_stories.py

@@ -26,6 +26,8 @@ class StoriesSensorLocFinder(SensorLocFinder):
             torch.argmax(eqs.to(torch.uint8), dim=-2),
             input_ids.shape[-1] - 3,
         ).clamp(max=input_ids.shape[-1] - 3)
+        aggregate_sensor_loc = locs[:, -1].unsqueeze(1)
+        locs = torch.cat([locs, aggregate_sensor_loc], dim=1)
         return locs
 
     

sensor_locs_from_token.py

@@ -13,4 +13,6 @@ class SensorLocFinderFromToken(SensorLocFinder):
     def find_sensor_locs(self, input_ids: torch.Tensor) -> torch.Tensor:
         flat_sensor_token_idxs = (input_ids == self.sensor_token_id).nonzero(as_tuple=True)[1]
         sensor_token_idxs = flat_sensor_token_idxs.view(-1, self.n_sensors)
+        aggregate_sensor_token_idx = sensor_token_idxs[:, -1].unsqueeze(1)
+        sensor_token_idxs = torch.cat([sensor_token_idxs, aggregate_sensor_token_idx], dim=1)
         return sensor_token_idxs

special_tokens_map.json

@@ -13,7 +13,7 @@
     "rstrip": false,
     "single_word": false
   },
-  "pad_token": "<|endoftext|>",
+  "pad_token": "Ġ.",
   "unk_token": {
     "content": "<|endoftext|>",
     "lstrip": false,

tokenizer.json

@@ -12,9 +12,9 @@
     },
     "direction": "Left",
     "pad_to_multiple_of": null,
-    "pad_id": 50256,
+    "pad_id": 764,
     "pad_type_id": 0,
-    "pad_token": "<|endoftext|>"
+    "pad_token": "Ġ."
   },
   "added_tokens": [
     {

tokenizer_config.json

@@ -318,7 +318,7 @@
   "clean_up_tokenization_spaces": true,
   "eos_token": "<|endoftext|>",
   "model_max_length": 2048,
-  "pad_token": "<|endoftext|>",
+  "pad_token": "Ġ.",
   "padding_side": "left",
   "return_token_type_ids": false,
   "tokenizer_class": "CodeGenTokenizer",

train.log

@@ -1,2 +1 @@
-[2024-12-17 07:27:38,728][accelerate.utils.other][WARNING] - Detected kernel version 5.4.0, which is below the recommended minimum of 5.5.0; this can cause the process to hang. It is recommended to upgrade the kernel to the minimum version or higher.
-[2024-12-17 07:27:38,922][accelerate.utils.other][WARNING] - Detected kernel version 5.4.0, which is below the recommended minimum of 5.5.0; this can cause the process to hang. It is recommended to upgrade the kernel to the minimum version or higher.
+[2024-12-19 09:55:18,350][accelerate.utils.other][WARNING] - Detected kernel version 5.4.0, which is below the recommended minimum of 5.5.0; this can cause the process to hang. It is recommended to upgrade the kernel to the minimum version or higher.