Verifiable-Coherent-NLU.ipynb

{"cells":[{"cell_type":"markdown","metadata":{"id":"XL9AU7zTgP9n"},"source":["### **Toward Consistent, Verifiable, and Coherent Commonsense Reasoning in Large LMs**\n","\n","This notebook provides source code for our two papers in Findings of EMNLP 2021:\n","\n","\n","1.  Shane Storks, Qiaozi Gao, Yichi Zhang, and Joyce Y. Chai (2021). *Tiered Reasoning for Intuitive Physics: Toward Verifiable Commonsense Language Understanding.* Findings of EMNLP 2021.\n","2.   Shane Storks and Joyce Y. Chai (2021). *Beyond the Tip of the Iceberg: Assessing Coherence of Text Classifiers.* Findings of EMNLP 2021.\n","\n","*If you have any questions or problems, please open an issue on our [GitHub repo](https://github.com/sled-group/Verifiable-Coherent-NLU) or email Shane Storks.*\n","\n","***First, configure the execution mode by selecting a few settings (expand cell if needed):***\n","\n","\n"]},{"cell_type":"markdown","metadata":{"id":"ct4cd2_TFYDk"},"source":["   0. (Colab only) Insert the path in your Google Drive to the folder where this notebook is located."]},{"cell_type":"code","execution_count":null,"metadata":{"id":"Vq9-dXJXFWh3"},"outputs":[],"source":["DRIVE_PATH = 'drive/My Drive/path/to/TRIP/'"]},{"cell_type":"markdown","metadata":{"id":"yxzL0hhHAHzN"},"source":["1.   Model type (choose from BERT large, RoBERTa large, RoBERTa large + MNLI, DeBERTa base, and DeBERTa large).\n","\n","\n","\n","\n"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"FFNe6vlTaHsP"},"outputs":[],"source":["# mode = 'bert' # BERT large\n","mode = 'roberta' # RoBERTa large\n","# mode = 'roberta_mnli' # RoBERTa large pre-trained on MNLI\n","# mode = 'deberta' # DeBERTa base for training on TRIP\n","# mode = 'deberta_large' # DeBERTa large for training on CE and ART"]},{"cell_type":"markdown","metadata":{"id":"KbJ-XeY1aCpD"},"source":["2.   Name of the task we want to train or evaluate on. Set `debug` to `True` to run quick training/evaluation jobs on only a small amount of data."]},{"cell_type":"code","execution_count":null,"metadata":{"id":"iAQGu6JMa-o6"},"outputs":[],"source":["task_name = 'trip'\n","# task_name = 'ce'\n","# task_name = 'art'\n","\n","debug = False"]},{"cell_type":"markdown","metadata":{"id":"aoWKXfQBd435"},"source":["3.   (If training models) Training batch size, learning rate, and maximum number of epochs. Settings for results in the paper are provided as examples."]},{"cell_type":"code","execution_count":null,"metadata":{"id":"UyFFcZtkeKwT"},"outputs":[],"source":["config_batch_size = 1\n","config_lr = 1e-5 # Selected learning rate for best RoBERTa-based model in TRIP paper\n","config_epochs = 10"]},{"cell_type":"markdown","metadata":{"id":"hH9a70CTaGpG"},"source":["4.   (For training TRIP models only) Configure the loss weighting scheme for training models here. We provide the 4 modes from the paper as examples.\n"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"UvQEiNBSACak"},"outputs":[],"source":["# Loss weights for (attributes, preconditions, effects, conflicts, story choices)\n","if task_name != 'trip':\n","  print(\"We do not need a loss weighting scheme for %s dataset. Ignoring this cell.\" % task_name)\n","# loss_weights = [0.0, 0.4, 0.4, 0.1, 0.1] # \"All losses\"\n","loss_weights = [0.0, 0.4, 0.4, 0.2, 0.0] # \"Omit story choice loss\"\n","# loss_weights = [0.0, 0.4, 0.4, 0.0, 0.2] # \"Omit conflict detection loss\"\n","# loss_weights = [0.0, 0.0, 0.0, 0.5, 0.5] # \"Omit state classification losses\""]},{"cell_type":"markdown","metadata":{"id":"zmpchQTIg3HZ"},"source":["   5. (If evaluating models) Provide the name of the pre-trained model directory here. This should be the name of a directory within the *saved_models* directory, which should be located where this notebook is. Names of provided pre-trained model directories are listed."]},{"cell_type":"code","execution_count":null,"metadata":{"id":"W8tH7UMZhI1N"},"outputs":[],"source":["# TRIP, all losses\n","# eval_model_dir = 'bert-large-uncased_cloze_1_5e-06_4_0.0-0.4-0.4-0.1-0.1_tiered_pipeline_ablate_attributes_states-logits'\n","# eval_model_dir = 'roberta-large_cloze_1_1e-05_7_0.0-0.4-0.4-0.1-0.1_tiered_pipeline_ablate_attributes_states-logits'\n","# eval_model_dir = 'microsoft-deberta-base_cloze_1_5e-06_5_0.0-0.4-0.4-0.1-0.1_tiered_pipeline_ablate_attributes_states-logits'\n","\n","# TRIP, no story classification loss\n","# eval_model_dir = 'bert-large-uncased_cloze_1_5e-05_8_0.0-0.4-0.4-0.2-0.0_tiered_pipeline_ablate_attributes_states-logits'\n","eval_model_dir = 'roberta-large_cloze_1_1e-05_5_0.0-0.4-0.4-0.2-0.0_tiered_pipeline_lc_ablate_attributes_states-logits' # Best model trained in the TRIP paper\n","# eval_model_dir = 'microsoft-deberta-base_cloze_1_5e-05_5_0.0-0.4-0.4-0.2-0.0_tiered_pipeline_ablate_attributes_states-logits'\n","\n","# TRIP, no conflict detection loss\n","# eval_model_dir = 'bert-large-uncased_cloze_1_1e-06_1_0.0-0.4-0.4-0.0-0.2_tiered_pipeline_ablate_attributes_states-logits'\n","# eval_model_dir = 'roberta-large_cloze_1_5e-06_8_0.0-0.4-0.4-0.0-0.2_tiered_pipeline_ablate_attributes_states-logits'\n","# eval_model_dir = 'microsoft-deberta-base_cloze_1_1e-06_3_0.0-0.4-0.4-0.0-0.2_tiered_pipeline_ablate_attributes_states-logits'\n","\n","# TRIP, no physical state classification loss\n","# eval_model_dir = 'bert-large-uncased_cloze_1_1e-05_3_0.0-0.0-0.0-0.5-0.5_tiered_pipeline_ablate_attributes_states-logits'\n","# eval_model_dir = 'roberta-large_cloze_1_1e-06_7_0.0-0.0-0.0-0.5-0.5_tiered_pipeline_ablate_attributes_states-logits'\n","# eval_model_dir = 'microsoft-deberta-base_cloze_1_5e-06_9_0.0-0.0-0.0-0.5-0.5_tiered_pipeline_ablate_attributes_states-logits'\n","\n","# CE\n","# eval_model_dir = 'bert-large-uncased_ConvEnt_32_7.5e-06_7_xval'\n","# eval_model_dir = 'roberta-large_ConvEnt_32_7.5e-06_9_xval'\n","# eval_model_dir = 'roberta-large-mnli_ConvEnt_32_7.5e-06_7_xval'\n","# eval_model_dir = 'microsoft-deberta-large_ConvEnt_16_1e-05_9_xval'\n","\n","# ART\n","# eval_model_dir = 'bert-large-uncased_art_64_5e-06_8'\n","# eval_model_dir = 'roberta-large_art_64_2.5e-06_4'\n","# eval_model_dir = 'DeBERTa-deberta-large_art_32_1e-06_8'"]},{"cell_type":"markdown","metadata":{"id":"3GA5VS3Sfgfz"},"source":["**For more configuration options, scroll down to the Train Models > Configure Hyperparameters cell for the task you're working on.**"]},{"cell_type":"markdown","metadata":{"id":"qqvj34KhLL0k"},"source":["# Setup\n","Run this block every time when starting up the notebook. It will get Colab ready, preprocess the data, and load model packages and classes we'll need later. May take several minutes to run for the first time.\n","\n","**If you get a `ModuleNotFoundError` for the `www` code base, try the following:**\n","\n","\n","1.   Ensure the DRIVE_PATH is set properly above.\n","2.   (Colab only) Verify that this notebook has access to your Google Drive (click the folder icon on the left and then the Google Drive icon).\n","2.   Try to restart the runtime and refresh your browser window.\n","2.   (Colab only) If the problem persists, revoke access to Google Drive and re-enable it.\n","\n","\n","\n"]},{"cell_type":"markdown","metadata":{"id":"xm7qzKnAnbU9"},"source":["## Colab Setup\n","\n","Enable auto reloading of code libraries from Google Drive, set up connection to Google Drive, and import some packages. 🔌"]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":5,"status":"ok","timestamp":1631304597639,"user":{"displayName":"Shane Storks","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14Gh8CqjwUclCWWIoFwUB05rZNHwyd6nEIrTXC6RVXw=s64","userId":"00590987742306216924"},"user_tz":240},"id":"U8h8hUVaqySd","outputId":"6cd18c92-af82-4446-81e2-ef91e34d61f8"},"outputs":[{"name":"stdout","output_type":"stream","text":["The autoreload extension is already loaded. To reload it, use:\n","  %reload_ext autoreload\n"]}],"source":["%load_ext autoreload\n","%autoreload 2"]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":3429,"status":"ok","timestamp":1631304601236,"user":{"displayName":"Shane Storks","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14Gh8CqjwUclCWWIoFwUB05rZNHwyd6nEIrTXC6RVXw=s64","userId":"00590987742306216924"},"user_tz":240},"id":"P3dNQeNNnkHD","outputId":"a9eaffd8-a622-496b-c5e0-80d5728c382b"},"outputs":[{"name":"stdout","output_type":"stream","text":["Requirement already satisfied: jsonlines in /usr/local/lib/python3.7/dist-packages (2.0.0)\n"]}],"source":["import os\n","import json\n","import sys\n","import torch\n","import random\n","import numpy as np\n","import spacy\n","!pip install jsonlines\n","\n","sys.path.append(DRIVE_PATH)"]},{"cell_type":"markdown","metadata":{"id":"zQLB_Y-wSsfk"},"source":["## Model Setup\n","\n","Next, we'll load up the transformer model, tokenizer, etc. ⏳"]},{"cell_type":"markdown","metadata":{"id":"FoY37xIF-oP7"},"source":["### Install HuggingFace transformers and other dependencies"]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":13405,"status":"ok","timestamp":1631304614620,"user":{"displayName":"Shane Storks","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14Gh8CqjwUclCWWIoFwUB05rZNHwyd6nEIrTXC6RVXw=s64","userId":"00590987742306216924"},"user_tz":240},"id":"_rp3vUVjT9I4","outputId":"639357f3-18e8-4ab1-f382-073891b24621"},"outputs":[{"name":"stdout","output_type":"stream","text":["Requirement already satisfied: transformers==4.2.2 in /usr/local/lib/python3.7/dist-packages (4.2.2)\n","Requirement already satisfied: filelock in /usr/local/lib/python3.7/dist-packages (from transformers==4.2.2) (3.0.12)\n","Requirement already satisfied: requests in /usr/local/lib/python3.7/dist-packages (from transformers==4.2.2) (2.23.0)\n","Requirement already satisfied: sacremoses in /usr/local/lib/python3.7/dist-packages (from transformers==4.2.2) (0.0.45)\n","Requirement already satisfied: tqdm>=4.27 in /usr/local/lib/python3.7/dist-packages (from transformers==4.2.2) (4.62.0)\n","Requirement already satisfied: packaging in /usr/local/lib/python3.7/dist-packages (from transformers==4.2.2) (21.0)\n","Requirement already satisfied: tokenizers==0.9.4 in /usr/local/lib/python3.7/dist-packages (from transformers==4.2.2) (0.9.4)\n","Requirement already satisfied: importlib-metadata in /usr/local/lib/python3.7/dist-packages (from transformers==4.2.2) (4.6.4)\n","Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.7/dist-packages (from transformers==4.2.2) (2019.12.20)\n","Requirement already satisfied: numpy in /usr/local/lib/python3.7/dist-packages (from transformers==4.2.2) (1.19.5)\n","Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.7/dist-packages (from importlib-metadata->transformers==4.2.2) (3.5.0)\n","Requirement already satisfied: typing-extensions>=3.6.4 in /usr/local/lib/python3.7/dist-packages (from importlib-metadata->transformers==4.2.2) (3.7.4.3)\n","Requirement already satisfied: pyparsing>=2.0.2 in /usr/local/lib/python3.7/dist-packages (from packaging->transformers==4.2.2) (2.4.7)\n","Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests->transformers==4.2.2) (2021.5.30)\n","Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests->transformers==4.2.2) (3.0.4)\n","Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests->transformers==4.2.2) (2.10)\n","Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from requests->transformers==4.2.2) (1.24.3)\n","Requirement already satisfied: click in /usr/local/lib/python3.7/dist-packages (from sacremoses->transformers==4.2.2) (7.1.2)\n","Requirement already satisfied: six in /usr/local/lib/python3.7/dist-packages (from sacremoses->transformers==4.2.2) (1.15.0)\n","Requirement already satisfied: joblib in /usr/local/lib/python3.7/dist-packages (from sacremoses->transformers==4.2.2) (1.0.1)\n","Requirement already satisfied: sentencepiece in /usr/local/lib/python3.7/dist-packages (0.1.96)\n","Requirement already satisfied: torch==1.7.1 in /usr/local/lib/python3.7/dist-packages (1.7.1)\n","Requirement already satisfied: torchvision==0.8.2 in /usr/local/lib/python3.7/dist-packages (0.8.2)\n","Requirement already satisfied: torchaudio==0.7.2 in /usr/local/lib/python3.7/dist-packages (0.7.2)\n","Requirement already satisfied: typing-extensions in /usr/local/lib/python3.7/dist-packages (from torch==1.7.1) (3.7.4.3)\n","Requirement already satisfied: numpy in /usr/local/lib/python3.7/dist-packages (from torch==1.7.1) (1.19.5)\n","Requirement already satisfied: pillow>=4.1.1 in /usr/local/lib/python3.7/dist-packages (from torchvision==0.8.2) (7.1.2)\n","Requirement already satisfied: deberta in /usr/local/lib/python3.7/dist-packages (0.1.12)\n","Requirement already satisfied: numpy in /usr/local/lib/python3.7/dist-packages (from deberta) (1.19.5)\n","Requirement already satisfied: spacy in /usr/local/lib/python3.7/dist-packages (from deberta) (2.2.4)\n","Requirement already satisfied: sentencepiece in /usr/local/lib/python3.7/dist-packages (from deberta) (0.1.96)\n","Requirement already satisfied: scipy in /usr/local/lib/python3.7/dist-packages (from deberta) (1.4.1)\n","Requirement already satisfied: pytest in /usr/local/lib/python3.7/dist-packages (from deberta) (3.6.4)\n","Requirement already satisfied: ujson in /usr/local/lib/python3.7/dist-packages (from deberta) (4.1.0)\n","Requirement already satisfied: tqdm in /usr/local/lib/python3.7/dist-packages (from deberta) (4.62.0)\n","Requirement already satisfied: torch in /usr/local/lib/python3.7/dist-packages (from deberta) (1.7.1)\n","Requirement already satisfied: sklearn in /usr/local/lib/python3.7/dist-packages (from deberta) (0.0)\n","Requirement already satisfied: nltk in /usr/local/lib/python3.7/dist-packages (from deberta) (3.2.5)\n","Requirement already satisfied: seqeval in /usr/local/lib/python3.7/dist-packages (from deberta) (1.2.2)\n","Requirement already satisfied: regex in /usr/local/lib/python3.7/dist-packages (from deberta) (2019.12.20)\n","Requirement already satisfied: laser in /usr/local/lib/python3.7/dist-packages (from deberta) (0.0.5)\n","Requirement already satisfied: psutil in /usr/local/lib/python3.7/dist-packages (from deberta) (5.4.8)\n","Requirement already satisfied: six in /usr/local/lib/python3.7/dist-packages (from nltk->deberta) (1.15.0)\n","Requirement already satisfied: setuptools in /usr/local/lib/python3.7/dist-packages (from pytest->deberta) (57.4.0)\n","Requirement already satisfied: atomicwrites>=1.0 in /usr/local/lib/python3.7/dist-packages (from pytest->deberta) (1.4.0)\n","Requirement already satisfied: py>=1.5.0 in /usr/local/lib/python3.7/dist-packages (from pytest->deberta) (1.10.0)\n","Requirement already satisfied: more-itertools>=4.0.0 in /usr/local/lib/python3.7/dist-packages (from pytest->deberta) (8.8.0)\n","Requirement already satisfied: attrs>=17.4.0 in /usr/local/lib/python3.7/dist-packages (from pytest->deberta) (21.2.0)\n","Requirement already satisfied: pluggy<0.8,>=0.5 in /usr/local/lib/python3.7/dist-packages (from pytest->deberta) (0.7.1)\n","Requirement already satisfied: scikit-learn>=0.21.3 in /usr/local/lib/python3.7/dist-packages (from seqeval->deberta) (0.22.2.post1)\n","Requirement already satisfied: joblib>=0.11 in /usr/local/lib/python3.7/dist-packages (from scikit-learn>=0.21.3->seqeval->deberta) (1.0.1)\n","Requirement already satisfied: cymem<2.1.0,>=2.0.2 in /usr/local/lib/python3.7/dist-packages (from spacy->deberta) (2.0.5)\n","Requirement already satisfied: murmurhash<1.1.0,>=0.28.0 in /usr/local/lib/python3.7/dist-packages (from spacy->deberta) (1.0.5)\n","Requirement already satisfied: preshed<3.1.0,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from spacy->deberta) (3.0.5)\n","Requirement already satisfied: catalogue<1.1.0,>=0.0.7 in /usr/local/lib/python3.7/dist-packages (from spacy->deberta) (1.0.0)\n","Requirement already satisfied: blis<0.5.0,>=0.4.0 in /usr/local/lib/python3.7/dist-packages (from spacy->deberta) (0.4.1)\n","Requirement already satisfied: requests<3.0.0,>=2.13.0 in /usr/local/lib/python3.7/dist-packages (from spacy->deberta) (2.23.0)\n","Requirement already satisfied: srsly<1.1.0,>=1.0.2 in /usr/local/lib/python3.7/dist-packages (from spacy->deberta) (1.0.5)\n","Requirement already satisfied: wasabi<1.1.0,>=0.4.0 in /usr/local/lib/python3.7/dist-packages (from spacy->deberta) (0.8.2)\n","Requirement already satisfied: plac<1.2.0,>=0.9.6 in /usr/local/lib/python3.7/dist-packages (from spacy->deberta) (1.1.3)\n","Requirement already satisfied: thinc==7.4.0 in /usr/local/lib/python3.7/dist-packages (from spacy->deberta) (7.4.0)\n","Requirement already satisfied: importlib-metadata>=0.20 in /usr/local/lib/python3.7/dist-packages (from catalogue<1.1.0,>=0.0.7->spacy->deberta) (4.6.4)\n","Requirement already satisfied: typing-extensions>=3.6.4 in /usr/local/lib/python3.7/dist-packages (from importlib-metadata>=0.20->catalogue<1.1.0,>=0.0.7->spacy->deberta) (3.7.4.3)\n","Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.7/dist-packages (from importlib-metadata>=0.20->catalogue<1.1.0,>=0.0.7->spacy->deberta) (3.5.0)\n","Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests<3.0.0,>=2.13.0->spacy->deberta) (3.0.4)\n","Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests<3.0.0,>=2.13.0->spacy->deberta) (2.10)\n","Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests<3.0.0,>=2.13.0->spacy->deberta) (2021.5.30)\n","Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from requests<3.0.0,>=2.13.0->spacy->deberta) (1.24.3)\n"]}],"source":["!pip install 'transformers==4.2.2'\n","!pip install sentencepiece\n","!pip3 install torch==1.7.1 torchvision==0.8.2 torchaudio==0.7.2\n","!pip install deberta"]},{"cell_type":"markdown","metadata":{"id":"Z4LFuLhzAa2j"},"source":["### Get Model Components"]},{"cell_type":"markdown","metadata":{"id":"2ZhhgV9c__TU"},"source":["Specify which model parameters from transformers we want to use:"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"uA6XunCb_gd9"},"outputs":[],"source":["if task_name in ['trip', 'ce']:\n","  multiple_choice = False\n","elif task_name == 'art':\n","  multiple_choice = True\n","else:\n","  raise ValueError(\"Task name should be set to 'trip', 'ce', or 'art' in the first cell of the notebook!\")\n","\n","if mode == 'bert':\n","  model_name = 'bert-large-uncased'\n","elif mode == 'roberta':\n","  model_name = 'roberta-large'\n","elif mode == 'roberta_mnli':\n","  model_name = 'roberta-large-mnli'\n","elif mode == 'deberta':\n","  model_name = 'microsoft/deberta-base'\n","elif mode == 'deberta_large':\n","  model_name = 'microsoft/deberta-large'"]},{"cell_type":"markdown","metadata":{"id":"hgFIl1MJ-185"},"source":["Load the tokenizer:"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"etkxf75f-9Gj"},"outputs":[],"source":["from transformers import BertTokenizer, RobertaTokenizer, DebertaTokenizer, AlbertTokenizer, T5Tokenizer, GPT2Tokenizer\n","\n","from DeBERTa import deberta\n","if mode in ['bert']:\n","  tokenizer_class = BertTokenizer\n","elif mode in ['roberta', 'roberta_mnli']:\n","  tokenizer_class = RobertaTokenizer\n","elif mode in ['deberta', 'deberta_large']:\n","  tokenizer_class = DebertaTokenizer\n","\n","tokenizer = tokenizer_class.from_pretrained(model_name, \n","                                                do_lower_case = False, \n","                                                cache_dir=os.path.join(DRIVE_PATH, 'cache'))"]},{"cell_type":"markdown","metadata":{"id":"d0iYZG6bBGIf"},"source":["Load the model and optimizer:\n","\n"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"1PxFghcDBPm_"},"outputs":[],"source":["from transformers import BertForSequenceClassification, RobertaForSequenceClassification, DebertaForSequenceClassification, AlbertForSequenceClassification, AdamW\n","from transformers import BertForMultipleChoice, RobertaForMultipleChoice, AlbertForMultipleChoice, DebertaModel\n","from transformers import BertModel, RobertaModel, AlbertModel, DebertaModel, T5Model, T5EncoderModel, GPT2Model\n","from transformers import RobertaForMaskedLM\n","from transformers import BertConfig, RobertaConfig, DebertaConfig, AlbertConfig, T5Config, GPT2Config\n","from www.model.transformers_ext import DebertaForMultipleChoice\n","from torch.optim import Adam\n","if not multiple_choice:\n","  if mode == 'bert':\n","    model_class = BertForSequenceClassification\n","    config_class = BertConfig\n","    emb_class = BertModel\n","  elif mode in ['roberta', 'roberta_mnli']:\n","    model_class = RobertaForSequenceClassification\n","    config_class = RobertaConfig\n","    emb_class = RobertaModel\n","    lm_class = RobertaForMaskedLM\n","  elif mode in ['deberta', 'deberta_large']:\n","    model_class = DebertaForSequenceClassification\n","    config_class = DebertaConfig\n","    emb_class = DebertaModel\n","else:\n","  if mode == 'bert':\n","    model_class = BertForMultipleChoice\n","    config_class = BertConfig\n","    emb_class = BertModel    \n","  elif mode in ['roberta', 'roberta_mnli']:\n","    model_class = RobertaForMultipleChoice\n","    config_class = RobertaConfig\n","    emb_class = RobertaModel\n","    lm_class = RobertaForMaskedLM\n","  elif mode in ['deberta', 'deberta_large']:\n","    model_class = DebertaForMultipleChoice\n","    config_class = DebertaConfig\n","    emb_class = DebertaModel"]},{"cell_type":"markdown","metadata":{"id":"QzFnAVtuUmpQ"},"source":["## Data Setup\n","\n","Preprocess the dataset."]},{"cell_type":"markdown","metadata":{"id":"lKY0hTEgnQgB"},"source":["### Preprocessing\n","\n","Construct the dataset from the .txt files collected from AMT. Save a backup copy in Drive."]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":4250,"status":"ok","timestamp":1631304619289,"user":{"displayName":"Shane Storks","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14Gh8CqjwUclCWWIoFwUB05rZNHwyd6nEIrTXC6RVXw=s64","userId":"00590987742306216924"},"user_tz":240},"id":"aE-LOkJ4nWuu","outputId":"61921ed0-f763-4280-d0ea-5979458d51ca"},"outputs":[{"name":"stdout","output_type":"stream","text":["Preprocessed examples:\n","{\n","  story_id: \n","    13,\n","  worker_id: \n","    A32W24TWSWXW,\n","  type: \n","    None,\n","  idx: \n","    None,\n","  aug: \n","    False,\n","  actor: \n","    John,\n","  location: \n","    kitchen,\n","  objects: \n","    cabinet, counter, knife, pan, potato, pizza,\n","  sentences: \n","    [\n","      John was getting the snacks ready for the party.\n","      John opened the cabinet, took out a pan and put it on the counter.\n","      John opened the fridge and got out the pizza.\n","      John put the pizza on the pan and put them into the oven.\n","      John took a knife and cut the hot pizza in eight slices.\n","    ],\n","  length: \n","    5,\n","  example_id: \n","    13,\n","  plausible: \n","    True,\n","  breakpoint: \n","    -1,\n","  confl_sents: \n","    [],\n","  confl_pairs: \n","    [],\n","  states: \n","    [\n","      {'h_location': [['John', 0]], 'conscious': [['John', 2]], 'wearing': [['John', 0]], 'h_wet': [['John', 0]], 'hygiene': [['John', 0]], 'location': [['snacks', 0], ['party', 0]], 'exist': [['snacks', 4], ['party', 2]], 'clean': [['snacks', 0], ['party', 0]], 'power': [['snacks', 0], ['party', 0]], 'functional': [['snacks', 2], ['party', 2]], 'pieces': [['snacks', 0], ['party', 0]], 'wet': [['snacks', 0], ['party', 0]], 'open': [['snacks', 0], ['party', 0]], 'temperature': [['snacks', 0], ['party', 0]], 'solid': [['snacks', 0], ['party', 0]], 'contain': [['snacks', 0], ['party', 0]], 'running': [['snacks', 0], ['party', 0]], 'moveable': [['snacks', 2], ['party', 2]], 'mixed': [['snacks', 0], ['party', 0]], 'edible': [['snacks', 0], ['party', 0]]}\n","      {'h_location': [['John', 0]], 'conscious': [['John', 2]], 'wearing': [['John', 0]], 'h_wet': [['John', 0]], 'hygiene': [['John', 0]], 'location': [['counter', 0], ['pan', 7], ['cabinet', 0]], 'exist': [['counter', 2], ['pan', 2], ['cabinet', 2]], 'clean': [['counter', 0], ['pan', 0], ['cabinet', 0]], 'power': [['counter', 0], ['pan', 0], ['cabinet', 0]], 'functional': [['counter', 2], ['pan', 2], ['cabinet', 2]], 'pieces': [['counter', 0], ['pan', 0], ['cabinet', 0]], 'wet': [['counter', 0], ['pan', 0], ['cabinet', 0]], 'open': [['counter', 0], ['pan', 0], ['cabinet', 0]], 'temperature': [['counter', 0], ['pan', 0], ['cabinet', 0]], 'solid': [['counter', 0], ['pan', 0], ['cabinet', 0]], 'contain': [['counter', 6], ['pan', 0], ['cabinet', 8]], 'running': [['counter', 0], ['pan', 0], ['cabinet', 0]], 'moveable': [['counter', 0], ['pan', 2], ['cabinet', 0]], 'mixed': [['counter', 0], ['pan', 0], ['cabinet', 0]], 'edible': [['counter', 0], ['pan', 0], ['cabinet', 0]]}\n","      {'h_location': [['John', 0]], 'conscious': [['John', 2]], 'wearing': [['John', 0]], 'h_wet': [['John', 0]], 'hygiene': [['John', 0]], 'location': [['fridge', 0], ['pizza', 7]], 'exist': [['fridge', 2], ['pizza', 2]], 'clean': [['fridge', 0], ['pizza', 0]], 'power': [['fridge', 2], ['pizza', 0]], 'functional': [['fridge', 2], ['pizza', 2]], 'pieces': [['fridge', 0], ['pizza', 0]], 'wet': [['fridge', 0], ['pizza', 0]], 'open': [['fridge', 4], ['pizza', 0]], 'temperature': [['fridge', 0], ['pizza', 1]], 'solid': [['fridge', 0], ['pizza', 0]], 'contain': [['fridge', 8], ['pizza', 0]], 'running': [['fridge', 2], ['pizza', 0]], 'moveable': [['fridge', 2], ['pizza', 2]], 'mixed': [['fridge', 0], ['pizza', 0]], 'edible': [['fridge', 0], ['pizza', 0]]}\n","      {'h_location': [['John', 0]], 'conscious': [['John', 2]], 'wearing': [['John', 0]], 'h_wet': [['John', 0]], 'hygiene': [['John', 0]], 'location': [['oven', 0], ['pizza', 3], ['pan', 6]], 'exist': [['oven', 2], ['pizza', 2], ['pan', 2]], 'clean': [['oven', 0], ['pizza', 0], ['pan', 0]], 'power': [['oven', 2], ['pizza', 0], ['pan', 0]], 'functional': [['oven', 2], ['pizza', 2], ['pan', 2]], 'pieces': [['oven', 0], ['pizza', 0], ['pan', 0]], 'wet': [['oven', 0], ['pizza', 0], ['pan', 0]], 'open': [['oven', 8], ['pizza', 0], ['pan', 0]], 'temperature': [['oven', 2], ['pizza', 0], ['pan', 0]], 'solid': [['oven', 0], ['pizza', 0], ['pan', 0]], 'contain': [['oven', 6], ['pizza', 0], ['pan', 4]], 'running': [['oven', 0], ['pizza', 0], ['pan', 0]], 'moveable': [['oven', 0], ['pizza', 2], ['pan', 2]], 'mixed': [['oven', 0], ['pizza', 0], ['pan', 0]], 'edible': [['oven', 0], ['pizza', 0], ['pan', 0]]}\n","      {'h_location': [['John', 0]], 'conscious': [['John', 2]], 'wearing': [['John', 0]], 'h_wet': [['John', 0]], 'hygiene': [['John', 0]], 'location': [['knife', 2], ['slices', 2], ['hot pizza', 0]], 'exist': [['knife', 2], ['slices', 2], ['hot pizza', 2]], 'clean': [['knife', 0], ['slices', 0], ['hot pizza', 0]], 'power': [['knife', 0], ['slices', 0], ['hot pizza', 0]], 'functional': [['knife', 2], ['slices', 2], ['hot pizza', 2]], 'pieces': [['knife', 0], ['slices', 0], ['hot pizza', 4]], 'wet': [['knife', 0], ['slices', 0], ['hot pizza', 0]], 'open': [['knife', 0], ['slices', 0], ['hot pizza', 0]], 'temperature': [['knife', 0], ['slices', 0], ['hot pizza', 2]], 'solid': [['knife', 0], ['slices', 0], ['hot pizza', 0]], 'contain': [['knife', 0], ['slices', 0], ['hot pizza', 0]], 'running': [['knife', 0], ['slices', 0], ['hot pizza', 0]], 'moveable': [['knife', 2], ['slices', 2], ['hot pizza', 2]], 'mixed': [['knife', 0], ['slices', 0], ['hot pizza', 0]], 'edible': [['knife', 0], ['slices', 0], ['hot pizza', 0]]}\n","    ],\n","}\n","\n","\n","{\n","  story_id: \n","    13,\n","  worker_id: \n","    A32W24TWSWXW,\n","  type: \n","    cloze,\n","  idx: \n","    0,\n","  aug: \n","    False,\n","  actor: \n","    John,\n","  location: \n","    kitchen,\n","  objects: \n","    cabinet, counter, knife, pan, potato, pizza,\n","  sentences: \n","    [\n","      John was getting the snacks ready for the party.\n","      John opened the cabinet, took out a pan and put it on the counter.\n","      John opened the fridge and got out the pizza.\n","      John put the pizza on the pan and put them into the oven.\n","      John called the pizza joint to deliver a pizza.\n","    ],\n","  length: \n","    5,\n","  example_id: \n","    13-C0,\n","  plausible: \n","    False,\n","  breakpoint: \n","    4,\n","  confl_sents: \n","    [\n","      2\n","      3\n","    ],\n","  confl_pairs: \n","    [\n","      [2, 4]\n","      [3, 4]\n","    ],\n","  states: \n","    [\n","      {'h_location': [['John', 0]], 'conscious': [['John', 2]], 'wearing': [['John', 0]], 'h_wet': [['John', 0]], 'hygiene': [['John', 0]], 'location': [['snacks', 0], ['party', 0]], 'exist': [['snacks', 4], ['party', 2]], 'clean': [['snacks', 0], ['party', 0]], 'power': [['snacks', 0], ['party', 0]], 'functional': [['snacks', 2], ['party', 2]], 'pieces': [['snacks', 0], ['party', 0]], 'wet': [['snacks', 0], ['party', 0]], 'open': [['snacks', 0], ['party', 0]], 'temperature': [['snacks', 0], ['party', 0]], 'solid': [['snacks', 0], ['party', 0]], 'contain': [['snacks', 0], ['party', 0]], 'running': [['snacks', 0], ['party', 0]], 'moveable': [['snacks', 2], ['party', 2]], 'mixed': [['snacks', 0], ['party', 0]], 'edible': [['snacks', 0], ['party', 0]]}\n","      {'h_location': [['John', 0]], 'conscious': [['John', 2]], 'wearing': [['John', 0]], 'h_wet': [['John', 0]], 'hygiene': [['John', 0]], 'location': [['counter', 0], ['pan', 7], ['cabinet', 0]], 'exist': [['counter', 2], ['pan', 2], ['cabinet', 2]], 'clean': [['counter', 0], ['pan', 0], ['cabinet', 0]], 'power': [['counter', 0], ['pan', 0], ['cabinet', 0]], 'functional': [['counter', 2], ['pan', 2], ['cabinet', 2]], 'pieces': [['counter', 0], ['pan', 0], ['cabinet', 0]], 'wet': [['counter', 0], ['pan', 0], ['cabinet', 0]], 'open': [['counter', 0], ['pan', 0], ['cabinet', 0]], 'temperature': [['counter', 0], ['pan', 0], ['cabinet', 0]], 'solid': [['counter', 0], ['pan', 0], ['cabinet', 0]], 'contain': [['counter', 6], ['pan', 0], ['cabinet', 8]], 'running': [['counter', 0], ['pan', 0], ['cabinet', 0]], 'moveable': [['counter', 0], ['pan', 2], ['cabinet', 0]], 'mixed': [['counter', 0], ['pan', 0], ['cabinet', 0]], 'edible': [['counter', 0], ['pan', 0], ['cabinet', 0]]}\n","      {'h_location': [['John', 0]], 'conscious': [['John', 2]], 'wearing': [['John', 0]], 'h_wet': [['John', 0]], 'hygiene': [['John', 0]], 'location': [['fridge', 0], ['pizza', 7]], 'exist': [['fridge', 2], ['pizza', 2]], 'clean': [['fridge', 0], ['pizza', 0]], 'power': [['fridge', 2], ['pizza', 0]], 'functional': [['fridge', 2], ['pizza', 2]], 'pieces': [['fridge', 0], ['pizza', 0]], 'wet': [['fridge', 0], ['pizza', 0]], 'open': [['fridge', 4], ['pizza', 0]], 'temperature': [['fridge', 0], ['pizza', 1]], 'solid': [['fridge', 0], ['pizza', 0]], 'contain': [['fridge', 8], ['pizza', 0]], 'running': [['fridge', 2], ['pizza', 0]], 'moveable': [['fridge', 2], ['pizza', 2]], 'mixed': [['fridge', 0], ['pizza', 0]], 'edible': [['fridge', 0], ['pizza', 0]]}\n","      {'h_location': [['John', 0]], 'conscious': [['John', 2]], 'wearing': [['John', 0]], 'h_wet': [['John', 0]], 'hygiene': [['John', 0]], 'location': [['oven', 0], ['pizza', 3], ['pan', 6]], 'exist': [['oven', 2], ['pizza', 2], ['pan', 2]], 'clean': [['oven', 0], ['pizza', 0], ['pan', 0]], 'power': [['oven', 2], ['pizza', 0], ['pan', 0]], 'functional': [['oven', 2], ['pizza', 2], ['pan', 2]], 'pieces': [['oven', 0], ['pizza', 0], ['pan', 0]], 'wet': [['oven', 0], ['pizza', 0], ['pan', 0]], 'open': [['oven', 8], ['pizza', 0], ['pan', 0]], 'temperature': [['oven', 2], ['pizza', 0], ['pan', 0]], 'solid': [['oven', 0], ['pizza', 0], ['pan', 0]], 'contain': [['oven', 6], ['pizza', 0], ['pan', 4]], 'running': [['oven', 0], ['pizza', 0], ['pan', 0]], 'moveable': [['oven', 0], ['pizza', 2], ['pan', 2]], 'mixed': [['oven', 0], ['pizza', 0], ['pan', 0]], 'edible': [['oven', 0], ['pizza', 0], ['pan', 0]]}\n","      {'h_location': [['John', 0]], 'conscious': [['John', 2]], 'wearing': [['John', 0]], 'h_wet': [['John', 0]], 'hygiene': [['John', 0]], 'location': [['pizza', 0]], 'exist': [['pizza', 4]], 'clean': [['pizza', 0]], 'power': [['pizza', 0]], 'functional': [['pizza', 2]], 'pieces': [['pizza', 0]], 'wet': [['pizza', 0]], 'open': [['pizza', 0]], 'temperature': [['pizza', 0]], 'solid': [['pizza', 0]], 'contain': [['pizza', 0]], 'running': [['pizza', 0]], 'moveable': [['pizza', 2]], 'mixed': [['pizza', 0]], 'edible': [['pizza', 0]]}\n","    ],\n","}\n","\n","\n","{\n","  story_id: \n","    13,\n","  worker_id: \n","    A32W24TWSWXW,\n","  type: \n","    order,\n","  idx: \n","    2,\n","  aug: \n","    False,\n","  actor: \n","    John,\n","  location: \n","    kitchen,\n","  objects: \n","    cabinet, counter, knife, pan, potato, pizza,\n","  sentences: \n","    [\n","      John was getting the snacks ready for the party.\n","      John opened the cabinet, took out a pan and put it on the counter.\n","      John put the pizza on the pan and put them into the oven.\n","      John opened the fridge and got out the pizza.\n","      John took a knife and cut the hot pizza in eight slices.\n","    ],\n","  length: \n","    5,\n","  example_id: \n","    13-O2,\n","  plausible: \n","    False,\n","  breakpoint: \n","    3,\n","  confl_sents: \n","    [\n","      2\n","    ],\n","  confl_pairs: \n","    [],\n","  states: \n","    [\n","      {'h_location': [['John', 0]], 'conscious': [['John', 2]], 'wearing': [['John', 0]], 'h_wet': [['John', 0]], 'hygiene': [['John', 0]], 'location': [['snacks', 0], ['party', 0]], 'exist': [['snacks', 4], ['party', 2]], 'clean': [['snacks', 0], ['party', 0]], 'power': [['snacks', 0], ['party', 0]], 'functional': [['snacks', 2], ['party', 2]], 'pieces': [['snacks', 0], ['party', 0]], 'wet': [['snacks', 0], ['party', 0]], 'open': [['snacks', 0], ['party', 0]], 'temperature': [['snacks', 0], ['party', 0]], 'solid': [['snacks', 0], ['party', 0]], 'contain': [['snacks', 0], ['party', 0]], 'running': [['snacks', 0], ['party', 0]], 'moveable': [['snacks', 2], ['party', 2]], 'mixed': [['snacks', 0], ['party', 0]], 'edible': [['snacks', 0], ['party', 0]]}\n","      {'h_location': [['John', 0]], 'conscious': [['John', 2]], 'wearing': [['John', 0]], 'h_wet': [['John', 0]], 'hygiene': [['John', 0]], 'location': [['counter', 0], ['pan', 7], ['cabinet', 0]], 'exist': [['counter', 2], ['pan', 2], ['cabinet', 2]], 'clean': [['counter', 0], ['pan', 0], ['cabinet', 0]], 'power': [['counter', 0], ['pan', 0], ['cabinet', 0]], 'functional': [['counter', 2], ['pan', 2], ['cabinet', 2]], 'pieces': [['counter', 0], ['pan', 0], ['cabinet', 0]], 'wet': [['counter', 0], ['pan', 0], ['cabinet', 0]], 'open': [['counter', 0], ['pan', 0], ['cabinet', 0]], 'temperature': [['counter', 0], ['pan', 0], ['cabinet', 0]], 'solid': [['counter', 0], ['pan', 0], ['cabinet', 0]], 'contain': [['counter', 6], ['pan', 0], ['cabinet', 8]], 'running': [['counter', 0], ['pan', 0], ['cabinet', 0]], 'moveable': [['counter', 0], ['pan', 2], ['cabinet', 0]], 'mixed': [['counter', 0], ['pan', 0], ['cabinet', 0]], 'edible': [['counter', 0], ['pan', 0], ['cabinet', 0]]}\n","      {'h_location': [['John', 0]], 'conscious': [['John', 2]], 'wearing': [['John', 0]], 'h_wet': [['John', 0]], 'hygiene': [['John', 0]], 'location': [['oven', 0], ['pizza', 3], ['pan', 6]], 'exist': [['oven', 2], ['pizza', 2], ['pan', 2]], 'clean': [['oven', 0], ['pizza', 0], ['pan', 0]], 'power': [['oven', 2], ['pizza', 0], ['pan', 0]], 'functional': [['oven', 2], ['pizza', 2], ['pan', 2]], 'pieces': [['oven', 0], ['pizza', 0], ['pan', 0]], 'wet': [['oven', 0], ['pizza', 0], ['pan', 0]], 'open': [['oven', 8], ['pizza', 0], ['pan', 0]], 'temperature': [['oven', 2], ['pizza', 0], ['pan', 0]], 'solid': [['oven', 0], ['pizza', 0], ['pan', 0]], 'contain': [['oven', 6], ['pizza', 0], ['pan', 4]], 'running': [['oven', 0], ['pizza', 0], ['pan', 0]], 'moveable': [['oven', 0], ['pizza', 2], ['pan', 2]], 'mixed': [['oven', 0], ['pizza', 0], ['pan', 0]], 'edible': [['oven', 0], ['pizza', 0], ['pan', 0]]}\n","      {'h_location': [['John', 0]], 'conscious': [['John', 2]], 'wearing': [['John', 0]], 'h_wet': [['John', 0]], 'hygiene': [['John', 0]], 'location': [['fridge', 0], ['pizza', 7]], 'exist': [['fridge', 2], ['pizza', 2]], 'clean': [['fridge', 0], ['pizza', 0]], 'power': [['fridge', 2], ['pizza', 0]], 'functional': [['fridge', 2], ['pizza', 2]], 'pieces': [['fridge', 0], ['pizza', 0]], 'wet': [['fridge', 0], ['pizza', 0]], 'open': [['fridge', 4], ['pizza', 0]], 'temperature': [['fridge', 0], ['pizza', 1]], 'solid': [['fridge', 0], ['pizza', 0]], 'contain': [['fridge', 8], ['pizza', 0]], 'running': [['fridge', 2], ['pizza', 0]], 'moveable': [['fridge', 2], ['pizza', 2]], 'mixed': [['fridge', 0], ['pizza', 0]], 'edible': [['fridge', 0], ['pizza', 0]]}\n","      {'h_location': [['John', 0]], 'conscious': [['John', 2]], 'wearing': [['John', 0]], 'h_wet': [['John', 0]], 'hygiene': [['John', 0]], 'location': [['knife', 2], ['slices', 2], ['hot pizza', 0]], 'exist': [['knife', 2], ['slices', 2], ['hot pizza', 2]], 'clean': [['knife', 0], ['slices', 0], ['hot pizza', 0]], 'power': [['knife', 0], ['slices', 0], ['hot pizza', 0]], 'functional': [['knife', 2], ['slices', 2], ['hot pizza', 2]], 'pieces': [['knife', 0], ['slices', 0], ['hot pizza', 4]], 'wet': [['knife', 0], ['slices', 0], ['hot pizza', 0]], 'open': [['knife', 0], ['slices', 0], ['hot pizza', 0]], 'temperature': [['knife', 0], ['slices', 0], ['hot pizza', 2]], 'solid': [['knife', 0], ['slices', 0], ['hot pizza', 0]], 'contain': [['knife', 0], ['slices', 0], ['hot pizza', 0]], 'running': [['knife', 0], ['slices', 0], ['hot pizza', 0]], 'moveable': [['knife', 2], ['slices', 2], ['hot pizza', 2]], 'mixed': [['knife', 0], ['slices', 0], ['hot pizza', 0]], 'edible': [['knife', 0], ['slices', 0], ['hot pizza', 0]]}\n","    ],\n","}\n","\n","\n","{\n","  story_id: \n","    33,\n","  worker_id: \n","    A1F01FVEPYCPHO,\n","  type: \n","    None,\n","  idx: \n","    None,\n","  aug: \n","    False,\n","  actor: \n","    Mary,\n","  location: \n","    bathroom,\n","  objects: \n","    washing machine, cabinet, toothpaste, bleach, socks, mirror,\n","  sentences: \n","    [\n","      Mary took off her socks.\n","      Mary put the socks in the washing machine.\n","      Mary opened the cabinet.\n","      Mary took out the toothbrush and toothpaste.\n","      Mary brushed her teeth while looking in the mirror.\n","    ],\n","  length: \n","    5,\n","  example_id: \n","    33,\n","  plausible: \n","    True,\n","  breakpoint: \n","    -1,\n","  confl_sents: \n","    [],\n","  confl_pairs: \n","    [],\n","  states: \n","    [\n","      {'h_location': [['Mary', 0]], 'conscious': [['Mary', 2]], 'wearing': [['Mary', 8]], 'h_wet': [['Mary', 0]], 'hygiene': [['Mary', 0]], 'location': [['socks', 5]], 'exist': [['socks', 2]], 'clean': [['socks', 0]], 'power': [['socks', 0]], 'functional': [['socks', 2]], 'pieces': [['socks', 0]], 'wet': [['socks', 0]], 'open': [['socks', 0]], 'temperature': [['socks', 0]], 'solid': [['socks', 0]], 'contain': [['socks', 0]], 'running': [['socks', 0]], 'moveable': [['socks', 2]], 'mixed': [['socks', 0]], 'edible': [['socks', 0]]}\n","      {'h_location': [['Mary', 0]], 'conscious': [['Mary', 2]], 'wearing': [['Mary', 0]], 'h_wet': [['Mary', 0]], 'hygiene': [['Mary', 0]], 'location': [['socks', 6], ['washing machine', 0]], 'exist': [['socks', 2], ['washing machine', 2]], 'clean': [['socks', 0], ['washing machine', 0]], 'power': [['socks', 0], ['washing machine', 0]], 'functional': [['socks', 2], ['washing machine', 2]], 'pieces': [['socks', 0], ['washing machine', 0]], 'wet': [['socks', 0], ['washing machine', 0]], 'open': [['socks', 0], ['washing machine', 8]], 'temperature': [['socks', 0], ['washing machine', 0]], 'solid': [['socks', 0], ['washing machine', 0]], 'contain': [['socks', 0], ['washing machine', 6]], 'running': [['socks', 0], ['washing machine', 0]], 'moveable': [['socks', 2], ['washing machine', 2]], 'mixed': [['socks', 0], ['washing machine', 0]], 'edible': [['socks', 0], ['washing machine', 0]]}\n","      {'h_location': [['Mary', 0]], 'conscious': [['Mary', 2]], 'wearing': [['Mary', 0]], 'h_wet': [['Mary', 0]], 'hygiene': [['Mary', 0]], 'location': [['cabinet', 0]], 'exist': [['cabinet', 2]], 'clean': [['cabinet', 0]], 'power': [['cabinet', 0]], 'functional': [['cabinet', 2]], 'pieces': [['cabinet', 0]], 'wet': [['cabinet', 0]], 'open': [['cabinet', 4]], 'temperature': [['cabinet', 0]], 'solid': [['cabinet', 0]], 'contain': [['cabinet', 0]], 'running': [['cabinet', 0]], 'moveable': [['cabinet', 2]], 'mixed': [['cabinet', 0]], 'edible': [['cabinet', 0]]}\n","      {'h_location': [['Mary', 0]], 'conscious': [['Mary', 2]], 'wearing': [['Mary', 0]], 'h_wet': [['Mary', 0]], 'hygiene': [['Mary', 0]], 'location': [['toothbrush', 7], ['toothpaste', 7]], 'exist': [['toothbrush', 2], ['toothpaste', 2]], 'clean': [['toothbrush', 0], ['toothpaste', 0]], 'power': [['toothbrush', 0], ['toothpaste', 0]], 'functional': [['toothbrush', 2], ['toothpaste', 2]], 'pieces': [['toothbrush', 0], ['toothpaste', 0]], 'wet': [['toothbrush', 0], ['toothpaste', 0]], 'open': [['toothbrush', 0], ['toothpaste', 0]], 'temperature': [['toothbrush', 0], ['toothpaste', 0]], 'solid': [['toothbrush', 0], ['toothpaste', 0]], 'contain': [['toothbrush', 0], ['toothpaste', 0]], 'running': [['toothbrush', 0], ['toothpaste', 0]], 'moveable': [['toothbrush', 2], ['toothpaste', 2]], 'mixed': [['toothbrush', 0], ['toothpaste', 0]], 'edible': [['toothbrush', 0], ['toothpaste', 0]]}\n","      {'h_location': [['Mary', 0]], 'conscious': [['Mary', 2]], 'wearing': [['Mary', 0]], 'h_wet': [['Mary', 0]], 'hygiene': [['Mary', 6]], 'location': [['mirror', 0], ['teeth', 0]], 'exist': [['mirror', 2], ['teeth', 2]], 'clean': [['mirror', 0], ['teeth', 6]], 'power': [['mirror', 0], ['teeth', 0]], 'functional': [['mirror', 2], ['teeth', 2]], 'pieces': [['mirror', 0], ['teeth', 0]], 'wet': [['mirror', 0], ['teeth', 0]], 'open': [['mirror', 0], ['teeth', 0]], 'temperature': [['mirror', 0], ['teeth', 0]], 'solid': [['mirror', 0], ['teeth', 0]], 'contain': [['mirror', 0], ['teeth', 0]], 'running': [['mirror', 0], ['teeth', 0]], 'moveable': [['mirror', 2], ['teeth', 0]], 'mixed': [['mirror', 0], ['teeth', 0]], 'edible': [['mirror', 0], ['teeth', 0]]}\n","    ],\n","}\n","\n","\n"]}],"source":["from www.utils import print_dict\n","\n","partitions = ['train', 'dev', 'test']\n","subtasks = ['cloze', 'order']\n","\n","# We can split the data into multiple json files later\n","data_file = os.path.join(DRIVE_PATH, 'all_data/www.json')\n","with open(data_file, 'r') as f:\n","  dataset = json.load(f)\n","\n","print('Preprocessed examples:')\n","for ex_idx in [0,1,5,10]:\n","  ex = dataset['dev'][list(dataset['dev'].keys())[ex_idx]]\n","  print_dict(ex)"]},{"cell_type":"markdown","metadata":{"id":"k_0WsycpFMdb"},"source":["### Data Filtering and Sampling\n","Since there is a big imbalance between plausible/implausible class labels, we will upsample the plausible stories.\n","\n","For now, we will also break the dataset into two sub-datasets: cloze and ordering.\n","\n"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"-twYzY5rF1Mi"},"outputs":[],"source":["cloze_dataset = {p: [] for p in dataset}\n","order_dataset = {p: [] for p in dataset}\n","\n","for p in dataset:\n","  for exid in dataset[p]:\n","    ex = dataset[p][exid]\n","\n","    if ex['type'] == None:\n","      continue\n","    \n","    ex_plaus = dataset[p][str(ex['story_id'])]\n","\n","    if ex['type'] == 'cloze':\n","      cloze_dataset[p].append(ex)\n","      cloze_dataset[p].append(ex_plaus) # For every implausible story, add a copy of its corresponding plausible story\n","\n","    # Exclude augmented ordering examples from dev and test, since the breakpoints aren't always accurate in those\n","    elif ex['type'] == 'order' and not (p != 'train' and ex['aug']): \n","      order_dataset[p].append(ex)\n","      order_dataset[p].append(ex_plaus)"]},{"cell_type":"markdown","metadata":{"id":"Cz5tcmScJrka"},"source":["\n","\n","### Convert TRIP to Two-Story Classification Task\n","\n","Ready the TRIP dataset for two-story classification."]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":3982,"status":"ok","timestamp":1631304623775,"user":{"displayName":"Shane Storks","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14Gh8CqjwUclCWWIoFwUB05rZNHwyd6nEIrTXC6RVXw=s64","userId":"00590987742306216924"},"user_tz":240},"id":"Af976ygKJv7W","outputId":"8335b25d-0f7b-4404-db5c-2dbc4f65b2f8"},"outputs":[{"name":"stdout","output_type":"stream","text":["Cloze label distribution (train):\n","[(1, 400), (0, 399)]\n","Cloze label distribution (dev):\n","[(0, 161), (1, 161)]\n","Cloze label distribution (test):\n","[(1, 176), (0, 175)]\n","{\n","  example_id: \n","    0-C0,\n","  stories: \n","    [\n","      {'story_id': 0, 'worker_id': 'A1F01FVEPYCPHO', 'type': 'cloze', 'idx': 0, 'aug': False, 'actor': 'Tom', 'location': 'kitchen', 'objects': 'dustbin, microwave, pan, plate, cereal, soup', 'sentences': ['Tom bought a new dustbin for the kitchen.', 'Tom threw a broken plate in the dustbin.', 'Tom got some soup from the fridge.', 'Tom put the soup in the microwave.', 'Tom ate the cold soup.'], 'length': 5, 'example_id': '0-C0', 'plausible': False, 'breakpoint': 4, 'confl_sents': [3], 'confl_pairs': [[3, 4]], 'states': [{'h_location': [['Tom', 0]], 'conscious': [['Tom', 2]], 'wearing': [['Tom', 0]], 'h_wet': [['Tom', 0]], 'hygiene': [['Tom', 0]], 'location': [['dustbin', 6]], 'exist': [['dustbin', 4]], 'clean': [['dustbin', 0]], 'power': [['dustbin', 0]], 'functional': [['dustbin', 2]], 'pieces': [['dustbin', 0]], 'wet': [['dustbin', 0]], 'open': [['dustbin', 0]], 'temperature': [['dustbin', 0]], 'solid': [['dustbin', 0]], 'contain': [['dustbin', 0]], 'running': [['dustbin', 0]], 'moveable': [['dustbin', 2]], 'mixed': [['dustbin', 0]], 'edible': [['dustbin', 0]]}, {'h_location': [['Tom', 0]], 'conscious': [['Tom', 2]], 'wearing': [['Tom', 0]], 'h_wet': [['Tom', 0]], 'hygiene': [['Tom', 0]], 'location': [['dustbin', 0], ['plate', 6]], 'exist': [['dustbin', 2], ['plate', 2]], 'clean': [['dustbin', 0], ['plate', 5]], 'power': [['dustbin', 0], ['plate', 0]], 'functional': [['dustbin', 2], ['plate', 1]], 'pieces': [['dustbin', 0], ['plate', 0]], 'wet': [['dustbin', 0], ['plate', 0]], 'open': [['dustbin', 0], ['plate', 0]], 'temperature': [['dustbin', 0], ['plate', 0]], 'solid': [['dustbin', 0], ['plate', 0]], 'contain': [['dustbin', 6], ['plate', 0]], 'running': [['dustbin', 0], ['plate', 0]], 'moveable': [['dustbin', 0], ['plate', 2]], 'mixed': [['dustbin', 0], ['plate', 0]], 'edible': [['dustbin', 0], ['plate', 0]]}, {'h_location': [['Tom', 0]], 'conscious': [['Tom', 2]], 'wearing': [['Tom', 0]], 'h_wet': [['Tom', 0]], 'hygiene': [['Tom', 0]], 'location': [['fridge', 0], ['soup', 2]], 'exist': [['fridge', 2], ['soup', 2]], 'clean': [['fridge', 0], ['soup', 0]], 'power': [['fridge', 0], ['soup', 0]], 'functional': [['fridge', 2], ['soup', 2]], 'pieces': [['fridge', 0], ['soup', 0]], 'wet': [['fridge', 0], ['soup', 0]], 'open': [['fridge', 8], ['soup', 0]], 'temperature': [['fridge', 0], ['soup', 1]], 'solid': [['fridge', 0], ['soup', 0]], 'contain': [['fridge', 8], ['soup', 0]], 'running': [['fridge', 0], ['soup', 0]], 'moveable': [['fridge', 2], ['soup', 2]], 'mixed': [['fridge', 0], ['soup', 0]], 'edible': [['fridge', 0], ['soup', 0]]}, {'h_location': [['Tom', 0]], 'conscious': [['Tom', 2]], 'wearing': [['Tom', 0]], 'h_wet': [['Tom', 0]], 'hygiene': [['Tom', 0]], 'location': [['microwave', 0], ['soup', 3]], 'exist': [['microwave', 2], ['soup', 2]], 'clean': [['microwave', 0], ['soup', 0]], 'power': [['microwave', 2], ['soup', 0]], 'functional': [['microwave', 2], ['soup', 2]], 'pieces': [['microwave', 0], ['soup', 0]], 'wet': [['microwave', 0], ['soup', 0]], 'open': [['microwave', 8], ['soup', 0]], 'temperature': [['microwave', 0], ['soup', 0]], 'solid': [['microwave', 0], ['soup', 0]], 'contain': [['microwave', 6], ['soup', 0]], 'running': [['microwave', 0], ['soup', 0]], 'moveable': [['microwave', 2], ['soup', 2]], 'mixed': [['microwave', 0], ['soup', 0]], 'edible': [['microwave', 0], ['soup', 0]]}, {'h_location': [['Tom', 0]], 'conscious': [['Tom', 2]], 'wearing': [['Tom', 0]], 'h_wet': [['Tom', 0]], 'hygiene': [['Tom', 0]], 'location': [['soup', 1]], 'exist': [['soup', 3]], 'clean': [['soup', 0]], 'power': [['soup', 0]], 'functional': [['soup', 2]], 'pieces': [['soup', 0]], 'wet': [['soup', 0]], 'open': [['soup', 0]], 'temperature': [['soup', 7]], 'solid': [['soup', 0]], 'contain': [['soup', 0]], 'running': [['soup', 0]], 'moveable': [['soup', 2]], 'mixed': [['soup', 0]], 'edible': [['soup', 0]]}]}\n","      {'story_id': 0, 'worker_id': 'A1F01FVEPYCPHO', 'type': None, 'idx': None, 'aug': False, 'actor': 'Tom', 'location': 'kitchen', 'objects': 'dustbin, microwave, pan, plate, cereal, soup', 'sentences': ['Tom bought a new dustbin for the kitchen.', 'Tom threw a broken plate in the dustbin.', 'Tom got some soup from the fridge.', 'Tom put the soup in the microwave.', 'Tom turned on the microwave.'], 'length': 5, 'example_id': '0', 'plausible': True, 'breakpoint': -1, 'confl_sents': [], 'confl_pairs': [], 'states': [{'h_location': [['Tom', 0]], 'conscious': [['Tom', 2]], 'wearing': [['Tom', 0]], 'h_wet': [['Tom', 0]], 'hygiene': [['Tom', 0]], 'location': [['dustbin', 6]], 'exist': [['dustbin', 4]], 'clean': [['dustbin', 0]], 'power': [['dustbin', 0]], 'functional': [['dustbin', 2]], 'pieces': [['dustbin', 0]], 'wet': [['dustbin', 0]], 'open': [['dustbin', 0]], 'temperature': [['dustbin', 0]], 'solid': [['dustbin', 0]], 'contain': [['dustbin', 0]], 'running': [['dustbin', 0]], 'moveable': [['dustbin', 2]], 'mixed': [['dustbin', 0]], 'edible': [['dustbin', 0]]}, {'h_location': [['Tom', 0]], 'conscious': [['Tom', 2]], 'wearing': [['Tom', 0]], 'h_wet': [['Tom', 0]], 'hygiene': [['Tom', 0]], 'location': [['dustbin', 0], ['plate', 6]], 'exist': [['dustbin', 2], ['plate', 2]], 'clean': [['dustbin', 0], ['plate', 5]], 'power': [['dustbin', 0], ['plate', 0]], 'functional': [['dustbin', 2], ['plate', 1]], 'pieces': [['dustbin', 0], ['plate', 0]], 'wet': [['dustbin', 0], ['plate', 0]], 'open': [['dustbin', 0], ['plate', 0]], 'temperature': [['dustbin', 0], ['plate', 0]], 'solid': [['dustbin', 0], ['plate', 0]], 'contain': [['dustbin', 6], ['plate', 0]], 'running': [['dustbin', 0], ['plate', 0]], 'moveable': [['dustbin', 0], ['plate', 2]], 'mixed': [['dustbin', 0], ['plate', 0]], 'edible': [['dustbin', 0], ['plate', 0]]}, {'h_location': [['Tom', 0]], 'conscious': [['Tom', 2]], 'wearing': [['Tom', 0]], 'h_wet': [['Tom', 0]], 'hygiene': [['Tom', 0]], 'location': [['fridge', 0], ['soup', 2]], 'exist': [['fridge', 2], ['soup', 2]], 'clean': [['fridge', 0], ['soup', 0]], 'power': [['fridge', 0], ['soup', 0]], 'functional': [['fridge', 2], ['soup', 2]], 'pieces': [['fridge', 0], ['soup', 0]], 'wet': [['fridge', 0], ['soup', 0]], 'open': [['fridge', 8], ['soup', 0]], 'temperature': [['fridge', 0], ['soup', 1]], 'solid': [['fridge', 0], ['soup', 0]], 'contain': [['fridge', 8], ['soup', 0]], 'running': [['fridge', 0], ['soup', 0]], 'moveable': [['fridge', 2], ['soup', 2]], 'mixed': [['fridge', 0], ['soup', 0]], 'edible': [['fridge', 0], ['soup', 0]]}, {'h_location': [['Tom', 0]], 'conscious': [['Tom', 2]], 'wearing': [['Tom', 0]], 'h_wet': [['Tom', 0]], 'hygiene': [['Tom', 0]], 'location': [['microwave', 0], ['soup', 3]], 'exist': [['microwave', 2], ['soup', 2]], 'clean': [['microwave', 0], ['soup', 0]], 'power': [['microwave', 2], ['soup', 0]], 'functional': [['microwave', 2], ['soup', 2]], 'pieces': [['microwave', 0], ['soup', 0]], 'wet': [['microwave', 0], ['soup', 0]], 'open': [['microwave', 8], ['soup', 0]], 'temperature': [['microwave', 0], ['soup', 0]], 'solid': [['microwave', 0], ['soup', 0]], 'contain': [['microwave', 6], ['soup', 0]], 'running': [['microwave', 0], ['soup', 0]], 'moveable': [['microwave', 2], ['soup', 2]], 'mixed': [['microwave', 0], ['soup', 0]], 'edible': [['microwave', 0], ['soup', 0]]}, {'h_location': [['Tom', 0]], 'conscious': [['Tom', 2]], 'wearing': [['Tom', 0]], 'h_wet': [['Tom', 0]], 'hygiene': [['Tom', 0]], 'location': [['microwave', 0]], 'exist': [['microwave', 2]], 'clean': [['microwave', 0]], 'power': [['microwave', 2]], 'functional': [['microwave', 2]], 'pieces': [['microwave', 0]], 'wet': [['microwave', 0]], 'open': [['microwave', 1]], 'temperature': [['microwave', 0]], 'solid': [['microwave', 0]], 'contain': [['microwave', 2]], 'running': [['microwave', 4]], 'moveable': [['microwave', 2]], 'mixed': [['microwave', 0]], 'edible': [['microwave', 0]]}]}\n","    ],\n","  length: \n","    5,\n","  label: \n","    1,\n","  breakpoint: \n","    4,\n","  confl_sents: \n","    [\n","      3\n","    ],\n","  confl_pairs: \n","    [\n","      [3, 4]\n","    ],\n","}\n","\n","\n"]}],"source":["from www.utils import print_dict\n","import json\n","from collections import Counter\n","\n","data_file = os.path.join(DRIVE_PATH, 'all_data/www_2s_new.json')\n","with open(data_file, 'r') as f:\n","  cloze_dataset_2s, order_dataset_2s = json.load(f)  \n","\n","for p in cloze_dataset_2s:\n","  label_dist = Counter([ex['label'] for ex in cloze_dataset_2s[p]])\n","  print('Cloze label distribution (%s):' % p)\n","  print(label_dist.most_common())\n","print_dict(cloze_dataset_2s['train'][0])"]},{"cell_type":"markdown","metadata":{"id":"BxIYaEobhR7J"},"source":["---\n","\n","# TRIP Results\n","\n","Contains code for the tiered and random TRIP baselines."]},{"cell_type":"code","execution_count":null,"metadata":{"id":"mbUgyE0bbJqn"},"outputs":[],"source":["if task_name != 'trip':\n","  raise ValueError('Please configure task_name in first cell to \"trip\" to run TRIP results!')"]},{"cell_type":"markdown","metadata":{"id":"v7VlN2jUwvcC"},"source":["## Random Tiered Classifier for TRIP\n","\n","For the random baseline, we average the results of 10 runs. Running the below will report (mean, variance) for each evaluation partition."]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":41541,"status":"ok","timestamp":1631298579008,"user":{"displayName":"Shane Storks","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14Gh8CqjwUclCWWIoFwUB05rZNHwyd6nEIrTXC6RVXw=s64","userId":"00590987742306216924"},"user_tz":240},"id":"OGYE2UIiASDv","outputId":"c08efd4b-e466-4c47-e4aa-f06163969b8a"},"outputs":[{"name":"stderr","output_type":"stream","text":["[========================================================================] 100%\n","[========================================================================] 100%\n","[========================================================================] 100%\n"]}],"source":["from www.dataset.prepro import get_tiered_data\n","from www.dataset.featurize import add_bert_features_tiered, get_tensor_dataset_tiered\n","from collections import Counter\n","import numpy as np\n","from www.dataset.ann import att_to_num_classes, idx_to_att\n","from sklearn.metrics import accuracy_score, f1_score\n","from www.utils import print_dict\n","\n","tiered_dataset = cloze_dataset_2s\n","\n","seq_length = 16 # Max sequence length to pad to\n","\n","tiered_dataset = get_tiered_data(tiered_dataset)\n","tiered_dataset = add_bert_features_tiered(tiered_dataset, tokenizer, seq_length, add_segment_ids=True)"]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":178884,"status":"ok","timestamp":1631298787066,"user":{"displayName":"Shane Storks","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14Gh8CqjwUclCWWIoFwUB05rZNHwyd6nEIrTXC6RVXw=s64","userId":"00590987742306216924"},"user_tz":240},"id":"wL20bloxwxci","outputId":"127af389-dcda-4b94-bc88-91bff5c49911"},"outputs":[{"name":"stdout","output_type":"stream","text":["starting dev...\n","starting run 0...\n","starting run 1...\n","starting run 2...\n","starting run 3...\n","starting run 4...\n","starting run 5...\n","starting run 6...\n","starting run 7...\n","starting run 8...\n","starting run 9...\n","RANDOM BASELINE (dev, 10 runs)\n","{\n","  story_accuracy: \n","    (0.5021835255841467, 0.004908255286679719),\n","  confl_f1: \n","    (0.4835268355256053, 0.0014760650231936837),\n","  precondition_f1: \n","    (0.04002515428905156, 0.0002521250602361839),\n","  effect_f1: \n","    (0.04052704946285701, 0.00010604150750002376),\n","  verifiability: \n","    (0.0, 0.0),\n","  consistency: \n","    (0.11339359656906241, 0.0014283946461639883),\n","}\n","\n","\n","starting test...\n","starting run 0...\n","starting run 1...\n","starting run 2...\n","starting run 3...\n","starting run 4...\n","starting run 5...\n","starting run 6...\n","starting run 7...\n","starting run 8...\n","starting run 9...\n","RANDOM BASELINE (test, 10 runs)\n","{\n","  story_accuracy: \n","    (0.5003487453018209, 0.0016264689232753202),\n","  confl_f1: \n","    (0.48347260195624564, 0.0003382891708761066),\n","  precondition_f1: \n","    (0.04005627811829042, 5.681999736348582e-05),\n","  effect_f1: \n","    (0.04035201929460706, 7.494384750717566e-05),\n","  verifiability: \n","    (0.0, 0.0),\n","  consistency: \n","    (0.1055008501216, 0.0027929589440376643),\n","}\n","\n","\n"]}],"source":["from www.dataset.prepro import get_tiered_data, balance_labels\n","from www.dataset.featurize import add_bert_features_tiered, get_tensor_dataset_tiered\n","from collections import Counter\n","import numpy as np\n","from www.dataset.ann import att_to_num_classes, idx_to_att, att_default_values\n","from sklearn.metrics import accuracy_score, f1_score\n","from www.utils import print_dict\n","import numpy as np\n","\n","# Have to add BERT input IDs and tensorize again\n","num_runs = 10\n","stories = []\n","pred_stories = []\n","conflicts = []\n","pred_conflicts = []\n","preconditions = []\n","pred_preconditions = []\n","effects = []\n","pred_effects = []\n","verifiability = []\n","consistency = []\n","for p in tiered_dataset:\n","  if p == 'train':\n","    continue\n","  metr_avg = {}\n","  print('starting %s...' % p)\n","  for r in range(num_runs):\n","    print('starting run %s...' % str(r))\n","    for ex in tiered_dataset[p]:\n","      verifiable = True\n","      consistent = True\n","\n","      stories.append(ex['label'])\n","      pred_stories.append(np.random.randint(2))\n","\n","      if stories[-1] != pred_stories[-1]:\n","        verifiable = False\n","\n","      labels_ex_p = []\n","      preds_ex_p = []\n","\n","      labels_ex_e = []\n","      preds_ex_e = []\n","\n","      labels_ex_c = []\n","      preds_ex_c = []\n","\n","      for si, story in enumerate(ex['stories']):\n","        labels_story_p = []\n","        preds_story_p = []\n","\n","        labels_story_e = []\n","        preds_story_e = []      \n","\n","        for ent_ann in story['entities']:\n","          entity = ent_ann['entity']\n","\n","          if si == 1 - ex['label']:\n","            labels_ex_c.append(ent_ann['conflict_span_onehot'])\n","            pred = np.zeros(ent_ann['conflict_span_onehot'].shape)\n","            for cs in np.random.choice(len(pred), size=2, replace=False):\n","              pred[cs] = 1\n","            preds_ex_c.append(pred)\n","\n","          labels_ent = []\n","          preds_ent = []\n","          for s, sent_ann in enumerate(ent_ann['preconditions']):\n","            if s < len(story['sentences']):\n","              if entity in story['sentences'][s]:\n","\n","                labels_ent.append(sent_ann)\n","                sent_ann_pred = []\n","                for i, l in enumerate(sent_ann):\n","                  pl = np.random.randint(att_to_num_classes[idx_to_att[i]])\n","                  if pl > 0 and pl != att_default_values[idx_to_att[i]]:\n","                    if pl != l:\n","                      verifiable = False\n","                  sent_ann_pred.append(pl)\n","                preds_ent.append(sent_ann_pred)\n","\n","          labels_story_p.append(labels_ent)\n","          preds_story_p.append(preds_ent)\n","\n","          labels_ent = []\n","          preds_ent = []\n","          for s, sent_ann in enumerate(ent_ann['effects']):\n","            if s < len(story['sentences']):\n","              if entity in story['sentences'][s]:\n","    \n","                labels_ent.append(sent_ann)\n","                sent_ann_pred = []\n","                for i, l in enumerate(sent_ann):\n","                  pl = np.random.randint(att_to_num_classes[idx_to_att[i]])\n","                  if pl > 0 and pl != att_default_values[idx_to_att[i]]:\n","                    if pl != l:\n","                      verifiable = False\n","                  sent_ann_pred.append(pl)\n","                preds_ent.append(sent_ann_pred)\n","\n","          labels_story_e.append(labels_ent)\n","          preds_story_e.append(preds_ent)\n","\n","        labels_ex_p.append(labels_story_p)\n","        preds_ex_p.append(preds_story_p)\n","\n","        labels_ex_e.append(labels_story_e)\n","        preds_ex_e.append(preds_story_e)\n","\n","      conflicts.append(labels_ex_c)\n","      pred_conflicts.append(preds_ex_c)\n","\n","      preconditions.append(labels_ex_p)\n","      pred_preconditions.append(preds_ex_p)\n","\n","      effects.append(labels_ex_e)\n","      pred_effects.append(preds_ex_e)\n","\n","      p_confl = np.nonzero(np.sum(np.array(preds_ex_c), axis=0))[0]\n","      l_confl = np.nonzero(np.sum(np.array(labels_ex_c), axis=0))[0]\n","      assert len(l_confl) == 2, str(labels_ex_c)\n","      if not (p_confl[0] == l_confl[0] and p_confl[1] == l_confl[1]):\n","        verifiable = False    \n","        consistent = False\n","\n","      verifiability.append(1 if verifiable else 0)\n","      consistency.append(1 if consistent else 0)\n","\n","    # Compute metrics\n","    metr = {}\n","    metr['story_accuracy'] = accuracy_score(stories, pred_stories)\n","\n","    conflicts_flat = [c for c_ex in conflicts for c_ent in c_ex for c in c_ent]\n","    pred_conflicts_flat = [c for c_ex in pred_conflicts for c_ent in c_ex for c in c_ent]\n","    metr['confl_f1'] = f1_score(conflicts_flat, pred_conflicts_flat, average='macro')\n","\n","    preconditions_flat = [p for p_ex in preconditions for p_story in p_ex for p_sent in p_story for p_ent in p_sent for p in p_ent]\n","    pred_preconditions_flat = [p for p_ex in pred_preconditions for p_story in p_ex for p_sent in p_story for p_ent in p_sent for p in p_ent]\n","    metr['precondition_f1'] = f1_score(preconditions_flat, pred_preconditions_flat, average='macro')\n","\n","    effects_flat = [p for p_ex in effects for p_story in p_ex for p_sent in p_story for p_ent in p_sent for p in p_ent]\n","    pred_effects_flat = [p for p_ex in pred_effects for p_story in p_ex for p_sent in p_story for p_ent in p_sent for p in p_ent]\n","    metr['effect_f1'] = f1_score(effects_flat, pred_effects_flat, average='macro')\n","\n","    metr['verifiability'] = np.mean(verifiability)\n","    metr['consistency'] = np.mean(consistency)\n","\n","    for k in metr:\n","      if k not in metr_avg:\n","        metr_avg[k] = []\n","      metr_avg[k].append(metr[k])\n","\n","  for k in metr_avg:\n","    metr_avg[k] = (np.mean(metr_avg[k]), np.var(metr_avg[k]) ** 0.5)\n","  print('RANDOM BASELINE (%s, %s runs)' % (str(p), str(num_runs)))\n","  print_dict(metr_avg)"]},{"cell_type":"markdown","metadata":{"id":"5ctQweSlAceo"},"source":["## Transformer-Based Tiered Classifier for TRIP\n","\n","This is the baseline model presented in the paper. Based on the settings above, the below cells can be used for training and evaluating models.\n"]},{"cell_type":"markdown","metadata":{"id":"0q-xjfYU_cV8"},"source":["### Featurization for Tiered Classification\n","\n","Get the data ready for input to the model."]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":2388,"status":"ok","timestamp":1631299805307,"user":{"displayName":"Shane Storks","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14Gh8CqjwUclCWWIoFwUB05rZNHwyd6nEIrTXC6RVXw=s64","userId":"00590987742306216924"},"user_tz":240},"id":"gLCJXeMb_cV9","outputId":"d4fb68d9-a303-4427-d5e7-d797bc1838f9"},"outputs":[{"name":"stderr","output_type":"stream","text":["[========================================================================] 100%\n","[========================================================================] 100%\n","[========================================================================] 100%\n"]}],"source":["from www.dataset.prepro import get_tiered_data, balance_labels\n","from www.dataset.featurize import add_bert_features_tiered, get_tensor_dataset_tiered\n","from collections import Counter\n","\n","tiered_dataset = cloze_dataset_2s\n","\n","# Debug the code on a small amount of data\n","if debug:\n","  for k in tiered_dataset:\n","    tiered_dataset[k] = tiered_dataset[k][:20]\n","\n","# train_spans = True\n","train_spans = False\n","if train_spans:\n","  tiered_dataset = get_story_spans_2s(tiered_dataset, train_only=True)\n","  tiered_dataset['train'] = [ex for ex in tiered_dataset['train'] if ex['label'] != -1] # For now, ignore examples where both stories are plausible :(\n","\n","seq_length = 16 # Max sequence length to pad to\n","\n","tiered_dataset = get_tiered_data(tiered_dataset)\n","tiered_dataset = add_bert_features_tiered(tiered_dataset, tokenizer, seq_length, add_segment_ids=True)\n","\n","tiered_tensor_dataset = {}\n","max_story_length = max([len(ex['stories'][0]['sentences']) for p in tiered_dataset for ex in tiered_dataset[p]])\n","for p in tiered_dataset:\n","  tiered_tensor_dataset[p] = get_tensor_dataset_tiered(tiered_dataset[p], max_story_length, add_segment_ids=True)"]},{"cell_type":"markdown","metadata":{"id":"-fQ6wXQIBdq1"},"source":["### Train Models"]},{"cell_type":"markdown","metadata":{"id":"U-BMInyrBdq2"},"source":["#### Configure Hyperparameters\n","We will perform grid search over (batch size, learning rate). Configure the training sub-task, search space and set the maximum number of training epochs here. Currently configured for re-training the best RoBERTa-based model instance. Read code comments for more information.\n","\n","**Additional configuration options:**\n","* Change the `generate_learning_curve` variable to `True` to generate data for training curves in the style presented in the paper.\n","* You may ablate the input to the Conflict Detector based on a few pre-defined ablation modes. To do so, change the `ablation` variable based on the comments in the code."]},{"cell_type":"code","execution_count":null,"metadata":{"id":"tvfTuEYRBdq3"},"outputs":[],"source":["from www.dataset.ann import att_to_idx, att_to_num_classes, att_types\n","\n","subtask = 'cloze'\n","batch_sizes = [config_batch_size]\n","learning_rates = [config_lr]\n","epochs = config_epochs\n","eval_batch_size = 16\n","generate_learning_curve = False # Generate data for training curve figure in TRIP paper\n","\n","num_state_labels = {}\n","for att in att_to_idx:\n","  if att_types[att] == 'default':\n","    num_state_labels[att_to_idx[att]] = 3\n","  else:\n","    num_state_labels[att_to_idx[att]] = att_to_num_classes[att] # Location attributes fall into this since they don't have well-define pre- and post-condition yet\n","\n","# Ablation options:\n","# - attributes: skip attribute prediction phase\n","# - embeddings: DON'T input contextual embeddings to conflict detector\n","# - states: DON'T input states to conflict detector\n","# - states-labels: in states input to conflict detector, include predicted labels\n","# - states-logits: in states input to conflict detector, include state logits (preferred)\n","# - states-teacher-forcing: train conflict detector on ground truth state labels (not predictions)\n","# - states-attention: re-weight input to conflict detector with weights conditioned on states representation\n","ablation = ['attributes', 'states-logits'] # This is the default mode presented in the paper"]},{"cell_type":"markdown","metadata":{"id":"8fRC3cnLBdq3"},"source":["#### Perform Grid Search\n","\n","Perform hyperparameter tuning to find the best story classification model.\n"]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":59885,"status":"ok","timestamp":1631299694590,"user":{"displayName":"Shane Storks","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14Gh8CqjwUclCWWIoFwUB05rZNHwyd6nEIrTXC6RVXw=s64","userId":"00590987742306216924"},"user_tz":240},"id":"DWnCen7NBdq3","outputId":"e6a79d9b-b478-4ea5-e4ef-d50d2a22985d"},"outputs":[{"name":"stdout","output_type":"stream","text":["Beginning grid search for the cloze sub-task over 1 parameter combination(s)!\n","\n","TRAINING MODEL: bs=1, lr=1e-05\n"]},{"name":"stderr","output_type":"stream","text":["\r                                                                               \r\r[                                                                        ] N/A%"]},{"name":"stdout","output_type":"stream","text":["[0] Beginning epoch...\n"]},{"name":"stderr","output_type":"stream","text":["[########################################################################] 100%\n","[                                                                        ] N/A%"]},{"name":"stdout","output_type":"stream","text":["\tBeginning evaluation...\n","\t\tRunning prediction...\n"]},{"name":"stderr","output_type":"stream","text":["[########################################################################] 100%\n"]},{"name":"stdout","output_type":"stream","text":["\t\tComputing metrics...\n","\tFinished evaluation in 0:00:04s.\n","[0] Validation results:\n","[0] Preconditions:\n","{\n","  accuracy: \n","    0.9325384615384615,\n","  f1: \n","    0.12892786027865571,\n","  accuracy_0: \n","    0.9857692307692307,\n","  f1_0: \n","    0.3309445412873652,\n","  accuracy_1: \n","    0.9034615384615384,\n","  f1_1: \n","    0.3164275611234593,\n","  accuracy_2: \n","    0.9896153846153846,\n","  f1_2: \n","    0.33159353051098656,\n","  accuracy_3: \n","    0.9992307692307693,\n","  f1_3: \n","    0.49980761831473647,\n","  accuracy_4: \n","    1.0,\n","  f1_4: \n","    1.0,\n","  accuracy_5: \n","    0.963076923076923,\n","  f1_5: \n","    0.12264890282131662,\n","  accuracy_6: \n","    0.8703846153846154,\n","  f1_6: \n","    0.3102337377476181,\n","  accuracy_7: \n","    0.9946153846153846,\n","  f1_7: \n","    0.4986502121095257,\n","  accuracy_8: \n","    0.9776923076923076,\n","  f1_8: \n","    0.4943601711396344,\n","  accuracy_9: \n","    0.7534615384615385,\n","  f1_9: \n","    0.31274501508869895,\n","  accuracy_10: \n","    0.9911538461538462,\n","  f1_10: \n","    0.4977786362758354,\n","  accuracy_11: \n","    0.995,\n","  f1_11: \n","    0.49874686716791977,\n","  accuracy_12: \n","    0.8415384615384616,\n","  f1_12: \n","    0.3096523100726521,\n","  accuracy_13: \n","    0.9573076923076923,\n","  f1_13: \n","    0.3260627497216218,\n","  accuracy_14: \n","    0.9846153846153847,\n","  f1_14: \n","    0.330749354005168,\n","  accuracy_15: \n","    0.9830769230769231,\n","  f1_15: \n","    0.330488750969744,\n","  accuracy_16: \n","    0.9811538461538462,\n","  f1_16: \n","    0.33016242800750667,\n","  accuracy_17: \n","    0.5415384615384615,\n","  f1_17: \n","    0.30163489824262113,\n","  accuracy_18: \n","    0.9407692307692308,\n","  f1_18: \n","    0.32316025895098427,\n","  accuracy_19: \n","    0.9973076923076923,\n","  f1_19: \n","    0.49932601579048724,\n","}\n","\n","\n","[0] Effects:\n","{\n","  accuracy: \n","    0.9553461538461538,\n","  f1: \n","    0.1236313700729243,\n","  accuracy_0: \n","    0.9911538461538462,\n","  f1_0: \n","    0.4977786362758354,\n","  accuracy_1: \n","    0.8276923076923077,\n","  f1_1: \n","    0.3149981331799514,\n","  accuracy_2: \n","    0.98,\n","  f1_2: \n","    0.32996632996632996,\n","  accuracy_3: \n","    0.9992307692307693,\n","  f1_3: \n","    0.49980761831473647,\n","  accuracy_4: \n","    0.9965384615384615,\n","  f1_4: \n","    0.49913311500674246,\n","  accuracy_5: \n","    0.963076923076923,\n","  f1_5: \n","    0.12264890282131662,\n","  accuracy_6: \n","    0.875,\n","  f1_6: \n","    0.3111111111111111,\n","  accuracy_7: \n","    0.9903846153846154,\n","  f1_7: \n","    0.33172302737520126,\n","  accuracy_8: \n","    0.9776923076923076,\n","  f1_8: \n","    0.4943601711396344,\n","  accuracy_9: \n","    0.87,\n","  f1_9: \n","    0.3159968983559354,\n","  accuracy_10: \n","    0.9311538461538461,\n","  f1_10: \n","    0.32164208848146675,\n","  accuracy_11: \n","    0.9892307692307692,\n","  f1_11: \n","    0.33152874452178394,\n","  accuracy_12: \n","    0.9888461538461538,\n","  f1_12: \n","    0.33146393347514985,\n","  accuracy_13: \n","    0.9884615384615385,\n","  f1_13: \n","    0.33139909735654416,\n","  accuracy_14: \n","    0.9846153846153847,\n","  f1_14: \n","    0.330749354005168,\n","  accuracy_15: \n","    0.9788461538461538,\n","  f1_15: \n","    0.329770003239391,\n","  accuracy_16: \n","    0.9811538461538462,\n","  f1_16: \n","    0.33016242800750667,\n","  accuracy_17: \n","    0.8946153846153846,\n","  f1_17: \n","    0.4721883881445392,\n","  accuracy_18: \n","    0.9403846153846154,\n","  f1_18: \n","    0.48463825569871155,\n","  accuracy_19: \n","    0.9588461538461538,\n","  f1_19: \n","    0.3327999979658246,\n","}\n","\n","\n","[0] Conflicts:\n","{\n","  accuracy: \n","    0.9323076923076923,\n","  f1: \n","    0.482484076433121,\n","}\n","\n","\n","[0] Stories:\n","{\n","  accuracy: \n","    0.7,\n","  f1: \n","    0.6875000000000001,\n","  verifiability: \n","    0.0,\n","}\n","\n","\n","[0] Saving model checkpoint...\n","[0] Finished epoch.\n","Finished grid search! :)\n","Best validation *verifiability* 0.0 from model <none>.\n","Best validation *accuracy* 0.7 from model roberta-large_cloze_1_1e-05_0_0.0-0.4-0.4-0.2-0.0_tiered_pipeline_lc_ablate_attributes_states-logits.\n"]}],"source":["from torch.utils.data import DataLoader, RandomSampler, SequentialSampler\n","from transformers import get_linear_schedule_with_warmup\n","from www.model.train import train_epoch_tiered\n","from www.model.eval import evaluate_tiered, save_results, save_preds, add_entity_attribute_labels\n","from sklearn.metrics import accuracy_score, f1_score\n","from www.utils import print_dict, get_model_dir\n","from www.model.transformers_ext import TieredModelPipeline\n","from www.dataset.ann import att_to_num_classes\n","import shutil\n","import pandas as pd\n","\n","seed_val = 22 # Save random seed for reproducibility\n","random.seed(seed_val)\n","np.random.seed(seed_val)\n","torch.manual_seed(seed_val)\n","torch.cuda.manual_seed_all(seed_val)\n","\n","# We'll keep the validation data here with a constant eval batch size\n","dev_sampler = SequentialSampler(tiered_tensor_dataset['dev'])\n","dev_dataloader = DataLoader(tiered_tensor_dataset['dev'], sampler=dev_sampler, batch_size=eval_batch_size)\n","dev_dataset_name = subtask + '_%s_dev'\n","dev_ids = [ex['example_id'] for ex in tiered_dataset['dev']]\n","\n","all_losses = []\n","param_combos = []\n","combo_names = []\n","all_val_objs = []\n","output_dirs = []\n","best_obj = 0.0\n","best_model = '<none>'\n","best_dir = ''\n","best_obj2 = 0.0\n","best_model2 = '<none>'\n","best_dir2 = ''\n","\n","print('Beginning grid search for the %s sub-task over %s parameter combination(s)!' % (subtask, str(len(batch_sizes) * len(learning_rates))))\n","for bs in batch_sizes:\n","  for lr in learning_rates:\n","    print('\\nTRAINING MODEL: bs=%s, lr=%s' % (str(bs), str(lr)))\n","\n","    loss_values = []\n","    obj_values = []\n","\n","    # Set up training dataset with new batch size\n","    train_sampler = RandomSampler(tiered_tensor_dataset['train'])\n","    train_dataloader = DataLoader(tiered_tensor_dataset['train'], sampler=train_sampler, batch_size=bs)\n","\n","    # Set up model\n","    config = config_class.from_pretrained(model_name,\n","                                          cache_dir=os.path.join(DRIVE_PATH, 'cache'))    \n","    emb = emb_class.from_pretrained(model_name,\n","                                          config=config,\n","                                          cache_dir=os.path.join(DRIVE_PATH, 'cache'))    \n","    if torch.cuda.is_available():\n","      emb.cuda()\n","    device = emb.device\n","    max_story_length = max([len(ex['stories'][0]['sentences']) for p in tiered_dataset for ex in tiered_dataset[p]])\n","    model = TieredModelPipeline(emb, max_story_length, len(att_to_num_classes), num_state_labels,\n","                                config_class, model_name, device, \n","                                ablation=ablation, loss_weights=loss_weights).to(device)\n","\n","    # Set up optimizer\n","    optimizer = AdamW(model.parameters(), lr=lr)\n","    total_steps = len(train_dataloader) * epochs\n","    scheduler = get_linear_schedule_with_warmup(optimizer, num_warmup_steps=0, num_training_steps = total_steps)\n","\n","    train_lc_data = []\n","    val_lc_data = []\n","    for epoch in range(epochs):\n","      # Train the model for one epoch\n","      print('[%s] Beginning epoch...' % str(epoch))\n","\n","      epoch_loss, _ = train_epoch_tiered(model, optimizer, train_dataloader, device, seg_mode=False, \n","                                         build_learning_curves=generate_learning_curve, val_dataloader=dev_dataloader, \n","                                         train_lc_data=train_lc_data, val_lc_data=val_lc_data)\n","      \n","      # Save loss\n","      loss_values.append(epoch_loss)\n","\n","      # Validate on dev set\n","      validation_results = evaluate_tiered(model, dev_dataloader, device, [(accuracy_score, 'accuracy'), (f1_score, 'f1')], seg_mode=False, return_explanations=True)\n","      metr_attr, all_pred_atts, all_atts, \\\n","      metr_prec, all_pred_prec, all_prec, \\\n","      metr_eff, all_pred_eff, all_eff, \\\n","      metr_conflicts, all_pred_conflicts, all_conflicts, \\\n","      metr_stories, all_pred_stories, all_stories, explanations = validation_results[:16]\n","      explanations = add_entity_attribute_labels(explanations, tiered_dataset['dev'], list(att_to_num_classes.keys()))\n","\n","      print('[%s] Validation results:' % str(epoch))\n","      print('[%s] Preconditions:' % str(epoch))\n","      print_dict(metr_prec)\n","      print('[%s] Effects:' % str(epoch))\n","      print_dict(metr_eff)\n","      print('[%s] Conflicts:' % str(epoch))\n","      print_dict(metr_conflicts)\n","      print('[%s] Stories:' % str(epoch))\n","      print_dict(metr_stories)\n","\n","      # Save accuracy - want to maximize verifiability of tiered predictions\n","      ver = metr_stories['verifiability']\n","      acc = metr_stories['accuracy']\n","      obj_values.append(ver)\n","      \n","      # Save model checkpoint\n","      print('[%s] Saving model checkpoint...' % str(epoch))\n","      model_param_str = get_model_dir(model_name.replace('/', '-'), subtask, bs, lr, epoch) + '_' +  '-'.join([str(lw) for lw in loss_weights]) +  '_tiered_pipeline_lc'\n","      if train_spans:\n","        model_param_str += 'spans'\n","      if len(model.ablation) > 0:\n","        model_param_str += '_ablate_'\n","        model_param_str += '_'.join(model.ablation)\n","      output_dir = os.path.join(DRIVE_PATH, 'saved_models', model_param_str)\n","      output_dirs.append(output_dir)\n","      if not os.path.exists(output_dir):\n","        os.makedirs(output_dir)\n","\n","      save_results(metr_attr, output_dir, dev_dataset_name % 'attributes')\n","      save_results(metr_prec, output_dir, dev_dataset_name % 'preconditions')\n","      save_results(metr_eff, output_dir, dev_dataset_name % 'effects')\n","      save_results(metr_conflicts, output_dir, dev_dataset_name % 'conflicts')\n","      save_results(metr_stories, output_dir, dev_dataset_name % 'stories')\n","      save_results(explanations, output_dir, dev_dataset_name % 'explanations')\n","\n","      # Just save story preds\n","      save_preds(dev_ids, all_stories, all_pred_stories, output_dir, dev_dataset_name % 'stories')\n","\n","      emb = emb.module if hasattr(emb, 'module') else emb\n","      emb.save_pretrained(output_dir)\n","      torch.save(model, os.path.join(output_dir, 'classifiers.pth'))\n","      tokenizer.save_vocabulary(output_dir)\n","\n","      if ver > best_obj:\n","        best_obj = ver\n","        best_model = model_param_str\n","        best_dir = output_dir\n","      if acc > best_obj2:\n","        best_obj2 = acc\n","        best_model2 = model_param_str\n","        best_dir2 = output_dir        \n","\n","      for od in output_dirs:\n","        if od != best_dir and od != best_dir2 and os.path.exists(od):\n","          shutil.rmtree(od)\n","\n","      print('[%s] Finished epoch.' % str(epoch))\n","\n","    all_losses.append(loss_values)\n","    all_val_objs.append(obj_values)\n","    param_combos.append((bs, lr))\n","    combo_names.append('bs=%s, lr=%s' % (str(bs), str(lr)))\n","\n","print('Finished grid search! :)')\n","print('Best validation *verifiability* %s from model %s.' % (str(best_obj), best_model))\n","print('Best validation *accuracy* %s from model %s.' % (str(best_obj2), best_model2))\n","\n","if generate_learning_curve:\n","  print('Saving learning curve data...')\n","  train_lc_data = [subrecord for record in train_lc_data for subrecord in record] # flatten\n","  val_lc_data = [subrecord for record in val_lc_data for subrecord in record] # flatten\n","\n","  train_lc_data = pd.DataFrame(train_lc_data)\n","  print(os.path.join(best_dir if best_dir != '<none>' else best_dir2, 'learning_curve_data_train.csv'))\n","  train_lc_data.to_csv(os.path.join(best_dir if best_dir != '' else best_dir2, 'learning_curve_data_train.csv'), index=False)\n","  val_lc_data = pd.DataFrame(val_lc_data)\n","  val_lc_data.to_csv(os.path.join(best_dir if best_dir != '' else best_dir2, 'learning_curve_data_val.csv'), index=False)\n","  print('Learning curve data saved. %s rows saved for training, %s rows saved for validation.' % (str(len(train_lc_data.index)), str(len(val_lc_data.index))))"]},{"cell_type":"markdown","metadata":{"id":"2_jhT4gSBdq5"},"source":["Delete all non-best model checkpoints:"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"o7iILulEBdq6"},"outputs":[],"source":["import shutil\n","\n","# Delete non-best model checkpoints\n","for od in output_dirs:\n","  if od != best_dir and od != best_dir2 and os.path.exists(od):\n","    shutil.rmtree(od)"]},{"cell_type":"markdown","metadata":{"id":"aWFmGRhznl2T"},"source":["### Test Models\n","\n","Evaluate accuracy, consistency, and verifiability on the test set."]},{"cell_type":"markdown","metadata":{"id":"Rbvpm9irn3qL"},"source":["#### Load the Trained Model\n","\n","Load the trained model we want to probe and select the appropriate dataset. Paths to the pre-trained models presented in the paper are already provided (download links are found in GitHub repo)."]},{"cell_type":"code","execution_count":null,"metadata":{"id":"fM_lYqw9n3qM"},"outputs":[],"source":["from www.model.transformers_ext import TieredModelPipeline\n","from www.dataset.ann import att_to_num_classes, att_to_idx, att_types\n","\n","probe_model = eval_model_dir\n","probe_model = os.path.join(DRIVE_PATH, 'saved_models', probe_model)\n","\n","ablation = ['attributes', 'states-logits']\n","\n","if 'cloze' in probe_model:\n","  subtask = 'cloze'\n","elif 'order' in probe_model:\n","  subtask = 'order'\n","  \n","if subtask == 'cloze':\n","  subtask_dataset = cloze_dataset_2s\n","elif subtask == 'order':\n","  subtask_dataset = order_dataset_2s\n","\n","# Load the model\n","model = None\n","# model = torch.load(os.path.join(probe_model, 'classifiers.pth'), map_location=torch.device('cpu'))\n","model = torch.load(os.path.join(probe_model, 'classifiers.pth'))\n","if torch.cuda.is_available():\n","  model.cuda()\n","device = model.embedding.device\n","\n","for layer in model.precondition_classifiers:\n","  layer.eval()\n","for layer in model.effect_classifiers:\n","  layer.eval()"]},{"cell_type":"markdown","metadata":{"id":"RfXiCTA9KPjG"},"source":["#### Test the Model\n","\n","Run inference on the testing set of TRIP. Can simply edit the top-level `for` loop if you want to run inference on other partitions."]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":4255,"status":"ok","timestamp":1631299857967,"user":{"displayName":"Shane Storks","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14Gh8CqjwUclCWWIoFwUB05rZNHwyd6nEIrTXC6RVXw=s64","userId":"00590987742306216924"},"user_tz":240},"id":"FQX4bIxcKWlf","outputId":"1ee7ba5e-0426-47a3-f2cc-e251bd29056f"},"outputs":[{"name":"stderr","output_type":"stream","text":["\r                                                                               \r\r[                                                                        ] N/A%"]},{"name":"stdout","output_type":"stream","text":["Testing model: drive/My Drive/Colab Notebooks/Research/TRIP_replication/saved_models/roberta-large_cloze_1_1e-05_0_0.0-0.4-0.4-0.2-0.0_tiered_pipeline_lc_ablate_attributes_states-logits.\n","\tBeginning evaluation...\n","\t\tRunning prediction...\n"]},{"name":"stderr","output_type":"stream","text":["[########################################################################] 100%\n"]},{"name":"stdout","output_type":"stream","text":["\t\tComputing metrics...\n","\tFinished evaluation in 0:00:04s.\n","\n","PARTITION: test\n","Stories:\n","{\n","  accuracy: \n","    0.4,\n","  f1: \n","    0.3939393939393939,\n","  verifiability: \n","    0.0,\n","}\n","\n","\n","Conflicts:\n","{\n","  accuracy: \n","    0.9391666666666667,\n","  f1: \n","    0.4843145681134508,\n","}\n","\n","\n","Preconditions:\n","{\n","  accuracy: \n","    0.9269583333333333,\n","  f1: \n","    0.11782545514031922,\n","  accuracy_0: \n","    0.98125,\n","  f1_0: \n","    0.3301787592008412,\n","  accuracy_1: \n","    0.9066666666666666,\n","  f1_1: \n","    0.3171549336831366,\n","  accuracy_2: \n","    0.99,\n","  f1_2: \n","    0.3316582914572864,\n","  accuracy_3: \n","    0.9916666666666667,\n","  f1_3: \n","    0.33193863319386335,\n","  accuracy_4: \n","    0.9954166666666666,\n","  f1_4: \n","    0.3325676898447832,\n","  accuracy_5: \n","    0.9595833333333333,\n","  f1_5: \n","    0.10881942967845584,\n","  accuracy_6: \n","    0.8558333333333333,\n","  f1_6: \n","    0.30743900613680586,\n","  accuracy_7: \n","    0.99,\n","  f1_7: \n","    0.3316582914572864,\n","  accuracy_8: \n","    0.9933333333333333,\n","  f1_8: \n","    0.4983277591973244,\n","  accuracy_9: \n","    0.71625,\n","  f1_9: \n","    0.30886512295451407,\n","  accuracy_10: \n","    0.99625,\n","  f1_10: \n","    0.49906073888541014,\n","  accuracy_11: \n","    0.9958333333333333,\n","  f1_11: \n","    0.4989561586638831,\n","  accuracy_12: \n","    0.8254166666666667,\n","  f1_12: \n","    0.3229032470694753,\n","  accuracy_13: \n","    0.9533333333333334,\n","  f1_13: \n","    0.33131872040322746,\n","  accuracy_14: \n","    0.9925,\n","  f1_14: \n","    0.3320786281890422,\n","  accuracy_15: \n","    0.9825,\n","  f1_15: \n","    0.330390920554855,\n","  accuracy_16: \n","    0.9904166666666666,\n","  f1_16: \n","    0.33172842090572885,\n","  accuracy_17: \n","    0.5070833333333333,\n","  f1_17: \n","    0.30592670409157063,\n","  accuracy_18: \n","    0.9183333333333333,\n","  f1_18: \n","    0.47871416159860986,\n","  accuracy_19: \n","    0.9975,\n","  f1_19: \n","    0.3329161451814769,\n","}\n","\n","\n","Effects:\n","{\n","  accuracy: \n","    0.9504375,\n","  f1: \n","    0.10898644608927507,\n","  accuracy_0: \n","    0.9825,\n","  f1_0: \n","    0.4955863808322824,\n","  accuracy_1: \n","    0.8120833333333334,\n","  f1_1: \n","    0.30000769645193565,\n","  accuracy_2: \n","    0.9745833333333334,\n","  f1_2: \n","    0.32904269536470426,\n","  accuracy_3: \n","    0.9883333333333333,\n","  f1_3: \n","    0.33137747974294496,\n","  accuracy_4: \n","    0.9904166666666666,\n","  f1_4: \n","    0.33172842090572885,\n","  accuracy_5: \n","    0.9595833333333333,\n","  f1_5: \n","    0.10881942967845584,\n","  accuracy_6: \n","    0.8533333333333334,\n","  f1_6: \n","    0.3069544364508393,\n","  accuracy_7: \n","    0.985,\n","  f1_7: \n","    0.3308144416456759,\n","  accuracy_8: \n","    0.9933333333333333,\n","  f1_8: \n","    0.4983277591973244,\n","  accuracy_9: \n","    0.8566666666666667,\n","  f1_9: \n","    0.31535122778969804,\n","  accuracy_10: \n","    0.9129166666666667,\n","  f1_10: \n","    0.3182280319535221,\n","  accuracy_11: \n","    0.9895833333333334,\n","  f1_11: \n","    0.33158813263525305,\n","  accuracy_12: \n","    0.9891666666666666,\n","  f1_12: \n","    0.3315179444211702,\n","  accuracy_13: \n","    0.99375,\n","  f1_13: \n","    0.3322884012539185,\n","  accuracy_14: \n","    0.9908333333333333,\n","  f1_14: \n","    0.33179852099902335,\n","  accuracy_15: \n","    0.96875,\n","  f1_15: \n","    0.328042328042328,\n","  accuracy_16: \n","    0.9883333333333333,\n","  f1_16: \n","    0.33137747974294496,\n","  accuracy_17: \n","    0.8908333333333334,\n","  f1_17: \n","    0.31664405553294445,\n","  accuracy_18: \n","    0.9291666666666667,\n","  f1_18: \n","    0.48164146868250535,\n","  accuracy_19: \n","    0.9595833333333333,\n","  f1_19: \n","    0.3264582890353675,\n","}\n","\n","\n"]}],"source":["from torch.utils.data import DataLoader, RandomSampler, SequentialSampler\n","from www.model.eval import evaluate_tiered, save_results, save_preds, list_comparison, add_entity_attribute_labels\n","from sklearn.metrics import accuracy_score, precision_score, recall_score, f1_score, confusion_matrix\n","metrics = [(accuracy_score, 'accuracy'), (precision_score, 'precision'), (recall_score, 'recall'), (f1_score, 'f1')]\n","import numpy as np\n","from www.utils import print_dict\n","\n","print('Testing model: %s.' % probe_model)\n","\n","# May alter this depending on which partition(s) you want to run inference on\n","for p in tiered_dataset:\n","  if p != 'test':\n","    continue\n","\n","  p_dataset = tiered_dataset[p]\n","  p_tensor_dataset = tiered_tensor_dataset[p]\n","  p_sampler = SequentialSampler(p_tensor_dataset)\n","  p_dataloader = DataLoader(p_tensor_dataset, sampler=p_sampler, batch_size=16)\n","  dev_dataset_name = subtask + '_%s_' + p\n","  p_ids = [ex['example_id'] for ex in tiered_dataset[p]]\n","\n","  # Get preds and metrics on this partition\n","  metr_attr, all_pred_atts, all_atts, \\\n","  metr_prec, all_pred_prec, all_prec, \\\n","  metr_eff, all_pred_eff, all_eff, \\\n","  metr_conflicts, all_pred_conflicts, all_conflicts, \\\n","  metr_stories, all_pred_stories, all_stories, explanations = evaluate_tiered(model, p_dataloader, device, [(accuracy_score, 'accuracy'), (f1_score, 'f1')], seg_mode=False, return_explanations=True)\n","  explanations = add_entity_attribute_labels(explanations, tiered_dataset[p], list(att_to_num_classes.keys()))\n","\n","  save_results(metr_attr, probe_model, dev_dataset_name % 'attributes')\n","  save_results(metr_prec, probe_model, dev_dataset_name % 'preconditions')\n","  save_results(metr_eff, probe_model, dev_dataset_name % 'effects')\n","  save_results(metr_conflicts, probe_model, dev_dataset_name % 'conflicts')\n","  save_results(metr_stories, probe_model, dev_dataset_name % 'stories')\n","  save_results(explanations, probe_model, dev_dataset_name % 'explanations')\n","\n","  print('\\nPARTITION: %s' % p)\n","  print('Stories:')\n","  print_dict(metr_stories)\n","  print('Conflicts:')\n","  print_dict(metr_conflicts)\n","  print('Preconditions:')\n","  print_dict(metr_prec)\n","  print('Effects:')\n","  print_dict(metr_eff)"]},{"cell_type":"markdown","metadata":{"id":"ON1UAnbc8OOE"},"source":["#### Add Consistency Metric to Model Results\n","The intermediate conistency metric isn't included in the originally calculated metrics. This block adds the consistency metric to pre-existing model directory based on the tiered predictions. Generates a new `results_cloze_stories_final_[partition].json` file that includes the consistency metric.\n","\n"]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":387,"status":"ok","timestamp":1631299967551,"user":{"displayName":"Shane Storks","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14Gh8CqjwUclCWWIoFwUB05rZNHwyd6nEIrTXC6RVXw=s64","userId":"00590987742306216924"},"user_tz":240},"id":"k1obFel58pd-","outputId":"f8cc0c5a-756b-4879-deb2-3aa8489474c7"},"outputs":[{"name":"stdout","output_type":"stream","text":["Found 0 consistent preds in dev (versus 0 verifiable)\n","Found 0 consistent preds in test (versus 0 verifiable)\n"]}],"source":["import json\n","import os\n","\n","model_directories = [eval_model_dir]\n","\n","partitions = ['dev', 'test']\n","expl_fname = 'results_cloze_explanations_%s.json'\n","endtask_fname = 'results_cloze_stories_%s.json'\n","endtask_fname_new = 'results_cloze_stories_final_%s.json'\n","for md in model_directories:\n","  for p in partitions:\n","    explanations = json.load(open(os.path.join(DRIVE_PATH, 'saved_models', md, expl_fname % p), 'r'))\n","    endtask_results = json.load(open(os.path.join(DRIVE_PATH, 'saved_models', md, endtask_fname % p), 'r'))\n","\n","    consistent_preds = 0\n","    verifiable_preds = 0\n","    total = 0\n","    for expl in explanations:\n","      if expl['valid_explanation']:\n","        verifiable_preds += 1\n","      if expl['story_pred'] == expl['story_label']:\n","        if len(expl['conflict_pred']) == len(expl['conflict_label']) and expl['conflict_pred'][0] == expl['conflict_label'][0] and expl['conflict_pred'][1] == expl['conflict_label'][1]:\n","          expl['consistent'] = True\n","          consistent_preds += 1\n","        else:\n","          expl['consistent'] = False\n","      total += 1\n","\n","    endtask_results['consistency'] = float(consistent_preds) / total\n","    print('Found %s consistent preds in %s (versus %s verifiable)' % (str(consistent_preds), p, str(verifiable_preds)))\n","    json.dump(explanations, open(os.path.join(DRIVE_PATH, 'saved_models', md, (expl_fname % p).replace('explanations', 'explanations_consistency')), 'w'))\n","    json.dump(endtask_results, open(os.path.join(DRIVE_PATH, 'saved_models', md, endtask_fname_new % p), 'w'))"]},{"cell_type":"markdown","metadata":{"id":"S7C0wy5Dvtv2"},"source":["\n","# Conversational Entailment (CE) Results\n","\n","Code for the coherence experiments on CE."]},{"cell_type":"code","execution_count":null,"metadata":{"id":"YKxPGbnWbdIy"},"outputs":[],"source":["if task_name != 'ce':\n","  raise ValueError('Please configure task_name in first cell to \"ce\" to run CE results!')"]},{"cell_type":"markdown","metadata":{"id":"W7MADzgfvtv3"},"source":["## Load Conversational Entailment Dataset"]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":111,"status":"ok","timestamp":1631301900364,"user":{"displayName":"Shane Storks","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14Gh8CqjwUclCWWIoFwUB05rZNHwyd6nEIrTXC6RVXw=s64","userId":"00590987742306216924"},"user_tz":240},"id":"nSnNY2zbvtv3","outputId":"fb02836c-d14b-435a-96d7-e40dac522e82"},"outputs":[{"name":"stdout","output_type":"stream","text":["Reserved 40 dialog sources for training and validation.\n"]}],"source":["import xml.etree.ElementTree as ET\n","import pickle\n","cache_train = os.path.join(DRIVE_PATH, 'all_data/ConvEnt/ConvEnt_train_resplit.json')\n","cache_dev = os.path.join(DRIVE_PATH,'all_data/ConvEnt/ConvEnt_dev_resplit.json')\n","cache_test = os.path.join(DRIVE_PATH,'all_data/ConvEnt/ConvEnt_test_resplit.json')\n","ConvEnt_train = json.load(open(cache_train))\n","ConvEnt_dev = json.load(open(cache_dev))\n","ConvEnt_test = json.load(open(cache_test))\n","\n","# Combine train and dev and do cross-validation\n","cache_folds = os.path.join(DRIVE_PATH,'all_data/ConvEnt/ConvEnt_folds.pkl') # Folds used for results presented in paper\n","ConvEnt_train = ConvEnt_train + ConvEnt_dev\n","train_sources = list(set([ex['dialog_source'] for ex in ConvEnt_train]))\n","print(\"Reserved %s dialog sources for training and validation.\" % len(train_sources))\n","\n","no_folds = 8\n","if not os.path.exists(cache_folds):\n","  folds = []\n","  for k in range(no_folds):\n","    folds.append(np.random.choice(train_sources, size=5, replace=False))\n","    train_sources = [s for s in train_sources if s not in folds[-1]]\n","  assert len(train_sources) == 0\n","  print(folds)\n","  pickle.dump(folds, open(cache_folds, 'wb'))\n","else:\n","  folds = pickle.load(open(cache_folds, 'rb'))"]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":8,"status":"ok","timestamp":1631301900480,"user":{"displayName":"Shane Storks","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14Gh8CqjwUclCWWIoFwUB05rZNHwyd6nEIrTXC6RVXw=s64","userId":"00590987742306216924"},"user_tz":240},"id":"TBYhEsdovtv4","outputId":"c1c90590-7443-4f03-96e6-07e4ac429161"},"outputs":[{"name":"stdout","output_type":"stream","text":["train examples: 703\n","dev examples: 110\n","test examples: 172\n"]}],"source":["print('train examples:', len(ConvEnt_train))\n","print('dev examples:', len(ConvEnt_dev))\n","print('test examples:', len(ConvEnt_test))"]},{"cell_type":"markdown","metadata":{"id":"Msxt3xAhvtv6"},"source":["## Featurize Conversational Entailment"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"NBfqFw82vtv7"},"outputs":[],"source":["from www.dataset.featurize import add_bert_features_ConvEnt, get_tensor_dataset\n","import pickle\n","seq_length = 128\n","\n","ConvEnt_train = add_bert_features_ConvEnt(ConvEnt_train, tokenizer, seq_length, add_segment_ids=True)\n","ConvEnt_dev = add_bert_features_ConvEnt(ConvEnt_dev, tokenizer, seq_length, add_segment_ids=True)\n","ConvEnt_test = add_bert_features_ConvEnt(ConvEnt_test, tokenizer, seq_length, add_segment_ids=True)\n","\n","ConvEnt_train_folds = [[] for _ in range(no_folds)]\n","ConvEnt_dev_folds = [[] for _ in range(no_folds)]\n","for k in range(no_folds):\n","  ConvEnt_train_folds[k] = [ex for ex in ConvEnt_train if ex['dialog_source'] not in folds[k]]\n","  ConvEnt_dev_folds[k] = [ex for ex in ConvEnt_train if ex['dialog_source'] in folds[k]]\n","\n","  if debug:\n","    ConvEnt_train_folds[k] = ConvEnt_train_folds[k][:10]\n","    ConvEnt_dev_folds[k] = ConvEnt_dev_folds[k][:10]\n","\n","if debug:\n","  ConvEnt_train = ConvEnt_train[:10]\n","  ConvEnt_dev = ConvEnt_dev[:10]\n","  ConvEnt_test = ConvEnt_test[:10]\n","\n","ConvEnt_train_tensor = get_tensor_dataset(ConvEnt_train, label_key='label', add_segment_ids=True)\n","ConvEnt_test_tensor = get_tensor_dataset(ConvEnt_test, label_key='label', add_segment_ids=True)\n","\n","# Training sets for each validation fold\n","ConvEnt_train_folds_tensor = [get_tensor_dataset(ConvEnt_train_folds[k], label_key='label', add_segment_ids=True) for k in range(no_folds)]\n","ConvEnt_dev_folds_tensor = [get_tensor_dataset(ConvEnt_dev_folds[k], label_key='label', add_segment_ids=True) for k in range(no_folds)]"]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":5,"status":"ok","timestamp":1631301488436,"user":{"displayName":"Shane Storks","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14Gh8CqjwUclCWWIoFwUB05rZNHwyd6nEIrTXC6RVXw=s64","userId":"00590987742306216924"},"user_tz":240},"id":"2Wtek-Tmvtv7","outputId":"9d3f5110-edce-4d35-d4b2-23891d4da280"},"outputs":[{"name":"stdout","output_type":"stream","text":["train examples: 10\n","dev examples: 10\n","test examples: 10\n"]}],"source":["print('train examples:', len(ConvEnt_train))\n","print('dev examples:', len(ConvEnt_dev))\n","print('test examples:', len(ConvEnt_test))"]},{"cell_type":"markdown","metadata":{"id":"rn5Ywwvxvtv6"},"source":["## Train Models on Conversational Entailment"]},{"cell_type":"markdown","metadata":{"id":"Ytbj9Uxxvtv7"},"source":["### Train Models"]},{"cell_type":"markdown","metadata":{"id":"h9YL5qnRvtv7"},"source":["#### Configure Hyperparameters"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"fC5iz07Fvtv8"},"outputs":[],"source":["batch_sizes = [config_batch_size]\n","learning_rates = [config_lr]\n","epochs = config_epochs\n","eval_batch_size = 128"]},{"cell_type":"markdown","metadata":{"id":"Jss8T2xzvtv8"},"source":["#### Grid Search and Cross-Validation"]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":141231,"status":"ok","timestamp":1631301166877,"user":{"displayName":"Shane Storks","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14Gh8CqjwUclCWWIoFwUB05rZNHwyd6nEIrTXC6RVXw=s64","userId":"00590987742306216924"},"user_tz":240},"id":"5QOdgrufvtv8","outputId":"5d20813e-882f-4896-fe6e-e03db305d6d2"},"outputs":[{"name":"stdout","output_type":"stream","text":["Beginning grid search for ConvEnt over 1 parameter combination(s)!\n","\n","TRAINING MODEL: bs=1, lr=1e-05\n","Beginning fold 1/8...\n"]},{"name":"stderr","output_type":"stream","text":["Some weights of the model checkpoint at roberta-large were not used when initializing RobertaForSequenceClassification: ['lm_head.bias', 'lm_head.dense.weight', 'lm_head.dense.bias', 'lm_head.layer_norm.weight', 'lm_head.layer_norm.bias', 'lm_head.decoder.weight', 'roberta.pooler.dense.weight', 'roberta.pooler.dense.bias']\n","- This IS expected if you are initializing RobertaForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n","- This IS NOT expected if you are initializing RobertaForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n","Some weights of RobertaForSequenceClassification were not initialized from the model checkpoint at roberta-large and are newly initialized: ['classifier.dense.weight', 'classifier.dense.bias', 'classifier.out_proj.weight', 'classifier.out_proj.bias']\n","You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n"]},{"name":"stdout","output_type":"stream","text":["[0] Beginning epoch...\n","\tBeginning evaluation...\n","\t\tRunning prediction...\n","\t\tComputing metrics...\n","\tFinished evaluation in 0:00:00s.\n","[0] Validation results:\n","{\n","  accuracy: \n","    0.5,\n","}\n","\n","\n","[0] Finished epoch.\n","Beginning fold 2/8...\n"]},{"name":"stderr","output_type":"stream","text":["Some weights of the model checkpoint at roberta-large were not used when initializing RobertaForSequenceClassification: ['lm_head.bias', 'lm_head.dense.weight', 'lm_head.dense.bias', 'lm_head.layer_norm.weight', 'lm_head.layer_norm.bias', 'lm_head.decoder.weight', 'roberta.pooler.dense.weight', 'roberta.pooler.dense.bias']\n","- This IS expected if you are initializing RobertaForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n","- This IS NOT expected if you are initializing RobertaForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n","Some weights of RobertaForSequenceClassification were not initialized from the model checkpoint at roberta-large and are newly initialized: ['classifier.dense.weight', 'classifier.dense.bias', 'classifier.out_proj.weight', 'classifier.out_proj.bias']\n","You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n"]},{"name":"stdout","output_type":"stream","text":["[0] Beginning epoch...\n","\tBeginning evaluation...\n","\t\tRunning prediction...\n","\t\tComputing metrics...\n","\tFinished evaluation in 0:00:00s.\n","[0] Validation results:\n","{\n","  accuracy: \n","    0.7,\n","}\n","\n","\n","[0] Finished epoch.\n","Beginning fold 3/8...\n"]},{"name":"stderr","output_type":"stream","text":["Some weights of the model checkpoint at roberta-large were not used when initializing RobertaForSequenceClassification: ['lm_head.bias', 'lm_head.dense.weight', 'lm_head.dense.bias', 'lm_head.layer_norm.weight', 'lm_head.layer_norm.bias', 'lm_head.decoder.weight', 'roberta.pooler.dense.weight', 'roberta.pooler.dense.bias']\n","- This IS expected if you are initializing RobertaForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n","- This IS NOT expected if you are initializing RobertaForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n","Some weights of RobertaForSequenceClassification were not initialized from the model checkpoint at roberta-large and are newly initialized: ['classifier.dense.weight', 'classifier.dense.bias', 'classifier.out_proj.weight', 'classifier.out_proj.bias']\n","You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n"]},{"name":"stdout","output_type":"stream","text":["[0] Beginning epoch...\n","\tBeginning evaluation...\n","\t\tRunning prediction...\n","\t\tComputing metrics...\n","\tFinished evaluation in 0:00:00s.\n","[0] Validation results:\n","{\n","  accuracy: \n","    0.3,\n","}\n","\n","\n","[0] Finished epoch.\n","Beginning fold 4/8...\n"]},{"name":"stderr","output_type":"stream","text":["Some weights of the model checkpoint at roberta-large were not used when initializing RobertaForSequenceClassification: ['lm_head.bias', 'lm_head.dense.weight', 'lm_head.dense.bias', 'lm_head.layer_norm.weight', 'lm_head.layer_norm.bias', 'lm_head.decoder.weight', 'roberta.pooler.dense.weight', 'roberta.pooler.dense.bias']\n","- This IS expected if you are initializing RobertaForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n","- This IS NOT expected if you are initializing RobertaForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n","Some weights of RobertaForSequenceClassification were not initialized from the model checkpoint at roberta-large and are newly initialized: ['classifier.dense.weight', 'classifier.dense.bias', 'classifier.out_proj.weight', 'classifier.out_proj.bias']\n","You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n"]},{"name":"stdout","output_type":"stream","text":["[0] Beginning epoch...\n","\tBeginning evaluation...\n","\t\tRunning prediction...\n","\t\tComputing metrics...\n","\tFinished evaluation in 0:00:00s.\n","[0] Validation results:\n","{\n","  accuracy: \n","    0.6,\n","}\n","\n","\n","[0] Finished epoch.\n","Beginning fold 5/8...\n"]},{"name":"stderr","output_type":"stream","text":["Some weights of the model checkpoint at roberta-large were not used when initializing RobertaForSequenceClassification: ['lm_head.bias', 'lm_head.dense.weight', 'lm_head.dense.bias', 'lm_head.layer_norm.weight', 'lm_head.layer_norm.bias', 'lm_head.decoder.weight', 'roberta.pooler.dense.weight', 'roberta.pooler.dense.bias']\n","- This IS expected if you are initializing RobertaForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n","- This IS NOT expected if you are initializing RobertaForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n","Some weights of RobertaForSequenceClassification were not initialized from the model checkpoint at roberta-large and are newly initialized: ['classifier.dense.weight', 'classifier.dense.bias', 'classifier.out_proj.weight', 'classifier.out_proj.bias']\n","You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n"]},{"name":"stdout","output_type":"stream","text":["[0] Beginning epoch...\n","\tBeginning evaluation...\n","\t\tRunning prediction...\n","\t\tComputing metrics...\n","\tFinished evaluation in 0:00:00s.\n","[0] Validation results:\n","{\n","  accuracy: \n","    0.4,\n","}\n","\n","\n","[0] Finished epoch.\n","Beginning fold 6/8...\n"]},{"name":"stderr","output_type":"stream","text":["Some weights of the model checkpoint at roberta-large were not used when initializing RobertaForSequenceClassification: ['lm_head.bias', 'lm_head.dense.weight', 'lm_head.dense.bias', 'lm_head.layer_norm.weight', 'lm_head.layer_norm.bias', 'lm_head.decoder.weight', 'roberta.pooler.dense.weight', 'roberta.pooler.dense.bias']\n","- This IS expected if you are initializing RobertaForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n","- This IS NOT expected if you are initializing RobertaForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n","Some weights of RobertaForSequenceClassification were not initialized from the model checkpoint at roberta-large and are newly initialized: ['classifier.dense.weight', 'classifier.dense.bias', 'classifier.out_proj.weight', 'classifier.out_proj.bias']\n","You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n"]},{"name":"stdout","output_type":"stream","text":["[0] Beginning epoch...\n","\tBeginning evaluation...\n","\t\tRunning prediction...\n","\t\tComputing metrics...\n","\tFinished evaluation in 0:00:00s.\n","[0] Validation results:\n","{\n","  accuracy: \n","    0.3,\n","}\n","\n","\n","[0] Finished epoch.\n","Beginning fold 7/8...\n"]},{"name":"stderr","output_type":"stream","text":["Some weights of the model checkpoint at roberta-large were not used when initializing RobertaForSequenceClassification: ['lm_head.bias', 'lm_head.dense.weight', 'lm_head.dense.bias', 'lm_head.layer_norm.weight', 'lm_head.layer_norm.bias', 'lm_head.decoder.weight', 'roberta.pooler.dense.weight', 'roberta.pooler.dense.bias']\n","- This IS expected if you are initializing RobertaForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n","- This IS NOT expected if you are initializing RobertaForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n","Some weights of RobertaForSequenceClassification were not initialized from the model checkpoint at roberta-large and are newly initialized: ['classifier.dense.weight', 'classifier.dense.bias', 'classifier.out_proj.weight', 'classifier.out_proj.bias']\n","You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n"]},{"name":"stdout","output_type":"stream","text":["[0] Beginning epoch...\n","\tBeginning evaluation...\n","\t\tRunning prediction...\n","\t\tComputing metrics...\n","\tFinished evaluation in 0:00:00s.\n","[0] Validation results:\n","{\n","  accuracy: \n","    0.5,\n","}\n","\n","\n","[0] Finished epoch.\n","Beginning fold 8/8...\n"]},{"name":"stderr","output_type":"stream","text":["Some weights of the model checkpoint at roberta-large were not used when initializing RobertaForSequenceClassification: ['lm_head.bias', 'lm_head.dense.weight', 'lm_head.dense.bias', 'lm_head.layer_norm.weight', 'lm_head.layer_norm.bias', 'lm_head.decoder.weight', 'roberta.pooler.dense.weight', 'roberta.pooler.dense.bias']\n","- This IS expected if you are initializing RobertaForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n","- This IS NOT expected if you are initializing RobertaForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n","Some weights of RobertaForSequenceClassification were not initialized from the model checkpoint at roberta-large and are newly initialized: ['classifier.dense.weight', 'classifier.dense.bias', 'classifier.out_proj.weight', 'classifier.out_proj.bias']\n","You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n"]},{"name":"stdout","output_type":"stream","text":["[0] Beginning epoch...\n","\tBeginning evaluation...\n","\t\tRunning prediction...\n","\t\tComputing metrics...\n","\tFinished evaluation in 0:00:00s.\n","[0] Validation results:\n","{\n","  accuracy: \n","    0.4,\n","}\n","\n","\n","[0] Finished epoch.\n","Top performing param combos:\n","[((1, 1e-05, 0), 0.46249999999999997)]\n"]}],"source":["from torch.utils.data import DataLoader, RandomSampler, SequentialSampler\n","from transformers import get_linear_schedule_with_warmup\n","from www.model.train import train_epoch\n","from www.model.eval import evaluate, save_results, save_preds\n","from sklearn.metrics import accuracy_score\n","from www.utils import print_dict, get_model_dir\n","from collections import Counter\n","\n","seed_val = 22 # Save random seed for reproducibility\n","random.seed(seed_val)\n","np.random.seed(seed_val)\n","torch.manual_seed(seed_val)\n","torch.cuda.manual_seed_all(seed_val)\n","\n","assert len(batch_sizes) == 1\n","train_fold_sampler = [RandomSampler(f) for f in ConvEnt_train_folds_tensor]\n","train_fold_dataloader = [DataLoader(f, sampler=train_fold_sampler[i], batch_size=batch_sizes[0]) for i, f in enumerate(ConvEnt_train_folds_tensor)]\n","\n","dev_fold_sampler = [SequentialSampler(f) for f in ConvEnt_dev_folds_tensor]\n","dev_fold_dataloader = [DataLoader(f, sampler=dev_fold_sampler[i], batch_size=eval_batch_size) for i, f in enumerate(ConvEnt_dev_folds_tensor)]\n","\n","all_val_accs = Counter()\n","print('Beginning grid search for ConvEnt over %s parameter combination(s)!' % (str(len(batch_sizes) * len(learning_rates))))\n","for bs in batch_sizes:\n","  for lr in learning_rates:\n","    print('\\nTRAINING MODEL: bs=%s, lr=%s' % (str(bs), str(lr)))\n","\n","    for k in range(no_folds):\n","      print('Beginning fold %s/%s...' % (str(k+1), str(no_folds)))\n","\n","      # Set up model\n","      if 'mnli' not in mode:\n","        model = model_class.from_pretrained(model_name, \n","                                            cache_dir=os.path.join(DRIVE_PATH, 'cache'))\n","      else:\n","        config = config_class.from_pretrained(model_name.replace('-mnli',''),\n","                                        num_labels=3,\n","                                        cache_dir=os.path.join(DRIVE_PATH, 'cache'))\n","        model = model_class.from_pretrained(model_name, \n","                                            config=config,\n","                                            cache_dir=os.path.join(DRIVE_PATH, 'cache'))\n","        config.num_labels = 2\n","        model.num_labels = 2\n","        model.classifier = cls_head_class(config=config) # Need to bring in a classification head for only 2 labels\n","    \n","      model.cuda()\n","      device = model.device \n","\n","      # Set up optimizer\n","      optimizer = AdamW(model.parameters(), lr=lr)\n","      total_steps = len(train_fold_dataloader[k]) * epochs\n","      scheduler = get_linear_schedule_with_warmup(optimizer, num_warmup_steps=0, num_training_steps = total_steps)\n","\n","      for epoch in range(epochs):\n","        # Train the model for one epoch\n","        print('[%s] Beginning epoch...' % str(epoch))\n","\n","        epoch_loss, _ = train_epoch(model, optimizer, train_fold_dataloader[k], device, seg_mode=True if 'roberta' not in mode else False)\n","        \n","        # Validate on dev set\n","        results, _, _ = evaluate(model, dev_fold_dataloader[k], device, [(accuracy_score, 'accuracy')], seg_mode=True if 'roberta' not in mode else False)\n","        print('[%s] Validation results:' % str(epoch))\n","        print_dict(results)\n","\n","        # Save accuracy\n","        acc = results['accuracy']\n","        if (bs, lr, epoch) in all_val_accs:\n","          all_val_accs[(bs, lr, epoch)] += acc\n","        else:\n","          all_val_accs[(bs, lr, epoch)] = acc\n","        \n","      model.cpu()\n","      del model\n","      del optimizer\n","      del results\n","      del scheduler\n","      del total_steps\n","\n","      print('[%s] Finished epoch.' % str(epoch))\n","\n","for k in all_val_accs:\n","  all_val_accs[k] /= no_folds\n","\n","print('Top performing param combos:')\n","print(all_val_accs.most_common(5))\n","\n","save_fname = os.path.join(DRIVE_PATH, 'saved_models/%s_ConvEnt_xval_%s.pkl' % (model_name.replace('/','-'), '_'.join([str(lr) for lr in learning_rates])))\n","pickle.dump(all_val_accs, open(save_fname, 'wb'))"]},{"cell_type":"markdown","metadata":{"id":"8XhSrDlpI0aH"},"source":["#### Re-Train Best Model from Cross-Validation\n","\n","Re-train a model with the best parameters from the search above. If this isn't run directly after the above cell, replace `save_fname.split('/'[-1])` in `xval_fnames` with the name of the `pkl` file previously generated in the `saved_models` directory."]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":23469,"status":"ok","timestamp":1631301377078,"user":{"displayName":"Shane Storks","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14Gh8CqjwUclCWWIoFwUB05rZNHwyd6nEIrTXC6RVXw=s64","userId":"00590987742306216924"},"user_tz":240},"id":"RI_AWgyFIzUm","outputId":"f29073fc-87b1-4405-d4bd-e99e17624a00"},"outputs":[{"name":"stderr","output_type":"stream","text":["Some weights of the model checkpoint at roberta-large were not used when initializing RobertaForSequenceClassification: ['lm_head.bias', 'lm_head.dense.weight', 'lm_head.dense.bias', 'lm_head.layer_norm.weight', 'lm_head.layer_norm.bias', 'lm_head.decoder.weight', 'roberta.pooler.dense.weight', 'roberta.pooler.dense.bias']\n","- This IS expected if you are initializing RobertaForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n","- This IS NOT expected if you are initializing RobertaForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n","Some weights of RobertaForSequenceClassification were not initialized from the model checkpoint at roberta-large and are newly initialized: ['classifier.dense.weight', 'classifier.dense.bias', 'classifier.out_proj.weight', 'classifier.out_proj.bias']\n","You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n"]},{"name":"stdout","output_type":"stream","text":["[0] Beginning epoch...\n","[0] Saving model checkpoint...\n"]},{"data":{"text/plain":["('drive/My Drive/Colab Notebooks/Research/TRIP_replication/saved_models/roberta-large_ConvEnt_1_1e-05_0_xval/vocab.json',\n"," 'drive/My Drive/Colab Notebooks/Research/TRIP_replication/saved_models/roberta-large_ConvEnt_1_1e-05_0_xval/merges.txt')"]},"execution_count":42,"metadata":{},"output_type":"execute_result"}],"source":["from torch.utils.data import DataLoader, RandomSampler, SequentialSampler\n","from transformers import get_linear_schedule_with_warmup\n","from www.model.train import train_epoch\n","from www.model.eval import evaluate, save_results, save_preds\n","from sklearn.metrics import accuracy_score\n","from www.utils import print_dict, get_model_dir\n","from collections import Counter\n","\n","# Re-train the model with the best parameters from the grid search/cross-validation (with all folds)\n","xval_fnames = []\n","xval_fnames.append(save_fname.split('/')[-1])\n","\n","xval_results = Counter()\n","for fname in xval_fnames:\n","  xval_results += pickle.load(open(os.path.join(DRIVE_PATH, 'saved_models/', fname), 'rb'))\n","\n","batch_size, learning_rate, epochs = xval_results.most_common(1)[0][0]\n","epochs += 1\n","\n","# Set up model\n","if 'mnli' not in mode:\n","  model = model_class.from_pretrained(model_name, \n","                                      cache_dir=os.path.join(DRIVE_PATH, 'cache'))\n","else:\n","  config = config_class.from_pretrained(model_name.replace('-mnli',''),\n","                                  num_labels=3,\n","                                  cache_dir=os.path.join(DRIVE_PATH, 'cache'))\n","  model = model_class.from_pretrained(model_name, \n","                                      config=config,\n","                                      cache_dir=os.path.join(DRIVE_PATH, 'cache'))\n","  config.num_labels = 2\n","  model.num_labels = 2\n","  model.classifier = cls_head_class(config=config) # Need to bring in a classification head for only 2 labels\n","\n","model.cuda()\n","device = model.device \n","\n","train_sampler = RandomSampler(ConvEnt_train_tensor)\n","train_dataloader = DataLoader(ConvEnt_train_tensor, sampler=train_sampler, batch_size=batch_size)\n","\n","# Set up optimizer\n","optimizer = AdamW(model.parameters(), lr=learning_rate)\n","total_steps = len(train_dataloader) * epochs\n","scheduler = get_linear_schedule_with_warmup(optimizer, num_warmup_steps=0, num_training_steps = total_steps)\n","\n","for epoch in range(epochs):\n","  print('[%s] Beginning epoch...' % str(epoch))\n","  epoch_loss, _ = train_epoch(model, optimizer, train_dataloader, device, seg_mode=True if 'roberta' not in mode else False)\n","\n","print('[%s] Saving model checkpoint...' % str(epoch))\n","model_param_str = get_model_dir(model_name.replace('/','-'), 'ConvEnt', batch_size, learning_rate, epoch) + '_xval'\n","output_dir = os.path.join(DRIVE_PATH, 'saved_models', model_param_str)\n","if not os.path.exists(output_dir):\n","  os.makedirs(output_dir)\n","model = model.module if hasattr(model, 'module') else model\n","model.save_pretrained(output_dir)\n","tokenizer.save_vocabulary(output_dir)"]},{"cell_type":"markdown","metadata":{"id":"CADDFieTvtv9"},"source":["## Test Models on Conversational Entailment"]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":14850,"status":"ok","timestamp":1631301512340,"user":{"displayName":"Shane Storks","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14Gh8CqjwUclCWWIoFwUB05rZNHwyd6nEIrTXC6RVXw=s64","userId":"00590987742306216924"},"user_tz":240},"id":"-NJAbMNRvtv9","outputId":"a493868a-c762-4279-b73b-7b6e7b20e148"},"outputs":[{"name":"stdout","output_type":"stream","text":["Testing model: roberta-large_ConvEnt_1_1e-05_0_xval.\n","\tBeginning evaluation...\n","\t\tRunning prediction...\n","\t\tComputing metrics...\n","\tFinished evaluation in 0:00:00s.\n","Results (test):\n","{\n","  accuracy: \n","    0.7,\n","}\n","\n","\n"]}],"source":["from torch.utils.data import DataLoader, RandomSampler, SequentialSampler\n","from transformers import get_linear_schedule_with_warmup\n","from www.model.eval import evaluate, save_results, save_preds\n","from sklearn.metrics import accuracy_score\n","from www.utils import print_dict, get_model_dir\n","\n","best_model = eval_model_dir\n","\n","\n","best_model = os.path.join(DRIVE_PATH, 'saved_models', best_model)\n","\n","# Load the model\n","model = model_class.from_pretrained(best_model)\n","model.cuda()\n","device = model.device\n","\n","# Select appropriate dataset\n","if 'cloze' in best_model:\n","  subtask = 'cloze'\n","elif 'order' in best_model:\n","  subtask = 'order'\n","\n","test_sampler = SequentialSampler(ConvEnt_test_tensor)\n","test_dataloader = DataLoader(ConvEnt_test_tensor, sampler=test_sampler, batch_size=128)\n","test_dataset_name = '%s_%s' % ('ConvEnt', 'test')\n","test_ids = [str(ex['example_id']) for ex in ConvEnt_test]\n","\n","print('Testing model: %s.' % best_model.split('/')[-1])\n","\n","results, preds, labels = evaluate(model, test_dataloader, device, [(accuracy_score, 'accuracy')], seg_mode=True if 'roberta' not in mode else False)\n","save_results(results, best_model, test_dataset_name)\n","save_preds(test_ids, labels, preds, best_model, test_dataset_name)\n","\n","print('Results (%s):' % p)\n","print_dict(results)"]},{"cell_type":"markdown","metadata":{"id":"qy_A5hR9vtv9"},"source":["## Coherence Checks on Conversational Entailment"]},{"cell_type":"markdown","metadata":{"id":"GO1JOxnIvtv9"},"source":["### Load and Featurize Span Data"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"bOsSHJnCvtv-"},"outputs":[],"source":["from www.dataset.featurize import add_bert_features_ConvEnt, get_tensor_dataset\n","from www.dataset.prepro import get_ConvEnt_spans\n","import pickle\n","seq_length = 128\n","\n","merged_file = os.path.join(DRIVE_PATH, 'all_data/ConvEnt/ConvEnt_test_annotation_merged2.json')\n","ConvEnt_test = json.load(open(merged_file))\n","\n","ConvEnt_test = add_bert_features_ConvEnt(ConvEnt_test, tokenizer, seq_length, add_segment_ids=True)\n","\n","if debug:\n","  ConvEnt_test = ConvEnt_test[:10]\n","\n","# Some of the annotated examples are no longer in the test set :(\n","# ConvEnt_test = [ex for ex in ConvEnt_test if ex['id'] in test_ids]\n","\n","# Make span versions of the datasets\n","ConvEnt_test_spans = get_ConvEnt_spans(ConvEnt_test)\n","\n","# Add BERT features\n","ConvEnt_test_tensor = get_tensor_dataset(ConvEnt_test, label_key='label', add_segment_ids=True)\n","ConvEnt_test_spans_tensor = get_tensor_dataset(ConvEnt_test_spans, label_key='label', add_segment_ids=True)"]},{"cell_type":"markdown","metadata":{"id":"cAtipHPtvtv-"},"source":["### Load the Trained Model\n","\n","Load the trained model we want to probe and select the appropriate dataset."]},{"cell_type":"code","execution_count":null,"metadata":{"id":"NCssPtg_vtv-"},"outputs":[],"source":["probe_model = eval_model_dir\n","probe_model = os.path.join(DRIVE_PATH, 'saved_models', probe_model)\n","\n","# Load the model\n","model = model_class.from_pretrained(probe_model)\n","if torch.cuda.is_available():\n","  model.cuda()\n","device = model.device "]},{"cell_type":"markdown","metadata":{"id":"PIPh3HSXvtv-"},"source":["#### Load Trained Model's Base Predictions\n","\n","For comparison, we also want the preds and labels for the previous level."]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":20,"status":"ok","timestamp":1631301931718,"user":{"displayName":"Shane Storks","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14Gh8CqjwUclCWWIoFwUB05rZNHwyd6nEIrTXC6RVXw=s64","userId":"00590987742306216924"},"user_tz":240},"id":"U9lHmA2evtv-","outputId":"60d88a06-7376-4bd6-edcc-d846cf8aa7c6"},"outputs":[{"name":"stdout","output_type":"stream","text":["dict_keys(['73', '74', '75', '76', '77', '78', '79', '80', '81', '82'])\n"]}],"source":["from www.model.eval import load_preds\n","from www.utils import print_dict\n","\n","preds_base = {}\n","preds_base['test'] = load_preds(os.path.join(probe_model, 'preds_ConvEnt_test.tsv'))\n","print(preds_base['test'].keys())"]},{"cell_type":"markdown","metadata":{"id":"maZGCMvMvtv-"},"source":["### Check a Model\n","\n","Will print out strict and lenient coherence metrics."]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":3056,"status":"ok","timestamp":1631302156247,"user":{"displayName":"Shane Storks","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14Gh8CqjwUclCWWIoFwUB05rZNHwyd6nEIrTXC6RVXw=s64","userId":"00590987742306216924"},"user_tz":240},"id":"tl_M9hnqvtv-","outputId":"1143ab4e-d54d-471f-f186-ae09646ea059"},"outputs":[{"name":"stdout","output_type":"stream","text":["Testing model: drive/My Drive/Colab Notebooks/Research/TRIP_replication/saved_models/roberta-large_ConvEnt_1_1e-05_0_xval.\n","\tBeginning evaluation...\n","\t\tRunning prediction...\n","\t\tComputing metrics...\n","\tFinished evaluation in 0:00:02s.\n","\n","PARTITION: test\n","{\n","  lenient_coherence: \n","    0.4269000933706816,\n","  strict_coherence: \n","    0.4,\n","}\n","\n","\n"]},{"name":"stderr","output_type":"stream","text":["/usr/local/lib/python3.7/dist-packages/sklearn/metrics/_classification.py:1272: UndefinedMetricWarning: Precision is ill-defined and being set to 0.0 in labels with no predicted samples. Use `zero_division` parameter to control this behavior.\n","  _warn_prf(average, modifier, msg_start, len(result))\n"]}],"source":["from torch.utils.data import DataLoader, RandomSampler, SequentialSampler\n","from www.model.eval import evaluate, save_results, save_preds, list_comparison\n","from sklearn.metrics import accuracy_score, precision_score, recall_score, f1_score, confusion_matrix\n","metrics = [(accuracy_score, 'accuracy'), (precision_score, 'precision'), (recall_score, 'recall'), (f1_score, 'f1')]\n","import numpy as np\n","from www.utils import print_dict\n","\n","def is_polarized(smax, thres):\n","  return (abs(smax[0] - smax[1]) >= thres)\n","\n","print('Testing model: %s.' % probe_model)\n","\n","all_results = {}\n","p = 'test'\n","\n","p_dataset = ConvEnt_test_spans\n","p_tensor_dataset = ConvEnt_test_spans_tensor\n","p_sampler = SequentialSampler(p_tensor_dataset)\n","p_dataloader = DataLoader(p_tensor_dataset, sampler=p_sampler, batch_size=512)\n","p_dataset_name = '%s_spans_%s' % ('ConvEnt', p)\n","p_dataset_name_co = '%s_consistent_%s' % ('ConvEnt', p)\n","p_dataset_name_bp = '%s_breakpoints_%s' % ('ConvEnt', p)\n","p_dataset_name_ev = '%s_evidence_%s' % ('ConvEnt', p)\n","p_dataset_name_coh = '%s_coherent_%s' % ('ConvEnt', p)\n","p_ids = [str(ex['example_id']) for ex in ConvEnt_test_spans]\n","p_labels = [ex['label'] for ex in ConvEnt_test_spans]\n","\n","# Get span preds and save metrics\n","results, preds, labels = evaluate(model, p_dataloader, device, metrics, seg_mode=True if 'roberta' not in mode else False)\n","save_results(results, probe_model, p_dataset_name)\n","save_preds(p_ids, labels, preds, probe_model, p_dataset_name)\n","\n","# Convert substory preds into breakpoint preds for each example\n","ids_base = [str(ex['example_id']) for ex in ConvEnt_test]\n","\n","id_to_pred = {k: v for k,v in zip(p_ids, preds)}\n","id_to_label = {k: v for k,v in zip(p_ids, p_labels)}\n","\n","preds_entailment = []\n","labels_entailment = []\n","preds_consistent = []\n","preds_breakpoint = []\n","labels_breakpoint = []\n","preds_evidence = []\n","labels_evidence = []    \n","span_accuracies = []\n","span_accuracies_strict = []\n","preds_coherent = []\n","\n","for i, exid in enumerate(ids_base):\n","  ex = ConvEnt_test[i]\n","  ex['length'] = len(ex['turns'])\n","\n","  label_entailment = preds_base[p][exid]['label']\n","  pred_entailment = preds_base[p][exid]['pred']\n","  labels_entailment.append(label_entailment)\n","  preds_entailment.append(pred_entailment)\n","\n","  # Get ground truth breakpoint and evidence\n","  label_breakpoint = ex['conflict_pair'][1] if ex['conflict_pair'] is not None and len(ex['conflict_pair']) > 0 else 0\n","  labels_breakpoint.append(label_breakpoint)\n","  if label_breakpoint > 0:\n","    label_ev = ex['conflict_pair'][0]\n","  else:\n","    label_ev = -1\n","  labels_evidence.append(label_ev)\n","\n","  # Check consistency - any span that entails the hypothesis' superspans should also entail\n","  pred_consistent = True\n","  span_accuracy = 0.0\n","  span_accuracy_strict = 0.0\n","  pred_coherent = True\n","  \n","  no_spans = 0\n","  for sp1 in range(ex['length']):\n","    if not pred_consistent:\n","      break\n","\n","    for sp2 in range(sp1, ex['length']):\n","      if not pred_consistent:\n","        break\n","\n","      span_pred = id_to_pred[exid + '-sp%s:%s' % (str(sp1), str(sp2))]\n","      span_label = id_to_label[exid + '-sp%s:%s' % (str(sp1), str(sp2))]\n","\n","      if span_pred == span_label:\n","        span_accuracy += 1.0\n","        if label_entailment == pred_entailment:\n","            span_accuracy_strict += 1.0\n","      else:\n","        pred_coherent = False\n","      no_spans += 1\n","      # print('%s:%s\\t%s\\t(%s, %s)' % (str(sp1), str(sp2), str(span_pred), str(span_prob[0]), str(span_prob[1])))      \n","\n","      if span_pred == 1:\n","        if pred_entailment == 1:\n","          for sp3 in range(sp1+1):\n","            if not pred_consistent:\n","              break\n","\n","            for sp4 in range(sp2, ex['length']):\n","              if not pred_consistent:\n","                break\n","\n","              sspan_pred = id_to_pred[exid + '-sp%s:%s' % (str(sp3), str(sp4))]\n","\n","              if sspan_pred == 0:\n","                pred_consistent = False\n","                break\n","        elif pred_entailment == 0:\n","          pred_consistent = False\n","\n","  preds_consistent.append(1 if pred_consistent else 0)\n","  span_accuracies.append(span_accuracy / no_spans)\n","  span_accuracies_strict.append(span_accuracy_strict / no_spans)\n","  preds_coherent.append(1 if pred_coherent else 0)\n","\n","  # Check pred. breakpoint (verifiability) - will be first sentence where the model prediction becomes polarized, i.e., confidence > threshold\n","  pred_breakpoint = 0 # For now, 0 means -1, i.e., stories are entirely plausible - this shouldn't happen but it will (inconsistent?)\n","  for ss in range(1, ex['length']):\n","    if id_to_pred[exid + '-sp%s:%s' % (str(0), str(ss))] == 1:\n","      pred_breakpoint = ss\n","      break\n","  preds_breakpoint.append(pred_breakpoint)\n","\n","  # Check pred. evidence (verifiability)\n","  if pred_breakpoint > 0:\n","    pred_evidence = -1 \n","    for ss in range(0, pred_breakpoint+1):\n","      if id_to_pred[exid + '-sp%s:%s' % (str(0), str(ss))] == 1:\n","        pred_evidence = ss\n","  else:\n","    pred_evidence = -1 # This should never happen - it would be inconsistent if it did\n","  preds_evidence.append(pred_evidence)\n","\n","# Calculate tiered accuracy for model\n","acc = 0\n","acc_con = 0\n","acc_con_vbp = 0\n","acc_con_vbp_vev = 0\n","no_ex = len(ids_base)\n","for p_plaus, l_plaus, con, p_bp, l_bp, p_ev, l_ev in zip(preds_entailment, labels_entailment, preds_consistent, preds_breakpoint, labels_breakpoint, preds_evidence, labels_evidence):\n","  # Accuracy\n","  if p_plaus == l_plaus:\n","    acc += 1\n","    \n","    # Consistency\n","    if con == 1:\n","      acc_con += 1\n","    \n","      # Verifiability (breakpoint)\n","      if p_bp == l_bp:\n","        acc_con_vbp += 1\n","\n","        # Verifiability (evidence)\n","        if p_ev == l_ev:\n","          acc_con_vbp_vev += 1\n","\n","acc /= no_ex\n","acc_con /= no_ex\n","acc_con_vbp /= no_ex\n","acc_con_vbp_vev /= no_ex\n","\n","# all_results['acc'] = acc\n","# all_results['acc_con'] = acc_con\n","# all_results['acc_con_vbp'] = acc_con_vbp\n","# all_results['acc_con_vbp_vev'] = acc_con_vbp_vev\n","# all_results['span_accuracy'] = np.mean(span_accuracies)\n","\n","all_results['lenient_coherence'] = np.mean(span_accuracies_strict)\n","all_results['strict_coherence'] = np.mean(preds_coherent)\n","\n","best_preds_entailment = preds_entailment\n","best_preds_consistent = preds_consistent\n","best_preds_breakpoint = preds_breakpoint\n","best_preds_evidence = preds_evidence\n","best_preds_coherent = preds_coherent\n","    \n","print('\\nPARTITION: %s' % p)\n","print_dict(all_results)\n","\n","# Save preds for breakpoint and evidence\n","save_preds(ids_base, np.array(labels_breakpoint), best_preds_breakpoint, probe_model, p_dataset_name_bp)\n","save_preds(ids_base, np.array(labels_evidence), best_preds_evidence, probe_model, p_dataset_name_ev)\n","save_preds(ids_base, np.array([1 for p in best_preds_coherent]), best_preds_coherent, probe_model, p_dataset_name_coh)\n","\n","p_dataset_name_agg = '%s_tiers_agg_nostates_lenient_%s' % ('ConvEnt', p)\n","save_results(all_results, probe_model, p_dataset_name_agg)"]},{"cell_type":"markdown","metadata":{"id":"XnWDvQG7vtv-"},"source":["# ART Results\n","\n","Code for the coherence experiments on ART."]},{"cell_type":"code","execution_count":null,"metadata":{"id":"CvqFFHXebfrI"},"outputs":[],"source":["if task_name != 'art':\n","  raise ValueError('Please configure task_name in first cell to \"art\" to run ART results!')"]},{"cell_type":"markdown","metadata":{"id":"KxRnuX_fvtv_"},"source":["## Load ART dataset\n","\n","ART is originally gathered from [HuggingFace datasets](https://huggingface.co/docs/datasets/), but we added some of our own annotations for the coherence evaluation."]},{"cell_type":"code","execution_count":null,"metadata":{"id":"jGZGEObJvtv_"},"outputs":[],"source":["import os\n","fname = os.path.join(DRIVE_PATH, 'all_data/ART/art.json')\n","with open(fname, 'r') as f:\n","  art = json.load(f)"]},{"cell_type":"markdown","metadata":{"id":"apGoBEp_vtv_"},"source":["## Train Models on ART"]},{"cell_type":"markdown","metadata":{"id":"ukLFJ-Mfvtv_"},"source":["### Featurize ART"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"4LbWyaWKvtv_"},"outputs":[],"source":["from www.dataset.featurize import add_bert_features_art, get_tensor_dataset\n","seq_length = 32\n","\n","for p in art:\n","  for i in range(len(art[p])):\n","    art[p][i]['label'] -= 1 # Do this so labels start at 0\n","\n","  if debug:\n","    # Take 20 examples that we've annotated as the debug set so we can run the coherence metrics\n","    merged_file = os.path.join(DRIVE_PATH, 'all_data/ART/ART_test_rand200_annotation_merged2.json')\n","    ann_ids = [ex['id'] for ex in json.load(open(merged_file))]\n","     \n","    if p == 'train':\n","      art[p] = art[p][:20]\n","      art[p] = art[p][:20]\n","    elif p == 'val':\n","      art[p] = [ex for ex in art[p] if ex['id'] in ann_ids][:20]\n","\n","art_tensor = {}\n","for p in art:\n","  art[p] = add_bert_features_art(art[p], tokenizer, seq_length)\n","  art_tensor[p] = get_tensor_dataset(art[p], label_key='label')"]},{"cell_type":"markdown","metadata":{"id":"XERMrM56vtv_"},"source":["### Train Models\n","\n","Train models on ART. Note that ART's test set is not public, so we cannot test the model (unless we submit to their [leaderboard](https://leaderboard.allenai.org/anli/submissions/public))."]},{"cell_type":"markdown","metadata":{"id":"2u4dOjy4vtwA"},"source":["#### Configure Hyperparameters"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"Bz3-D5hVvtwA"},"outputs":[],"source":["batch_sizes = [config_batch_size]\n","learning_rates = [config_lr]\n","epochs = config_epochs\n","eval_batch_size = 128"]},{"cell_type":"markdown","metadata":{"id":"s-cs-hhJvtwA"},"source":["#### Grid Search"]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":28687,"status":"ok","timestamp":1631304042954,"user":{"displayName":"Shane Storks","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14Gh8CqjwUclCWWIoFwUB05rZNHwyd6nEIrTXC6RVXw=s64","userId":"00590987742306216924"},"user_tz":240},"id":"wRfuT81qvtwA","outputId":"e6e8ad02-9836-49e8-b0f3-615960aa5d80"},"outputs":[{"name":"stdout","output_type":"stream","text":["Beginning grid search for ART over 1 parameter combination(s)!\n","\n","TRAINING MODEL: bs=1, lr=1e-05\n"]},{"name":"stderr","output_type":"stream","text":["Some weights of the model checkpoint at roberta-large were not used when initializing RobertaForMultipleChoice: ['lm_head.bias', 'lm_head.dense.weight', 'lm_head.dense.bias', 'lm_head.layer_norm.weight', 'lm_head.layer_norm.bias', 'lm_head.decoder.weight']\n","- This IS expected if you are initializing RobertaForMultipleChoice from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n","- This IS NOT expected if you are initializing RobertaForMultipleChoice from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n","Some weights of RobertaForMultipleChoice were not initialized from the model checkpoint at roberta-large and are newly initialized: ['classifier.weight', 'classifier.bias']\n","You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n"]},{"name":"stdout","output_type":"stream","text":["[0] Beginning epoch...\n","\tBeginning evaluation...\n","\t\tRunning prediction...\n","\t\tComputing metrics...\n","\tFinished evaluation in 0:00:00s.\n","[0] Validation results:\n","{\n","  accuracy: \n","    0.7,\n","}\n","\n","\n","[0] Saving model checkpoint...\n","[0] Finished epoch.\n","Finished grid search! :)\n","Best validation accuracy 0.7 from model roberta-large_art_1_1e-05_0.\n"]}],"source":["from torch.utils.data import DataLoader, RandomSampler, SequentialSampler\n","from transformers import get_linear_schedule_with_warmup, RobertaForMultipleChoice, BertForMultipleChoice, RobertaConfig, BertConfig\n","from www.model.train import train_epoch\n","from www.model.eval import evaluate, save_results, save_preds\n","from sklearn.metrics import accuracy_score\n","from www.utils import print_dict, get_model_dir\n","\n","seed_val = 22 # Save random seed for reproducibility\n","random.seed(seed_val)\n","np.random.seed(seed_val)\n","torch.manual_seed(seed_val)\n","torch.cuda.manual_seed_all(seed_val)\n","\n","# We'll keep the validation data here with a constant eval batch size\n","dev_sampler = SequentialSampler(art_tensor['val'])\n","dev_dataloader = DataLoader(art_tensor['val'], sampler=dev_sampler, batch_size=eval_batch_size)\n","dev_dataset_name = 'art_val'\n","dev_ids = [str(ex['example_id']) for ex in art['val']]\n","\n","all_losses = []\n","param_combos = []\n","combo_names = []\n","all_val_accs = []\n","output_dirs = []\n","best_acc = 0.0\n","\n","print('Beginning grid search for ART over %s parameter combination(s)!' % (str(len(batch_sizes) * len(learning_rates))))\n","for bs in batch_sizes:\n","  for lr in learning_rates:\n","    print('\\nTRAINING MODEL: bs=%s, lr=%s' % (str(bs), str(lr)))\n","\n","    loss_values = []\n","    acc_values = []\n","\n","    # Set up training dataset with new batch size\n","    train_sampler = RandomSampler(art_tensor['train'])\n","    train_dataloader = DataLoader(art_tensor['train'], sampler=train_sampler, batch_size=bs)\n","\n","    # Set up model\n","    config = config_class.from_pretrained(model_name,\n","                                          num_labels=2,\n","                                          cache_dir=os.path.join(DRIVE_PATH, 'cache'))\n","    model = model_class.from_pretrained(model_name,\n","                                        config=config,\n","                                        cache_dir=os.path.join(DRIVE_PATH, 'cache'))\n","\n","    model.cuda()\n","    device = model.device \n","\n","    # Set up optimizer\n","    optimizer = AdamW(model.parameters(), lr=lr)\n","    total_steps = len(train_dataloader) * epochs\n","    scheduler = get_linear_schedule_with_warmup(optimizer, num_warmup_steps=0, num_training_steps = total_steps)\n","\n","    for epoch in range(epochs):\n","      # Train the model for one epoch\n","      print('[%s] Beginning epoch...' % str(epoch))\n","\n","      epoch_loss, _ = train_epoch(model, optimizer, train_dataloader, device)\n","      \n","      # Save loss\n","      loss_values.append(epoch_loss)\n","\n","      # Validate on dev set\n","      results, preds, labels = evaluate(model, dev_dataloader, device, [(accuracy_score, 'accuracy')])\n","      print('[%s] Validation results:' % str(epoch))\n","      print_dict(results)\n","\n","      # Save accuracy\n","      acc = results['accuracy']\n","      acc_values.append(acc)\n","      \n","      # Save model checkpoint\n","      print('[%s] Saving model checkpoint...' % str(epoch))\n","      model_param_str = get_model_dir(model_name.replace('/','-'), 'art', bs, lr, epoch)# + '_toy'\n","      output_dir = os.path.join(DRIVE_PATH, 'saved_models', model_param_str)\n","      output_dirs.append(output_dir)\n","      if not os.path.exists(output_dir):\n","        os.makedirs(output_dir)\n","      save_results(results, output_dir, dev_dataset_name)\n","      save_preds(dev_ids, labels, preds, output_dir, dev_dataset_name)\n","      model = model.module if hasattr(model, 'module') else model\n","      model.save_pretrained(output_dir)\n","      tokenizer.save_vocabulary(output_dir)\n","\n","      if acc > best_acc:\n","        best_acc = acc\n","        best_model = model_param_str\n","        best_dir = output_dir\n","\n","      print('[%s] Finished epoch.' % str(epoch))\n","\n","    all_losses.append(loss_values)\n","    all_val_accs.append(acc_values)\n","    param_combos.append((bs, lr))\n","    combo_names.append('bs=%s, lr=%s' % (str(bs), str(lr)))\n","\n","print('Finished grid search! :)')\n","print('Best validation accuracy %s from model %s.' % (best_acc, best_model))"]},{"cell_type":"markdown","metadata":{"id":"dPSVjSw9vtwB"},"source":["Delete non-best model checkpoints:"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"P-Pqf4g1vtwB"},"outputs":[],"source":["import shutil\n","\n","# Delete non-best model checkpoints\n","for od in output_dirs:\n","  if od != best_dir and os.path.exists(od):\n","    shutil.rmtree(od)"]},{"cell_type":"markdown","metadata":{"id":"LrNuHxTEvtwB"},"source":["## Coherence Checks on ART"]},{"cell_type":"markdown","metadata":{"id":"Egyx9BejvtwC"},"source":["### Load and Featurize Span Data"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"7mD8HXC9vtwC"},"outputs":[],"source":["from www.dataset.featurize import add_bert_features_art, get_tensor_dataset\n","from www.dataset.prepro import get_art_spans\n","import pickle\n","seq_length = 128\n","  \n","merged_file = os.path.join(DRIVE_PATH, 'all_data/ART/ART_test_rand200_annotation_merged2.json')\n","art_anns = json.load(open(merged_file))\n","\n","if debug:\n","  ann_ids = [ex['id'] for ex in art_anns]\n","  debug_ids = [ex['id'] for ex in art[p] if ex['id'] in ann_ids][:20]  \n","  art = [ex for ex in art_anns if ex['id'] in debug_ids]\n","\n","# Make span versions of the datasets\n","art_spans = get_art_spans(art)\n","\n","# Add BERT features\n","art = add_bert_features_art(art, tokenizer, seq_length, add_segment_ids=True)\n","art_spans = add_bert_features_art(art_spans, tokenizer, seq_length, add_segment_ids=True)\n","\n","# Add BERT features\n","art_tensor = get_tensor_dataset(art, label_key='label', add_segment_ids=True)\n","art_spans_tensor = get_tensor_dataset(art_spans, label_key='label', add_segment_ids=True)"]},{"cell_type":"markdown","metadata":{"id":"auYYxFc6vtwC"},"source":["### Load the Trained Model\n","\n","Load the trained model we want to probe and select the appropriate dataset."]},{"cell_type":"code","execution_count":null,"metadata":{"id":"ruiy_uXqvtwC"},"outputs":[],"source":["probe_model = eval_model_dir\n","probe_model = os.path.join(DRIVE_PATH, 'saved_models', probe_model)\n","  \n","# Load the model\n","model = model_class.from_pretrained(probe_model)\n","if torch.cuda.is_available():\n","  model.cuda()\n","device = model.device "]},{"cell_type":"markdown","metadata":{"id":"YwznwL-MvtwC"},"source":["#### Load Trained Model's Two-Story Classification Predictions\n","\n","For comparison, we also want the preds and labels for the previous level."]},{"cell_type":"code","execution_count":null,"metadata":{"id":"tdIPP8oPvtwC"},"outputs":[],"source":["from www.model.eval import load_preds\n","from www.utils import print_dict\n","\n","preds_base = {}\n","preds_base['val'] = load_preds(os.path.join(probe_model, 'preds_art_val.tsv'))"]},{"cell_type":"markdown","metadata":{"id":"fYRfQG6LvtwC"},"source":["### Calculate Coherence Metrics\n","\n","As ART is a multiple-choice task, we will need to tune the confidence threshold $\\rho$. This code will print out the strict and lenient coherence metrics, as well as the chosen $\\rho$ (`best_threshold`)."]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":3496,"status":"ok","timestamp":1631304420628,"user":{"displayName":"Shane Storks","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14Gh8CqjwUclCWWIoFwUB05rZNHwyd6nEIrTXC6RVXw=s64","userId":"00590987742306216924"},"user_tz":240},"id":"dCfpfrDXvtwD","outputId":"3d4bf905-11f1-41a9-889b-cb55114e67bb"},"outputs":[{"name":"stdout","output_type":"stream","text":["Testing model: drive/My Drive/Colab Notebooks/Research/TRIP_replication/saved_models/roberta-large_art_1_1e-05_0.\n","\tBeginning evaluation...\n","\t\tRunning prediction...\n","\t\tComputing metrics...\n","\tFinished evaluation in 0:00:03s.\n","\n","PARTITION: val \t METRIC: strict_coherence\n","chosen threshold: 1.0\n","{\n","  lenient_coherence: \n","    0.18246427120454473,\n","  strict_coherence: \n","    0.15,\n","  best_threshold: \n","    1.0,\n","}\n","\n","\n","\n","PARTITION: val \t METRIC: lenient_coherence\n","chosen threshold: 1.0\n","{\n","  lenient_coherence: \n","    0.18246427120454473,\n","  strict_coherence: \n","    0.15,\n","  best_threshold: \n","    1.0,\n","}\n","\n","\n"]},{"name":"stderr","output_type":"stream","text":["/usr/local/lib/python3.7/dist-packages/sklearn/metrics/_classification.py:1272: UndefinedMetricWarning: Precision is ill-defined and being set to 0.0 in labels with no predicted samples. Use `zero_division` parameter to control this behavior.\n","  _warn_prf(average, modifier, msg_start, len(result))\n","/usr/local/lib/python3.7/dist-packages/sklearn/metrics/_classification.py:1272: UndefinedMetricWarning: Recall is ill-defined and being set to 0.0 in labels with no true samples. Use `zero_division` parameter to control this behavior.\n","  _warn_prf(average, modifier, msg_start, len(result))\n"]}],"source":["from torch.utils.data import DataLoader, RandomSampler, SequentialSampler\n","from www.model.eval import evaluate, save_results, save_preds, list_comparison\n","from sklearn.metrics import accuracy_score, precision_score, recall_score, f1_score, confusion_matrix\n","metrics = [(accuracy_score, 'accuracy'), (precision_score, 'precision'), (recall_score, 'recall'), (f1_score, 'f1')]\n","import numpy as np\n","from www.utils import print_dict\n","\n","def is_polarized(smax, thres):\n","  return (abs(smax[0] - smax[1]) >= thres)\n","\n","print('Testing model: %s.' % probe_model)\n","\n","subtask = 'art'\n","p = 'val'\n","all_results = {}\n","\n","p_dataset = art_spans\n","p_tensor_dataset = art_spans_tensor\n","p_sampler = SequentialSampler(p_tensor_dataset)\n","p_dataloader = DataLoader(p_tensor_dataset, sampler=p_sampler, batch_size=128)\n","p_dataset_name = '%s_spans_%s' % (subtask, p)\n","p_dataset_name_co = '%s_consistent_%s' % (subtask, p)\n","p_dataset_name_bp = '%s_breakpoints_%s' % (subtask, p)\n","p_dataset_name_ev = '%s_evidence_%s' % (subtask, p)\n","p_dataset_name_coh = '%s_coherence_%s' % (subtask, p)\n","p_dataset_name_subset = '%s_rand200_%s' % (subtask, p)\n","p_ids = [ex['example_id'] for ex in art_spans]\n","p_labels = [ex['label'] for ex in art_spans]\n","\n","# Get span preds and save metrics\n","results, preds, labels, probs = evaluate(model, p_dataloader, device, metrics, seg_mode=True if 'roberta' not in mode else False, return_softmax=True)\n","save_results(results, probe_model, p_dataset_name)\n","save_preds(p_ids, labels, preds, probe_model, p_dataset_name)\n","\n","# Convert substory preds into breakpoint preds for each example\n","ids_base = [ex['example_id'] for ex in art]\n","\n","id_to_pred = {k: v for k,v in zip(p_ids, preds)}\n","id_to_prob = {k: v for k,v in zip(p_ids, probs)}\n","id_to_label = {k: v for k,v in zip(p_ids, p_labels)}\n","\n","for metric_to_optimize in ['strict_coherence', 'lenient_coherence']:\n","  # Get results dict ready\n","  # all_results['acc'] = 0.0\n","  # all_results['acc_con'] = 0.0\n","  # all_results['acc_con_vbp'] = 0.0\n","  # all_results['acc_con_vbp_vev'] = 0.0\n","  # all_results['span_accuracy'] = 0.0\n","  all_results['lenient_coherence'] = 0.0\n","  all_results['strict_coherence'] = 0.0\n","  span_accuracy = 0.0\n","  span_accuracy_strict = 0.0\n","  no_spans = 0\n","  for threshold in [0.05, 0.1, 0.15, 0.2, 0.25, 0.3, 0.35, 0.4, 0.45, 0.5, 0.55, 0.6, 0.65, 0.7, 0.75, 0.8, 0.85, 0.9, 0.95, 1.0]:\n","\n","    preds_plausible = []\n","    labels_plausible = []\n","    preds_consistent = []\n","    preds_breakpoint = []\n","    labels_breakpoint = []\n","    preds_evidence = []\n","    labels_evidence = []    \n","    span_accuracies = []\n","    span_accuracies_strict = []\n","    preds_coherent = []\n","\n","    for i, exid in enumerate(ids_base):\n","      ex = art[i]\n","      ex['length'] = 3\n","\n","      label_plausible = preds_base[p][exid]['label']\n","      pred_plausible = preds_base[p][exid]['pred']\n","      labels_plausible.append(label_plausible)\n","      preds_plausible.append(pred_plausible)\n","\n","      # Get ground truth breakpoint and evidence\n","      label_breakpoint = ex['conflict_pair'][1] if ex['conflict_pair'] is not None else 0\n","      labels_breakpoint.append(label_breakpoint)\n","      if label_breakpoint > 0:\n","        label_ev = ex['conflict_pair'][0]\n","      else:\n","        label_ev = -1\n","      labels_evidence.append(label_ev)\n","\n","      # Check consistency - for every span we confidently choose story X, we should also confidently choose story X for any span containing it\n","      pred_consistent = True\n","      for sp1 in range(ex['length']-1):\n","        if not pred_consistent:\n","          break\n","\n","        for sp2 in range(sp1+1, ex['length']):\n","          if not pred_consistent:\n","            break\n","\n","          span_pred = int(id_to_pred[exid + '-sp%s:%s' % (str(sp1), str(sp2))])\n","          span_prob = id_to_prob[exid + '-sp%s:%s' % (str(sp1), str(sp2))]\n","          span_label = max(id_to_label[exid + '-sp%s:%s' % (str(sp1), str(sp2))] - 1, -1)\n","\n","          span_pred3 = span_pred\n","          if not is_polarized(span_prob, threshold): # If not polarized, let's say -1\n","            span_pred3 = -1\n","\n","          pred_coherent = True\n","          if span_pred3 == span_label:\n","            span_accuracy += 1.0\n","            if label_plausible == pred_plausible:\n","              span_accuracy_strict += 1.0\n","          else:\n","            pred_coherent = False\n","          no_spans += 1\n","\n","          if is_polarized(span_prob, threshold):\n","            for sp3 in range(sp1+1):\n","              if not pred_consistent:\n","                break\n","\n","              for sp4 in range(sp2, ex['length']):\n","                if not pred_consistent:\n","                  break\n","\n","                sspan_pred = id_to_pred[exid + '-sp%s:%s' % (str(sp3), str(sp4))]\n","                sspan_prob = id_to_prob[exid + '-sp%s:%s' % (str(sp3), str(sp4))]\n","\n","                if not is_polarized(sspan_prob, threshold) or sspan_pred != span_pred:\n","                  pred_consistent = False\n","                  break\n","\n","      preds_consistent.append(1 if pred_consistent else 0)\n","      span_accuracies.append(span_accuracy / no_spans)\n","      span_accuracies_strict.append(span_accuracy_strict / no_spans)\n","      preds_coherent.append(1 if pred_coherent else 0)\n","\n","      # Check pred. breakpoint (verifiability) - will be first sentence where the model prediction becomes polarized, i.e., confidence > threshold\n","      pred_breakpoint  = 0 # For now, 0 means -1, i.e., stories are entirely plausible - this shouldn't happen but it will (inconsistent?)\n","      for ss in range(1, ex['length']):\n","        if is_polarized(id_to_prob[exid + '-sp%s:%s' % (str(0), str(ss))], threshold):\n","          pred_breakpoint = ss\n","          break\n","      preds_breakpoint.append(pred_breakpoint)\n","\n","      # Check pred. evidence (verifiability)\n","      if pred_breakpoint > 0:\n","        pred_evidence = -1 # Does this make sense for default value?\n","        for ss in range(0, pred_breakpoint):\n","          if is_polarized(id_to_prob[exid + '-sp%s:%s' % (str(ss), str(pred_breakpoint))], threshold):\n","            pred_evidence = ss\n","      else:\n","        pred_evidence = -1 # This should never happen - it would be inconsistent if it did?\n","      preds_evidence.append(pred_evidence)\n","\n","    # Calculate tiered accuracy for model\n","    acc = 0\n","    acc_con = 0\n","    acc_con_vbp = 0\n","    acc_con_vbp_vev = 0\n","    no_ex = len(ids_base)\n","    for p_plaus, l_plaus, con, p_bp, l_bp, p_ev, l_ev in zip(preds_plausible, labels_plausible, preds_consistent, preds_breakpoint, labels_breakpoint, preds_evidence, labels_evidence):\n","      # Accuracy\n","      if p_plaus == l_plaus:\n","        acc += 1\n","        \n","        # Consistency\n","        if con == 1:\n","          acc_con += 1\n","        \n","          # Verifiability (breakpoint)\n","          if p_bp == l_bp:\n","            acc_con_vbp += 1\n","\n","            # Verifiability (evidence)\n","            if p_ev == l_ev:\n","              acc_con_vbp_vev += 1\n","\n","    acc /= no_ex\n","    acc_con /= no_ex\n","    acc_con_vbp /= no_ex\n","    acc_con_vbp_vev /= no_ex\n","    span_acc = np.mean(span_accuracies)\n","    span_acc_strict = np.mean(span_accuracies_strict)\n","    coherence = np.mean(preds_coherent)\n","    # if coherence > all_results['coherence']: # !!!! this line is important\n","    # if span_acc > all_results['span_accuracy']: # !!!! this line is important\n","    if span_acc_strict > all_results[metric_to_optimize]: # !!!! this line is important\n","      # print('new best: %s' % str(acc_con_vbp_vev))\n","      best_thres = threshold\n","      \n","      # all_results['acc'] = acc\n","      # all_results['acc_con'] = acc_con\n","      # all_results['acc_con_vbp'] = acc_con_vbp\n","      # all_results['acc_con_vbp_vev'] = acc_con_vbp_vev\n","      # all_results['span_accuracy'] = span_acc\n","      all_results['lenient_coherence'] = span_acc_strict\n","      all_results['strict_coherence'] = coherence\n","\n","      best_preds_plausible = preds_plausible\n","      best_preds_consistent = preds_consistent\n","      best_preds_breakpoint = preds_breakpoint\n","      best_preds_evidence = preds_evidence\n","      best_preds_coherent = preds_coherent\n","      \n","  all_results['best_threshold'] = best_thres\n","  print('\\nPARTITION: %s \\t METRIC: %s' % (p, metric_to_optimize))\n","  print('chosen threshold: %s' % str(best_thres))\n","  print_dict(all_results)\n","\n","  # Save results\n","  p_dataset_name_agg = '%s_%s_%s' % (subtask, metric_to_optimize, p)\n","  save_results(all_results, probe_model, p_dataset_name_agg)"]}],"metadata":{"accelerator":"GPU","colab":{"authorship_tag":"ABX9TyN086hRntugR7AhDFlEbzLv","collapsed_sections":["XL9AU7zTgP9n","qqvj34KhLL0k","xm7qzKnAnbU9","FoY37xIF-oP7","QzFnAVtuUmpQ","lKY0hTEgnQgB","k_0WsycpFMdb","Cz5tcmScJrka","BxIYaEobhR7J","v7VlN2jUwvcC","5ctQweSlAceo","0q-xjfYU_cV8","-fQ6wXQIBdq1","U-BMInyrBdq2","8fRC3cnLBdq3","aWFmGRhznl2T","Rbvpm9irn3qL","RfXiCTA9KPjG","ON1UAnbc8OOE","W7MADzgfvtv3","Msxt3xAhvtv6","rn5Ywwvxvtv6","Ytbj9Uxxvtv7","h9YL5qnRvtv7","Jss8T2xzvtv8","8XhSrDlpI0aH","CADDFieTvtv9","maZGCMvMvtv-","XnWDvQG7vtv-","KxRnuX_fvtv_","apGoBEp_vtv_","ukLFJ-Mfvtv_","XERMrM56vtv_","Egyx9BejvtwC","auYYxFc6vtwC","fYRfQG6LvtwC"],"machine_shape":"hm","mount_file_id":"1RD-qTvUOr5V5r5QXWS4qCGVx2qM96ur2","name":"(code release) Consistent, Verifiable, and Coherent Reasoning for LMs","provenance":[]},"kernelspec":{"display_name":"Python 3","name":"python3"}},"nbformat":4,"nbformat_minor":0}