analysis.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Demo"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is the demo to showcase some analysis process. For the analysis for each task, we have provided a corresponding class. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "pycharm": {
     "is_executing": false
    }
   },
   "outputs": [],
   "source": [
    "# import analysis tools\n",
    "from analysis import SUMStat, D2TStat, WMTStat\n",
    "\n",
    "def truncate_print(l, n=10):\n",
    "    \"\"\" Print the first n items of a list\"\"\"\n",
    "    for i, x in enumerate(l):\n",
    "        if i == n:\n",
    "            print('...')\n",
    "            break\n",
    "        print(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Summarization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For all summarization datasets, including **REALSumm**, **SummEval** and **Newsroom**, the analysis tools are the same."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "pycharm": {
     "is_executing": false
    }
   },
   "outputs": [],
   "source": [
    "summ_stat = SUMStat('SUM/REALSumm/final_p.pkl') # The path to the scored file, _p means we have prompted metrics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "See what metrics are out there.<br>\n",
    "Since there are a lot, including P, R, F variants for some metrics as well as prompted metrics, we only print a truncated version of metrics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "pycharm": {
     "is_executing": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[All metrics]\n",
      "litepyramid_recall\n",
      "bert_score_p\n",
      "bert_score_r\n",
      "bert_score_f\n",
      "mover_score\n",
      "bart_score_src_hypo\n",
      "bart_score_hypo_ref\n",
      "bart_score_ref_hypo\n",
      "bart_score_avg_f\n",
      "bart_score_harm_f\n",
      "...\n",
      "[Automatic metrics]\n",
      "bert_score_p\n",
      "bert_score_r\n",
      "bert_score_f\n",
      "mover_score\n",
      "bart_score_src_hypo\n",
      "bart_score_hypo_ref\n",
      "bart_score_ref_hypo\n",
      "bart_score_avg_f\n",
      "bart_score_harm_f\n",
      "bart_score_cnn_src_hypo\n",
      "...\n",
      "[Human metrics]\n",
      "litepyramid_recall\n"
     ]
    }
   ],
   "source": [
    "print('[All metrics]')\n",
    "truncate_print(summ_stat.metrics) # change to print if you want to see all metrics\n",
    "print('[Automatic metrics]')\n",
    "truncate_print(summ_stat.auto_metrics)\n",
    "print('[Human metrics]')\n",
    "truncate_print(summ_stat.human_metrics)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can choose some metrics that we are interested in to conduct analysis.<br> \n",
    "For example, in **REALSumm**, we use recall-based metrics (e.g. bert_score_r, rouge1_r, bart_score_cnn_hypo_ref, ...)<br>\n",
    "For others, we use F-based metrics (for metrics that only consider hypo and ref) and src->hypo (for generation based metrics like bart_score and prism)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "pycharm": {
     "is_executing": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Human metric: litepyramid_recall\n",
      "metric                     spearman    kendalltau\n",
      "-----------------------  ----------  ------------\n",
      "rouge1_r                   0.497526      0.407974\n",
      "rougel_r                   0.488254      0.402523\n",
      "bart_score_cnn_hypo_ref    0.474608      0.374497\n",
      "bert_score_r               0.440398      0.346489\n",
      "rouge2_r                   0.4233        0.353119\n",
      "prism_hypo_ref             0.411005      0.323994\n",
      "mover_score                0.372353      0.290156\n"
     ]
    }
   ],
   "source": [
    "valid_metrics = [\n",
    "    'rouge1_r',\n",
    "    'rouge2_r',\n",
    "    'rougel_r',\n",
    "    'bert_score_r',\n",
    "    'mover_score',\n",
    "    'prism_hypo_ref',\n",
    "    'bart_score_cnn_hypo_ref'\n",
    "]\n",
    "\n",
    "# The first argument is the human metric considered.\n",
    "# The second argument is a list of considered automatic metrics, can omit it if all automatic metrics are considered\n",
    "summ_stat.evaluate_summary('litepyramid_recall', valid_metrics) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also see the performance of some prompt-based metrics."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "pycharm": {
     "is_executing": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Human metric: litepyramid_recall\n",
      "metric                                            spearman    kendalltau\n",
      "----------------------------------------------  ----------  ------------\n",
      "bart_score_cnn_hypo_ref_de_id est                 0.49539       0.392728\n",
      "bart_score_cnn_hypo_ref_de_Videlicet              0.491011      0.388237\n",
      "bart_score_cnn_hypo_ref_de_To give an instance    0.49081       0.387054\n",
      "bart_score_cnn_hypo_ref_de_To give an example     0.489033      0.38625\n",
      "bart_score_cnn_hypo_ref_de_As an illustration     0.488977      0.385511\n"
     ]
    }
   ],
   "source": [
    "valid_metrics = [\n",
    "    'bart_score_cnn_hypo_ref_de_id est',\n",
    "    'bart_score_cnn_hypo_ref_de_Videlicet',\n",
    "    'bart_score_cnn_hypo_ref_de_To give an instance',\n",
    "    'bart_score_cnn_hypo_ref_de_To give an example',\n",
    "    'bart_score_cnn_hypo_ref_de_As an illustration'\n",
    "]\n",
    "summ_stat.evaluate_summary('litepyramid_recall', valid_metrics)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To combine prompt-based metrics, run the following"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "pycharm": {
     "is_executing": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Human metric: litepyramid_recall\n",
      "metric                        spearman    kendalltau\n",
      "--------------------------  ----------  ------------\n",
      "bart_score_cnn_hypo_ref_de     0.48784      0.386398\n"
     ]
    }
   ],
   "source": [
    "summ_stat.combine_prompt()\n",
    "summ_stat.evaluate_summary('litepyramid_recall', ['bart_score_cnn_hypo_ref_de'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To conduct bootstrapping significant test, we provide the *sig_test_two ( )* and *sig_test ( )* method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "pycharm": {
     "is_executing": false
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 1000/1000 [01:28<00:00, 11.34it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "bart_score_cnn_hypo_ref is significantly better than bert_score_r\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "# The first two arguments are metrics that should be compared, the third argument is the human metric.\n",
    "m1 = 'bart_score_cnn_hypo_ref'\n",
    "m2 = 'bert_score_r'\n",
    "result = summ_stat.sig_test_two(m1, m2, 'litepyramid_recall')\n",
    "if result == 1:\n",
    "    print(f'{m1} is significantly better than {m2}')\n",
    "elif result == -1:\n",
    "    print(f'{m2} is significantly better than {m1}')\n",
    "else:\n",
    "    print('cannot decide')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "pycharm": {
     "is_executing": true
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 1000/1000 [01:28<00:00, 11.32it/s]\n",
      "100%|██████████| 1000/1000 [01:24<00:00, 11.81it/s]\n",
      "100%|██████████| 1000/1000 [01:26<00:00, 11.55it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Best metrics are: ['rouge1_r']\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "# The first arguments are a list of metrics considered\n",
    "# The second argument is the human metric\n",
    "summ_stat.sig_test(['rouge1_r', 'bart_score_cnn_hypo_ref', 'bert_score_r'], 'litepyramid_recall')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Factuality"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We use **Rank19** dataset and **QAGS_CNN** dataset to showcase some basic usages. The former uses accuracy as its evaluation metric while the latter uses pearson correlation."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Rank19"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We first print out the factuality accuracy obtained using different metrics for the **Rank19** dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "pycharm": {
     "is_executing": true
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "metric                           acc\n",
      "--------------------------  --------\n",
      "bart_score_cnn_src_hypo     0.836461\n",
      "bart_score_cnn_src_hypo_de  0.796247\n",
      "prism_src_hypo              0.780161\n",
      "bert_score_f                0.713137\n",
      "mover_score                 0.713137\n",
      "rouge2_f                    0.630027\n",
      "rougel_f                    0.587131\n",
      "rouge1_f                    0.568365\n"
     ]
    }
   ],
   "source": [
    "fact_stat = SUMStat('SUM/Rank19/final_p.pkl')\n",
    "fact_stat.combine_prompt()\n",
    "\n",
    "# Set valid metrics\n",
    "valid_metrics = [\n",
    "    'rouge1_f',\n",
    "    'rouge2_f',\n",
    "    'rougel_f',\n",
    "    'bert_score_f',\n",
    "    'mover_score',\n",
    "    'prism_src_hypo',\n",
    "    'bart_score_cnn_src_hypo',\n",
    "    'bart_score_cnn_src_hypo_de'\n",
    "]\n",
    "\n",
    "# Print accuracy, take a list of metrics\n",
    "fact_stat.get_fact_acc(valid_metrics)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Below are some methods that help to facilitate the siginificant test."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "pycharm": {
     "is_executing": true
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 1000/1000 [00:01<00:00, 744.17it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "bart_score_cnn_src_hypo is significantly better than bert_score_f\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "m1 = 'bart_score_cnn_src_hypo'\n",
    "m2 = 'bert_score_f'\n",
    "result = fact_stat.fact_acc_sig_test_two(m1, m2)\n",
    "if result == 1:\n",
    "    print(f'{m1} is significantly better than {m2}')\n",
    "elif result == -1:\n",
    "    print(f'{m2} is significantly better than {m1}')\n",
    "else:\n",
    "    print('cannot decide')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "pycharm": {
     "is_executing": true
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 1000/1000 [00:00<00:00, 2082.68it/s]\n",
      "100%|██████████| 1000/1000 [00:01<00:00, 666.78it/s]\n",
      "100%|██████████| 1000/1000 [00:01<00:00, 614.94it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Best metrics are: ['bart_score_cnn_src_hypo']\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "# Take a list of metrics, print the best metrics\n",
    "fact_stat.fact_acc_sig_test(['bart_score_cnn_src_hypo', 'prism_src_hypo', 'bert_score_f'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### QAGS_CNN"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "pycharm": {
     "is_executing": true
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "metric                        pearson\n",
      "--------------------------  ---------\n",
      "bart_score_cnn_src_hypo      0.734672\n",
      "bart_score_cnn_src_hypo_de   0.718525\n",
      "bert_score_f                 0.575994\n",
      "prism_src_hypo               0.478689\n",
      "rouge2_f                     0.459141\n",
      "mover_score                  0.41414\n",
      "rougel_f                     0.356889\n",
      "rouge1_f                     0.337667\n"
     ]
    }
   ],
   "source": [
    "fact_stat = SUMStat('SUM/QAGS_CNN/final_p.pkl')\n",
    "fact_stat.combine_prompt()\n",
    "\n",
    "# Set valid metrics\n",
    "valid_metrics = [\n",
    "    'rouge1_f',\n",
    "    'rouge2_f',\n",
    "    'rougel_f',\n",
    "    'bert_score_f',\n",
    "    'mover_score',\n",
    "    'prism_src_hypo',\n",
    "    'bart_score_cnn_src_hypo',\n",
    "    'bart_score_cnn_src_hypo_de'\n",
    "]\n",
    "\n",
    "# Print accuracy, take a list of metrics\n",
    "fact_stat.get_fact_pearson(valid_metrics)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "pycharm": {
     "is_executing": true
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 1000/1000 [00:00<00:00, 1177.00it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "bart_score_cnn_src_hypo is significantly better than bert_score_f\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "m1 = 'bart_score_cnn_src_hypo'\n",
    "m2 = 'bert_score_f'\n",
    "result = fact_stat.fact_pearson_sig_test_two(m1, m2)\n",
    "if result == 1:\n",
    "    print(f'{m1} is significantly better than {m2}')\n",
    "elif result == -1:\n",
    "    print(f'{m2} is significantly better than {m1}')\n",
    "else:\n",
    "    print('cannot decide')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "pycharm": {
     "is_executing": true
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 1000/1000 [00:00<00:00, 1986.75it/s]\n",
      "100%|██████████| 1000/1000 [00:00<00:00, 1156.13it/s]\n",
      "100%|██████████| 1000/1000 [00:00<00:00, 1173.93it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Best metrics are: ['bart_score_cnn_src_hypo']\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "# Take a list of metrics, print the best metrics\n",
    "fact_stat.fact_pearson_sig_test(['bart_score_cnn_src_hypo', 'prism_src_hypo', 'bert_score_f'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data-to-Text"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For all data-to-text datasets, including **BAGEL**, **SFHOT** and **SFRES**, the analysis tools are the same."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "pycharm": {
     "is_executing": true
    }
   },
   "outputs": [],
   "source": [
    "d2t_stat = D2TStat('D2T/BAGEL/final_p.pkl')\n",
    "d2t_stat.combine_prompt() # combine the prompt-based resutls"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "See what metrics are out there. For data-to-text datasets, the human metrics are *informativeness*, *naturalness* and *quality*."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "pycharm": {
     "is_executing": true
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[All metrics]\n",
      "informativeness\n",
      "naturalness\n",
      "quality\n",
      "bert_score_p\n",
      "bert_score_r\n",
      "bert_score_f\n",
      "mover_score\n",
      "bart_score_ref_hypo\n",
      "bart_score_hypo_ref\n",
      "bart_score_avg_f\n",
      "...\n",
      "[Automatic metrics]\n",
      "bert_score_p\n",
      "bert_score_r\n",
      "bert_score_f\n",
      "mover_score\n",
      "bart_score_ref_hypo\n",
      "bart_score_hypo_ref\n",
      "bart_score_avg_f\n",
      "bart_score_harm_f\n",
      "bart_score_cnn_ref_hypo\n",
      "bart_score_cnn_hypo_ref\n",
      "...\n",
      "[Human metrics]\n",
      "informativeness\n",
      "naturalness\n",
      "quality\n"
     ]
    }
   ],
   "source": [
    "print('[All metrics]')\n",
    "truncate_print(d2t_stat.metrics) # change to print if you want to see all metrics\n",
    "print('[Automatic metrics]')\n",
    "truncate_print(d2t_stat.auto_metrics)\n",
    "print('[Human metrics]')\n",
    "truncate_print(d2t_stat.human_metrics)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can print out the correlation w.r.t. human judgement as below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "pycharm": {
     "is_executing": true
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Human metric: informativeness\n",
      "metric                      spearman    kendalltau\n",
      "------------------------  ----------  ------------\n",
      "bart_score_para_avg_f_de    0.335997      0.248525\n",
      "bart_score_para_avg_f       0.329896      0.246686\n",
      "prism_avg                   0.304946      0.224797\n",
      "bert_score_f                0.289118      0.217179\n",
      "mover_score                 0.283694      0.20884\n",
      "rouge1_f                    0.234338      0.177972\n",
      "rouge2_f                    0.198585      0.151011\n",
      "rougel_f                    0.188592      0.145508\n"
     ]
    }
   ],
   "source": [
    "# Set valid metrics\n",
    "valid_metrics = [\n",
    "    'rouge1_f',\n",
    "    'rouge2_f',\n",
    "    'rougel_f',\n",
    "    'bert_score_f',\n",
    "    'mover_score',\n",
    "    'prism_avg',\n",
    "    'bart_score_para_avg_f',\n",
    "    'bart_score_para_avg_f_de'\n",
    "]\n",
    "\n",
    "# The first argument is human metric while the latter is a list of metrics considered.\n",
    "d2t_stat.evaluate_text('informativeness', valid_metrics)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To perform significant test, use *sig_test_two ( )* method"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "pycharm": {
     "is_executing": true
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 1000/1000 [01:19<00:00, 12.54it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "bart_score_para_avg_f is significantly better than prism_avg\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "m1 = 'bart_score_para_avg_f'\n",
    "m2 = 'prism_avg'\n",
    "\n",
    "# The first two arguments are metrics that should be compared, the third argument is the human metric.\n",
    "result = d2t_stat.sig_test_two(m1, m2, 'informativeness')\n",
    "\n",
    "if result == 1:\n",
    "    print(f'{m1} is significantly better than {m2}')\n",
    "elif result == -1:\n",
    "    print(f'{m2} is significantly better than {m1}')\n",
    "else:\n",
    "    print('cannot decide')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Machine Translation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For all language pairs, the analysis tools are the same. We begin by looking at reference length statistics."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "pycharm": {
     "is_executing": true
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Mean reference length: 17.75\n",
      "Max reference length: 180\n",
      "Min reference length: 1\n",
      "20% percentile: 10.0\n",
      "80% percentile: 25.0\n",
      "90% percentile: 31.0\n"
     ]
    }
   ],
   "source": [
    "wmt_stat = WMTStat('WMT/kk-en/final_p.pkl')\n",
    "wmt_stat.print_ref_len()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we print out k-tau for all automatic metrics."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 64%|██████▎   | 7/11 [00:00<00:00, 69.64it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "All metrics\n",
      "['bleu', 'chrf', 'bleurt', 'prism', 'comet', 'bert_score', 'bart_score', 'bart_score_cnn', 'bart_score_para', 'bart_score_para_en_Such as', 'bart_score_para_de_Such as']\n",
      "\n",
      "\n",
      "All k-tau\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 11/11 [00:00<00:00, 64.20it/s]\n",
      "100%|██████████| 2/2 [00:00<00:00, 67.73it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "metric                         k-tau\n",
      "--------------------------  --------\n",
      "bart_score_para_de_Such as  0.38014\n",
      "bart_score_para             0.378495\n",
      "comet                       0.378289\n",
      "bart_score_para_en_Such as  0.375822\n",
      "bleurt                      0.371505\n",
      "prism                       0.362048\n",
      "bert_score                  0.350535\n",
      "bart_score_cnn              0.347656\n",
      "bart_score                  0.324424\n",
      "chrf                        0.322985\n",
      "bleu                        0.276316\n",
      "\n",
      "\n",
      "k-tau for some metrics\n",
      "metric              k-tau\n",
      "---------------  --------\n",
      "bart_score_para  0.378495\n",
      "prism            0.362048\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "print('All metrics')\n",
    "print(wmt_stat.metrics) # Print out all metrics\n",
    "print('\\n')\n",
    "print('All k-tau')\n",
    "wmt_stat.print_ktau()\n",
    "print('\\n')\n",
    "print('k-tau for some metrics')\n",
    "wmt_stat.print_ktau(['prism', 'bart_score_para'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To print out the k-tau over certain reference length, run the following."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 9728/9728 [00:00<00:00, 648147.63it/s]\n",
      "100%|██████████| 11/11 [00:00<00:00, 179.12it/s]\n",
      "100%|██████████| 9728/9728 [00:00<00:00, 625838.84it/s]\n",
      "100%|██████████| 2/2 [00:00<00:00, 194.46it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "All k-tau\n",
      "Considered samples: 3545\n",
      "metric                         k-tau\n",
      "--------------------------  --------\n",
      "comet                       0.351763\n",
      "bart_score_para             0.335966\n",
      "bert_score                  0.332581\n",
      "bart_score_para_de_Such as  0.332017\n",
      "bart_score_para_en_Such as  0.331453\n",
      "prism                       0.322426\n",
      "bleurt                      0.321862\n",
      "chrf                        0.318477\n",
      "bart_score                  0.305501\n",
      "bleu                        0.300987\n",
      "bart_score_cnn              0.300987\n",
      "\n",
      "\n",
      "k-tau for some metrics\n",
      "Considered samples: 3545\n",
      "metric              k-tau\n",
      "---------------  --------\n",
      "bart_score_para  0.335966\n",
      "prism            0.322426\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "print('All k-tau')\n",
    "wmt_stat.print_len_ktau(15, 25)\n",
    "print('\\n')\n",
    "print('k-tau for some metrics')\n",
    "wmt_stat.print_len_ktau(15, 25, ['prism', 'bart_score_para'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To perform significant test, use *sig_test_two ()*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 1000/1000 [00:33<00:00, 29.77it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "bart_score_para is significantly better than bleurt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "m1 = 'bart_score_para'\n",
    "m2 = 'bleurt'\n",
    "\n",
    "# The first two arguments are metrics that should be compared, the third argument is the human metric.\n",
    "result = wmt_stat.sig_test_two(m1, m2)\n",
    "\n",
    "if result == 1:\n",
    "    print(f'{m1} is significantly better than {m2}')\n",
    "elif result == -1:\n",
    "    print(f'{m2} is significantly better than {m1}')\n",
    "else:\n",
    "    print('cannot decide')"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  },
  "pycharm": {
   "stem_cell": {
    "cell_type": "raw",
    "metadata": {
     "collapsed": false
    },
    "source": []
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}