3c-Data_Cleaning_KeFVP_emotion2vec.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "d912700f-d412-4827-bf51-73285938ce34",
   "metadata": {},
   "source": [
    "# Earnings Call Project: Data Cleaning with EMOTION2VEC\n",
    "<br>\n",
    "CIS 831 Deep Learning – Term Project<br>\n",
    "Kansas State University\n",
    "<br><br>\n",
    "James Chapman<br>\n",
    "John Woods<br>\n",
    "Nathan Diehl<br>\n",
    "<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b0d09ba3-32b9-47a0-a079-1375b22e3afe",
   "metadata": {},
   "source": [
    "##  With this notebook I want to use the target data from the [GitHub repository](https://github.com/hankniu01/KeFVP/tree/main) for the paper [KeFVP: Knowledge-enhanced Financial Volatility Prediction](https://aclanthology.org/2023.findings-emnlp.770.pdf)\n",
    "\n",
    "This notebook creates data used for training/testing.\n",
    "- investigate KeFVP, compared to ours\n",
    "- Grabs the targets from KeFVP\n",
    "- Corrects Praat features for both datasets\n",
    "- Combines features audio (Praat) & text (6 different) and targets AND EMOTION2VEC \n",
    "- save 9 numpy files specifically for HTML (for each audio/text pair)\n",
    "    - train (features, targets, secondary_targets)\n",
    "    - validation (features, targets, secondary_targets)\n",
    "    - test (features, targets, secondary_targets)\n",
    "- KeFVP also dropped about the same number of meetings as we did! But we can compare our results to KeFVP directly<br>\n",
    "\n",
    "- 12 meetings were removed from our data because we could not find stock data (Yahoo or alphadvantage)<br>\n",
    "- 218 meetings were removed from MAEC <br>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "a88d0789-36df-4250-90e7-1485c0dc9dfe",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import os\n",
    "from datetime import datetime\n",
    "from tqdm import tqdm\n",
    "\n",
    "tqdm.pandas()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "f5822f8a-897e-4fd5-b77b-1c38e8d7ae07",
   "metadata": {},
   "outputs": [],
   "source": [
    "MAEC_dir = 'data/MAEC/MAEC_Dataset' # https://github.com/Earnings-Call-Dataset/MAEC-A-Multimodal-Aligned-Earnings-Conference-Call-Dataset-for-Financial-Risk-Prediction\n",
    "\n",
    "############# too big for GitHub ########################\n",
    "############# stored on local disk ######################\n",
    "original_data_dir = r\"D:\\original_dataset\" # https://github.com/GeminiLn/EarningsCall_Dataset \n",
    "MAEC_audio_dir = r\"D:\\MAEC_audio\" \n",
    "# there is a link for the audio data in the MAEC GitHub, but it does not work\n",
    "# I emailed the authors, and they send another link.\n",
    "# There is like a half-million files, but only 19 GB\n",
    "# https://drive.google.com/file/d/1m1GRCHgKn9Vz9IFMC_SpCog6uP3-gFgY/view?usp=drive_link \n",
    "\n",
    "# from Feature Engineering\n",
    "glove_features = pd.read_csv('data/data_prep/glove_features.csv')\n",
    "praat_features = pd.read_csv('data/data_prep/praat_features.csv', low_memory=False)\n",
    "MAEC_glove_features = pd.read_csv('data/data_prep/MAEC_glove_features.csv')\n",
    "MAEC_praat_features = pd.read_csv('data/data_prep/MAEC_praat_features.csv', low_memory=False)\n",
    "\n",
    "# from Feature Engineering MORE\n",
    "RoBERTa_features = pd.read_csv('data/data_prep/RoBERTa_features.csv', low_memory=False)\n",
    "MAEC_RoBERTa_features = pd.read_csv('data/data_prep/MAEC_RoBERTa_features.csv', low_memory=False)\n",
    "# Roberta with averages\n",
    "RoBERTa_features2 = pd.read_csv('data/data_prep/RoBERTa_features2.csv', low_memory=False)\n",
    "MAEC_RoBERTa_features2 = pd.read_csv('data/data_prep/MAEC_RoBERTa_features2.csv', low_memory=False)\n",
    "\n",
    "investopedia_features = pd.read_csv('data/data_prep/investopedia_features.csv', low_memory=False)\n",
    "MAEC_investopedia_features = pd.read_csv('data/data_prep/MAEC_investopedia_features.csv', low_memory=False)\n",
    "bge_features = pd.read_csv('data/data_prep/bge_features.csv', low_memory=False)\n",
    "MAEC_bge_features = pd.read_csv('data/data_prep/MAEC_bge_features.csv', low_memory=False)\n",
    "bge_base_features = pd.read_csv('data/data_prep/bge_base_features.csv', low_memory=False)\n",
    "MAEC_bge_base_features = pd.read_csv('data/data_prep/MAEC_bge_base_features.csv', low_memory=False)\n",
    "\n",
    "################################### \n",
    "# # add emotion2vec\n",
    "emotion2vec = pd.read_csv('data/data_prep/emotion2vec.csv', low_memory=False)\n",
    "MAEC_emotion2vec = pd.read_csv('data/data_prep/MAEC_emotion2vec.csv', low_memory=False)\n",
    "MAEC_emotion2vec = MAEC_emotion2vec.rename(columns={'Company':'Ticker'})\n",
    "###################################"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "32fd74f4-2546-4c1f-8939-206d43abdf9a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Loop through the directory, each folder represents an earnings conference call; the folders are named as \"CompanyName_Date\".\n",
    "filename_data = []\n",
    "for filename in os.listdir(original_data_dir):\n",
    "    company_name, date_str = filename.rsplit('_', 1)\n",
    "    date_str = date_str.split('.')[0] \n",
    "    date = datetime.strptime(date_str, \"%Y%m%d\").strftime(\"%Y-%m-%d\")\n",
    "    filename_data.append([company_name, date])\n",
    "filename_data = pd.DataFrame(filename_data, columns=[\"Company\", \"Date\"])\n",
    "company_ticker = pd.read_csv('data/data_prep/company_ticker.csv')\n",
    "filename_data = filename_data.merge(company_ticker, on=\"Company\", how=\"left\")\n",
    "\n",
    "# Loop through the directory, each folder represents an earnings conference call; the folders are named as \"Date_CompanyName\".\n",
    "MAEC_filename_data = []\n",
    "for filename in os.listdir(MAEC_dir):\n",
    "    date_str, ticker = filename.rsplit('_', 1)\n",
    "    date_str = date_str.split('.')[0] \n",
    "    date = datetime.strptime(date_str, \"%Y%m%d\").strftime(\"%Y-%m-%d\")\n",
    "    MAEC_filename_data.append([ticker, date])\n",
    "MAEC_filename_data = pd.DataFrame(MAEC_filename_data, columns=[\"Ticker\", \"Date\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fbb88572-dfc3-4e8f-80c3-9319683e4262",
   "metadata": {},
   "source": [
    "# Add TARGET of the regression\n",
    "\n",
    "**n-day volatility predictions**: The predicted average volatility over the following n days.<br>\n",
    "\n",
    "$$\n",
    "v[0,n] = \\ln \\left( \\sqrt{ \\frac{1}{n} \\sum_{i=1}^{n} (r_i - \\bar{r})^2 } \\right)\n",
    "$$\n",
    "\n",
    "Where:\n",
    "- \\( r_i \\) is the stock return on day \\(i\\),\n",
    "- \\( \\bar{r} \\) is the average stock return over \\(n\\) days.\n",
    "\n",
    "The stock return \\(r_i\\) is defined as:\n",
    "\n",
    "$$\n",
    "r_i = \\frac{P_i - P_{i-1}}{P_{i-1}}\n",
    "$$\n",
    "\n",
    "Where \\(P_i\\) is the adjusted closing price of the stock on day \\(i\\).\n",
    "\n",
    "For **single-day log volatility**, we estimate it using the **daily log absolute return**:\n",
    "\n",
    "$$\n",
    "v_n = \\ln \\left( \\left| \\frac{P_n - P_{n-1}}{P_{n-1}} \\right| \\right)\n",
    "$$\n",
    "\n",
    "Where:\n",
    "- \\(P_n\\) is the adjusted closing price of the stock on day \\(n\\),\n",
    "- \\(P_{n-1}\\) is the adjusted closing price on the previous day.\n",
    "\n",
    "Our multi-task learning objective is to **simultaneously predict** these two quantities:\n",
    "- \\(v[0,n]\\): The average volatility over \\(n\\) days (the main task).\n",
    "- \\(v_n\\): The single-day volatility (the auxiliary task).\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "f118fbe1-9457-4d6e-9577-9702da80bda7",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "\n",
    "def get_KeFVP(n_days_filename, single_day_filename):\n",
    "    n_days_data = pd.read_csv(f'data/KeFVP_price_data/{n_days_filename}.csv')\n",
    "    n_days_data['Date'] = pd.to_datetime(n_days_data[['year', 'month', 'day']])\n",
    "    n_days_data.drop_duplicates(subset=['Date', 'ticker'], inplace=True)\n",
    "    # drop this row specifically, is no transcript/ udio for this\n",
    "    n_days_data = n_days_data[~((n_days_data['Date'] == '2017-10-31') & \n",
    "                                        (n_days_data['name'] == 'American Tower Corp A'))]\n",
    "    # single day volatility\n",
    "    single_day_data = pd.read_csv(f'data/KeFVP_price_data/{single_day_filename}.csv')\n",
    "    single_day_data['Date'] = pd.to_datetime(single_day_data[['year', 'month', 'day']])\n",
    "    single_day_data.drop_duplicates(subset=['Date', 'ticker'], inplace=True)\n",
    "    single_day_data = single_day_data[['future_Single_3', 'future_Single_7', 'future_Single_15', \n",
    "                                       'future_Single_30', 'Date', 'ticker', 'name']].copy()# , 'name'\n",
    "    # merge\n",
    "    target_data = n_days_data[['future_3', 'future_7', 'future_15', 'future_30', 'Date', 'ticker', 'name']].copy()# , 'name'\n",
    "    target_data = pd.merge(target_data, single_day_data, how=\"left\",on = ['Date','ticker', 'name'])\n",
    "    target_data = target_data.rename(columns={'ticker':'Ticker', \n",
    "                                              'name':'Company',\n",
    "                                            'future_3':'3_day',\n",
    "                                            'future_7':'7_day',\n",
    "                                            'future_15':'15_day',\n",
    "                                            'future_30':'30_day',\n",
    "                                            'future_Single_3':'3_day_single',\n",
    "                                            'future_Single_7':'7_day_single',\n",
    "                                            'future_Single_15':'15_day_single',\n",
    "                                            'future_Single_30':'30_day_single'})\n",
    "    print('should all be the same,- ', len(n_days_data), len(single_day_data), len(target_data))\n",
    "    return target_data\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f4da9006",
   "metadata": {},
   "source": [
    "Investigate KeFVP target data. Compared to our data from Yahoo/Alphadvantage"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "dc2b8434",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "should all be the same,-  391 391 391\n",
      "should all be the same,-  56 56 56\n",
      "should all be the same,-  111 112 111\n",
      "272\n",
      "280\n",
      "280\n",
      "21\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Ticker_x</th>\n",
       "      <th>Company</th>\n",
       "      <th>Ticker_y</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>ADS</td>\n",
       "      <td>Alliance Data Systems</td>\n",
       "      <td>BFH</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>ABC</td>\n",
       "      <td>AmerisourceBergen Corp</td>\n",
       "      <td>COR</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>ANTM</td>\n",
       "      <td>Anthem Inc.</td>\n",
       "      <td>ELV</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>35</th>\n",
       "      <td>BLL</td>\n",
       "      <td>Ball Corp</td>\n",
       "      <td>BALL</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>54</th>\n",
       "      <td>CTL</td>\n",
       "      <td>CenturyLink Inc</td>\n",
       "      <td>LUMN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>58</th>\n",
       "      <td>XEC</td>\n",
       "      <td>Cimarex Energy</td>\n",
       "      <td>CTRA</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>95</th>\n",
       "      <td>FB</td>\n",
       "      <td>Facebook, Inc.</td>\n",
       "      <td>META</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>103</th>\n",
       "      <td>FBHS</td>\n",
       "      <td>Fortune Brands Home &amp; Security</td>\n",
       "      <td>FBIN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>106</th>\n",
       "      <td>GPS</td>\n",
       "      <td>Gap Inc.</td>\n",
       "      <td>GAP</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>177</th>\n",
       "      <td>MYL</td>\n",
       "      <td>Mylan N.V.</td>\n",
       "      <td>VTRS</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>200</th>\n",
       "      <td>PKI</td>\n",
       "      <td>PerkinElmer</td>\n",
       "      <td>RVTY</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>212</th>\n",
       "      <td>RHI.F</td>\n",
       "      <td>Red Hat Inc.</td>\n",
       "      <td>RHT</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>252</th>\n",
       "      <td>TWTR</td>\n",
       "      <td>Twitter, Inc.</td>\n",
       "      <td>X</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    Ticker_x                         Company Ticker_y\n",
       "18       ADS           Alliance Data Systems      BFH\n",
       "24       ABC          AmerisourceBergen Corp      COR\n",
       "27      ANTM                     Anthem Inc.      ELV\n",
       "35       BLL                       Ball Corp     BALL\n",
       "54       CTL                 CenturyLink Inc     LUMN\n",
       "58       XEC                  Cimarex Energy     CTRA\n",
       "95        FB                  Facebook, Inc.     META\n",
       "103     FBHS  Fortune Brands Home & Security     FBIN\n",
       "106      GPS                        Gap Inc.      GAP\n",
       "177      MYL                      Mylan N.V.     VTRS\n",
       "200      PKI                     PerkinElmer     RVTY\n",
       "212    RHI.F                    Red Hat Inc.      RHT\n",
       "252     TWTR                   Twitter, Inc.        X"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "\n",
    "train_n_days_filename = 'train_split_Avg_Series_WITH_LOG'\n",
    "train_single_day_filename = 'train_split_SeriesSingleDayVol3'\n",
    "val_n_days_filename = 'val_split_Avg_Series_WITH_LOG'\n",
    "val_single_day_filename = 'val_split_SeriesSingleDayVol3'\n",
    "test_n_days_filename = 'test_split_Avg_Series_WITH_LOG'\n",
    "test_single_day_filename = 'test_split_SeriesSingleDayVol3'\n",
    "\n",
    "train_targets = get_KeFVP(train_n_days_filename, train_single_day_filename)\n",
    "val_targets = get_KeFVP(val_n_days_filename, val_single_day_filename)\n",
    "test_targets = get_KeFVP(test_n_days_filename, test_single_day_filename)\n",
    "\n",
    "# merge all  train/validate/test\n",
    "EC_dataset = pd.concat([train_targets, val_targets, test_targets], ignore_index=True)\n",
    "\n",
    "# grab all unique ticker/company\n",
    "KeFVP_ticker_Company = EC_dataset[['Ticker','Company']].copy()\n",
    "KeFVP_ticker_Company['Ticker'] = KeFVP_ticker_Company['Ticker'].str.upper()\n",
    "KeFVP_ticker_Company.drop_duplicates(inplace=True)\n",
    "print(len(KeFVP_ticker_Company))\n",
    "\n",
    "# original unique ticker/company\n",
    "company_ticker = pd.read_csv('data/data_prep/company_ticker.csv')\n",
    "print(len(company_ticker))\n",
    "combo = pd.merge(KeFVP_ticker_Company, company_ticker, how=\"right\",on = ['Company'])\n",
    "print(len(combo))\n",
    "print(len(combo[combo['Ticker_y']!=combo['Ticker_x']]))\n",
    "problems = combo[combo['Ticker_y']!=combo['Ticker_x']].copy()\n",
    "problems.dropna(inplace=True)\n",
    "problems"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "28c2a6cf",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Ticker_x</th>\n",
       "      <th>Company</th>\n",
       "      <th>Ticker_y</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>110</th>\n",
       "      <td>NaN</td>\n",
       "      <td>General Growth Properties Inc.</td>\n",
       "      <td>GGP</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>120</th>\n",
       "      <td>NaN</td>\n",
       "      <td>Harris Corporation</td>\n",
       "      <td>LHX</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>135</th>\n",
       "      <td>NaN</td>\n",
       "      <td>Ingersoll-Rand PLC</td>\n",
       "      <td>IR</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>160</th>\n",
       "      <td>NaN</td>\n",
       "      <td>Lowe's Cos.</td>\n",
       "      <td>LOW</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>170</th>\n",
       "      <td>NaN</td>\n",
       "      <td>Michael Kors Holdings</td>\n",
       "      <td>CPRI</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>222</th>\n",
       "      <td>NaN</td>\n",
       "      <td>SCANA Corp</td>\n",
       "      <td>SCG</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>235</th>\n",
       "      <td>NaN</td>\n",
       "      <td>Symantec Corp.</td>\n",
       "      <td>GEN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>264</th>\n",
       "      <td>NaN</td>\n",
       "      <td>Viacom Inc.</td>\n",
       "      <td>VIACOM</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    Ticker_x                         Company Ticker_y\n",
       "110      NaN  General Growth Properties Inc.      GGP\n",
       "120      NaN              Harris Corporation      LHX\n",
       "135      NaN              Ingersoll-Rand PLC       IR\n",
       "160      NaN                     Lowe's Cos.      LOW\n",
       "170      NaN           Michael Kors Holdings     CPRI\n",
       "222      NaN                      SCANA Corp      SCG\n",
       "235      NaN                  Symantec Corp.      GEN\n",
       "264      NaN                     Viacom Inc.   VIACOM"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# companies that are not included in the KeFVP  data\n",
    "combo[combo.isnull().any(axis=1)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "34e2c3e8",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Empty DataFrame\n",
      "Columns: [Ticker, Company]\n",
      "Index: []\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Company</th>\n",
       "      <th>Ticker</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>110</th>\n",
       "      <td>General Growth Properties Inc.</td>\n",
       "      <td>GGP</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>120</th>\n",
       "      <td>Harris Corporation</td>\n",
       "      <td>LHX</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>135</th>\n",
       "      <td>Ingersoll-Rand PLC</td>\n",
       "      <td>IR</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>160</th>\n",
       "      <td>Lowe's Cos.</td>\n",
       "      <td>LOW</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>170</th>\n",
       "      <td>Michael Kors Holdings</td>\n",
       "      <td>CPRI</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>222</th>\n",
       "      <td>SCANA Corp</td>\n",
       "      <td>SCG</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>235</th>\n",
       "      <td>Symantec Corp.</td>\n",
       "      <td>GEN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>264</th>\n",
       "      <td>Viacom Inc.</td>\n",
       "      <td>VIACOM</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                            Company  Ticker\n",
       "110  General Growth Properties Inc.     GGP\n",
       "120              Harris Corporation     LHX\n",
       "135              Ingersoll-Rand PLC      IR\n",
       "160                     Lowe's Cos.     LOW\n",
       "170           Michael Kors Holdings    CPRI\n",
       "222                      SCANA Corp     SCG\n",
       "235                  Symantec Corp.     GEN\n",
       "264                     Viacom Inc.  VIACOM"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# COMPANY matches between the 2 data sets\n",
    "# this is good because the tickers are different sometimes (mergers/name changes)\n",
    "print(KeFVP_ticker_Company[~KeFVP_ticker_Company['Company'].isin(company_ticker.Company.unique())])\n",
    "company_ticker[~company_ticker['Company'].isin(KeFVP_ticker_Company.Company.unique())]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bc663d1f-3ead-4e9f-b9a3-d96522e03012",
   "metadata": {},
   "source": [
    "# Clean Praat features\n",
    "\n",
    "- Replace '--undefined--' data\n",
    "- Convert to numeric float64\n",
    "- Compare our Praat data (MAEC), which was calculated from MP3 files.\n",
    "    - To the provided Praat data (MAEC) [GitHub](https://github.com/Earnings-Call-Dataset/MAEC-A-Multimodal-Aligned-Earnings-Conference-Call-Dataset-for-Financial-Risk-Prediction/tree/master)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "26a5c76d-47ab-437b-8e0f-7068d5d68856",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 30/30 [00:00<00:00, 71.89it/s]\n",
      "100%|██████████| 30/30 [00:00<00:00, 550.95it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 89722 entries, 0 to 89721\n",
      "Data columns (total 33 columns):\n",
      " #   Column                        Non-Null Count  Dtype  \n",
      "---  ------                        --------------  -----  \n",
      " 0   Mean pitch                    89722 non-null  float64\n",
      " 1   Standard deviation            89722 non-null  float64\n",
      " 2   Minimum pitch                 89722 non-null  float64\n",
      " 3   Maximum pitch                 89722 non-null  float64\n",
      " 4   Number of pulses              89722 non-null  float64\n",
      " 5   Number of periods             89722 non-null  float64\n",
      " 6   Mean period                   89722 non-null  float64\n",
      " 7   Mean intensity                89722 non-null  float64\n",
      " 8   Minimum intensity             89722 non-null  float64\n",
      " 9   Maximum intensity             89722 non-null  float64\n",
      " 10  Standard deviation of period  89722 non-null  float64\n",
      " 11  Fraction of unvoiced          89722 non-null  float64\n",
      " 12  Number of voice breaks        89722 non-null  float64\n",
      " 13  Degree of voice breaks        89722 non-null  float64\n",
      " 14  Jitter local                  89722 non-null  float64\n",
      " 15  Jitter local absolute         89722 non-null  float64\n",
      " 16  Jitter rap                    89722 non-null  float64\n",
      " 17  Jitter ppq5                   89722 non-null  float64\n",
      " 18  Jitter ddp                    89722 non-null  float64\n",
      " 19  Shimmer local                 89722 non-null  float64\n",
      " 20  Shimmer local dB              89722 non-null  float64\n",
      " 21  Shimmer apq3                  89722 non-null  float64\n",
      " 22  Shimmer apq5                  89722 non-null  float64\n",
      " 23  Shimmer apq11                 89722 non-null  float64\n",
      " 24  Shimmer dda                   89722 non-null  float64\n",
      " 25  Mean autocorrelation          89722 non-null  float64\n",
      " 26  Mean NHR                      89722 non-null  float64\n",
      " 27  Mean HNR                      89722 non-null  float64\n",
      " 28  Audio Length                  89722 non-null  float64\n",
      " 29  Company                       89722 non-null  object \n",
      " 30  Date                          89722 non-null  int64  \n",
      " 31  Sentence_num                  89722 non-null  int64  \n",
      " 32  audio_file                    89722 non-null  object \n",
      "dtypes: float64(29), int64(2), object(2)\n",
      "memory usage: 22.6+ MB\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "praat_features = praat_features.replace(['--undefined--', '--undefined-', '--undefined-- ',  '--'], np.nan)\n",
    "\n",
    "# Convert all columns into float64 except \n",
    "cols_to_convert = praat_features.columns.difference(['Company', 'Date', 'audio_file'])\n",
    "praat_features[cols_to_convert] = praat_features[cols_to_convert].progress_apply(pd.to_numeric)\n",
    "# impute median on all NULL (--undefined--)\n",
    "praat_features[cols_to_convert] = praat_features[cols_to_convert].progress_apply(\n",
    "    lambda col: col.fillna(col.median()) if pd.api.types.is_numeric_dtype(col) else col\n",
    ")\n",
    "praat_features.info(verbose=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "647bdc27-138f-47ab-aafb-6cf12a52b995",
   "metadata": {},
   "source": [
    "# Final datasets\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "92a50004",
   "metadata": {},
   "outputs": [],
   "source": [
    "max_sent_len = 523\n",
    "\n",
    "# The maximum sentences per meeting is 523 \n",
    "# Ensure each meeting has 523 sentences, add zeros to the end\n",
    "def add_zero_padding(group):\n",
    "    complete_index = pd.Index(np.arange(1, (max_sent_len +1)), name='Sentence_num')\n",
    "    group = group.set_index('Sentence_num').reindex(complete_index).reset_index()\n",
    "    group['Date'] = group['Date'].ffill().bfill()\n",
    "    group['Company'] = group['Company'].ffill().bfill()\n",
    "    group.fillna(0.0, inplace=True)\n",
    "    return group\n",
    "\n",
    "def combine_audio_text_targets(audio_features, text_features, num_features, save_directory, \n",
    "                               train_n_days_filename, train_single_day_filename, val_n_days_filename, \n",
    "                               val_single_day_filename, test_n_days_filename, test_single_day_filename ):\n",
    "    \n",
    "    features = pd.merge(audio_features, text_features, how=\"left\",on = ['Company','Date','Sentence_num'])\n",
    "    ##################################\n",
    "    # add emotion2vec\n",
    "    features = pd.merge(features, emotion2vec, how=\"left\",on = ['Company','Date','Sentence_num'])\n",
    "    num_features += 9\n",
    "    ##################################\n",
    "    features = features.drop(['Shimmer apq11','Audio Length','audio_file'], axis=1)\n",
    "    features = features.groupby(['Company', 'Date'], group_keys=False).progress_apply(add_zero_padding).reset_index(drop=True)\n",
    "    features.fillna(0, inplace=True)\n",
    "\n",
    "    # match the targets to the features\n",
    "    # (target values will be duplicated 523 times, for each meeting, but that's okay for now)\n",
    "    # we need the date and company columns to sort\n",
    "    train_targets = get_KeFVP(train_n_days_filename, train_single_day_filename)\n",
    "    val_targets = get_KeFVP(val_n_days_filename, val_single_day_filename)\n",
    "    test_targets = get_KeFVP(test_n_days_filename, test_single_day_filename)\n",
    "\n",
    "    train_targets['Date'] = train_targets['Date'].astype(str).str.replace('-', '').astype('Int64')\n",
    "    val_targets['Date'] = val_targets['Date'].astype(str).str.replace('-', '').astype('Int64')\n",
    "    test_targets['Date'] = test_targets['Date'].astype(str).str.replace('-', '').astype('Int64')\n",
    "\n",
    "    train_dataset = pd.merge(train_targets, features, how=\"left\",on = ['Company','Date'])\n",
    "    val_dataset = pd.merge(val_targets, features, how=\"left\",on = ['Company','Date'])\n",
    "    test_dataset = pd.merge(test_targets, features, how=\"left\",on = ['Company','Date'])\n",
    "\n",
    "    train_dataset.fillna(0, inplace=True)\n",
    "    val_dataset.fillna(0, inplace=True)\n",
    "    test_dataset.fillna(0, inplace=True)\n",
    "\n",
    "    train_dataset = train_dataset.sort_values(by=['Date', 'Company', 'Sentence_num'], ascending=[True, True, True])\n",
    "    val_dataset = val_dataset.sort_values(by=['Date', 'Company', 'Sentence_num'], ascending=[True, True, True])\n",
    "    test_dataset = test_dataset.sort_values(by=['Date', 'Company', 'Sentence_num'], ascending=[True, True, True])\n",
    "\n",
    "    train_dataset['Date'] = pd.to_datetime(train_dataset['Date'].astype(str), format='%Y%m%d')\n",
    "    val_dataset['Date'] = pd.to_datetime(val_dataset['Date'].astype(str), format='%Y%m%d')\n",
    "    test_dataset['Date'] = pd.to_datetime(test_dataset['Date'].astype(str), format='%Y%m%d')\n",
    "    \n",
    "    train_features = train_dataset.drop(['Ticker','Company','Date','Sentence_num','3_day','3_day_single','7_day',\n",
    "                                         '7_day_single', '15_day','15_day_single','30_day','30_day_single'], axis=1).to_numpy()\n",
    "    val_features = val_dataset.drop(['Ticker','Company','Date','Sentence_num','3_day','3_day_single','7_day',\n",
    "                                      '7_day_single', '15_day','15_day_single','30_day','30_day_single'], axis=1).to_numpy()\n",
    "    test_features = test_dataset.drop(['Ticker','Company','Date','Sentence_num','3_day','3_day_single','7_day',\n",
    "                                       '7_day_single', '15_day','15_day_single','30_day','30_day_single'], axis=1).to_numpy()\n",
    "    \n",
    "    train_targets = train_dataset[['3_day','3_day_single','7_day','7_day_single', '15_day','15_day_single','30_day','30_day_single']].copy()\n",
    "    val_targets = val_dataset[['3_day','3_day_single','7_day','7_day_single', '15_day','15_day_single','30_day','30_day_single']].copy()\n",
    "    test_targets = test_dataset[['3_day','3_day_single','7_day','7_day_single', '15_day','15_day_single','30_day','30_day_single']].copy()\n",
    "\n",
    "    train_targets.drop_duplicates(inplace=True)\n",
    "    val_targets.drop_duplicates(inplace=True)\n",
    "    test_targets.drop_duplicates(inplace=True)\n",
    "\n",
    "    #############################  save  ######################################################################\n",
    "    if not os.path.exists(f'data/{save_directory}'):\n",
    "        os.makedirs(f'data/{save_directory}')\n",
    "\n",
    "    # FEATURES- Reshape the NumPy array to have dimensions ( # meetings, 523 sentences (with padding), num_features)\n",
    "    np.save(f'data/{save_directory}/train_features.npy', train_features.reshape(int(len(train_features)/max_sent_len), max_sent_len, num_features))\n",
    "    np.save(f'data/{save_directory}/val_features.npy', val_features.reshape(int(len(val_features)/max_sent_len), max_sent_len, num_features))\n",
    "    np.save(f'data/{save_directory}/test_features.npy', test_features.reshape(int(len(test_features)/max_sent_len), max_sent_len, num_features))\n",
    "    \n",
    "    np.save(f'data/{save_directory}/train_targets_3.npy', train_targets['3_day'].to_numpy())\n",
    "    np.save(f'data/{save_directory}/val_targets_3.npy', val_targets['3_day'].to_numpy())\n",
    "    np.save(f'data/{save_directory}/test_targets_3.npy', test_targets['3_day'].to_numpy())\n",
    "    np.save(f'data/{save_directory}/train_secondary_targets_3.npy', train_targets['3_day_single'].to_numpy())\n",
    "    np.save(f'data/{save_directory}/val_secondary_targets_3.npy', val_targets['3_day_single'].to_numpy())\n",
    "    np.save(f'data/{save_directory}/test_secondary_targets_3.npy', test_targets['3_day_single'].to_numpy())\n",
    "\n",
    "    np.save(f'data/{save_directory}/train_targets_7.npy', train_targets['7_day'].to_numpy())\n",
    "    np.save(f'data/{save_directory}/val_targets_7.npy', val_targets['7_day'].to_numpy())\n",
    "    np.save(f'data/{save_directory}/test_targets_7.npy', test_targets['7_day'].to_numpy())\n",
    "    np.save(f'data/{save_directory}/train_secondary_targets_7.npy', train_targets['7_day_single'].to_numpy())\n",
    "    np.save(f'data/{save_directory}/val_secondary_targets_7.npy', val_targets['7_day_single'].to_numpy())\n",
    "    np.save(f'data/{save_directory}/test_secondary_targets_7.npy', test_targets['7_day_single'].to_numpy())\n",
    "\n",
    "    np.save(f'data/{save_directory}/train_targets_15.npy', train_targets['15_day'].to_numpy())\n",
    "    np.save(f'data/{save_directory}/val_targets_15.npy', val_targets['15_day'].to_numpy())\n",
    "    np.save(f'data/{save_directory}/test_targets_15.npy', test_targets['15_day'].to_numpy())\n",
    "    np.save(f'data/{save_directory}/train_secondary_targets_15.npy', train_targets['15_day_single'].to_numpy())\n",
    "    np.save(f'data/{save_directory}/val_secondary_targets_15.npy', val_targets['15_day_single'].to_numpy())\n",
    "    np.save(f'data/{save_directory}/test_secondary_targets_15.npy', test_targets['15_day_single'].to_numpy())\n",
    "\n",
    "    np.save(f'data/{save_directory}/train_targets_30.npy', train_targets['30_day'].to_numpy())\n",
    "    np.save(f'data/{save_directory}/val_targets_30.npy', val_targets['30_day'].to_numpy())\n",
    "    np.save(f'data/{save_directory}/test_targets_30.npy', test_targets['30_day'].to_numpy())\n",
    "    np.save(f'data/{save_directory}/train_secondary_targets_30.npy', train_targets['30_day_single'].to_numpy())\n",
    "    np.save(f'data/{save_directory}/val_secondary_targets_30.npy', val_targets['30_day_single'].to_numpy())\n",
    "    np.save(f'data/{save_directory}/test_secondary_targets_30.npy', test_targets['30_day_single'].to_numpy())\n",
    "        "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "c3a22103",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 572/572 [00:05<00:00, 101.62it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "should all be the same,-  391 391 391\n",
      "should all be the same,-  56 56 56\n",
      "should all be the same,-  111 112 111\n"
     ]
    }
   ],
   "source": [
    "praat_features['Date'] = praat_features['Date'].astype('Int64')\n",
    "\n",
    "# these will be different for the MAEC data\n",
    "train_n_days_filename = 'train_split_Avg_Series_WITH_LOG'\n",
    "train_single_day_filename = 'train_split_SeriesSingleDayVol3'\n",
    "val_n_days_filename = 'val_split_Avg_Series_WITH_LOG'\n",
    "val_single_day_filename = 'val_split_SeriesSingleDayVol3'\n",
    "test_n_days_filename = 'test_split_Avg_Series_WITH_LOG'\n",
    "test_single_day_filename = 'test_split_SeriesSingleDayVol3'\n",
    "\n",
    "# add emotion2vec to directory name\n",
    "combine_audio_text_targets(praat_features, RoBERTa_features, 1051, 'KeFVP_emotion2vec_RoBERTa',\n",
    "                           train_n_days_filename, train_single_day_filename, val_n_days_filename,\n",
    "                           val_single_day_filename, test_n_days_filename, test_single_day_filename \n",
    ")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "32620af4",
   "metadata": {},
   "outputs": [],
   "source": [
    "# grouped_counts = test_features.groupby(['Date', 'Company']).size().reset_index(name='Occurrences')\n",
    "\n",
    "# min_occurrences = grouped_counts[grouped_counts['Occurrences'] <500]\n",
    "\n",
    "# min_occurrences\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "9db3c2de",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 572/572 [00:05<00:00, 102.20it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "should all be the same,-  391 391 391\n",
      "should all be the same,-  56 56 56\n",
      "should all be the same,-  111 112 111\n"
     ]
    }
   ],
   "source": [
    "combine_audio_text_targets(praat_features, RoBERTa_features2, 1051, 'KeFVP_emotion2vec_RoBERTa2',\n",
    "                           train_n_days_filename, train_single_day_filename, val_n_days_filename,\n",
    "                           val_single_day_filename, test_n_days_filename, test_single_day_filename \n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "08656b30",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 572/572 [00:04<00:00, 127.87it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "should all be the same,-  391 391 391\n",
      "should all be the same,-  56 56 56\n",
      "should all be the same,-  111 112 111\n"
     ]
    }
   ],
   "source": [
    "combine_audio_text_targets(praat_features, investopedia_features, 795, 'KeFVP_emotion2vec_investopedia',\n",
    "                           train_n_days_filename, train_single_day_filename, val_n_days_filename,\n",
    "                           val_single_day_filename, test_n_days_filename, test_single_day_filename \n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "a6cd6232",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 572/572 [00:05<00:00, 101.24it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "should all be the same,-  391 391 391\n",
      "should all be the same,-  56 56 56\n",
      "should all be the same,-  111 112 111\n"
     ]
    }
   ],
   "source": [
    "combine_audio_text_targets(praat_features, bge_features, 1051, 'KeFVP_emotion2vec_bge',\n",
    "                           train_n_days_filename, train_single_day_filename, val_n_days_filename,\n",
    "                           val_single_day_filename, test_n_days_filename, test_single_day_filename \n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "1db4955a",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 572/572 [00:04<00:00, 137.12it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "should all be the same,-  391 391 391\n",
      "should all be the same,-  56 56 56\n",
      "should all be the same,-  111 112 111\n"
     ]
    }
   ],
   "source": [
    "combine_audio_text_targets(praat_features, bge_base_features, 795, 'KeFVP_emotion2vec_bge_base',\n",
    "                           train_n_days_filename, train_single_day_filename, val_n_days_filename,\n",
    "                           val_single_day_filename, test_n_days_filename, test_single_day_filename \n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "e3e95595",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 572/572 [00:02<00:00, 270.10it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "should all be the same,-  391 391 391\n",
      "should all be the same,-  56 56 56\n",
      "should all be the same,-  111 112 111\n"
     ]
    }
   ],
   "source": [
    "combine_audio_text_targets(praat_features, glove_features, 327, 'KeFVP_emotion2vec_glove',\n",
    "                           train_n_days_filename, train_single_day_filename, val_n_days_filename,\n",
    "                           val_single_day_filename, test_n_days_filename, test_single_day_filename \n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7402b950-2a10-45b5-bc00-2b72f70603f9",
   "metadata": {},
   "source": [
    "# Final MAEC dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "10052c5c",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 30/30 [00:01<00:00, 21.56it/s]\n",
      "100%|██████████| 30/30 [00:00<00:00, 215.53it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 394277 entries, 0 to 394276\n",
      "Data columns (total 33 columns):\n",
      " #   Column                        Non-Null Count   Dtype  \n",
      "---  ------                        --------------   -----  \n",
      " 0   Mean pitch                    394277 non-null  float64\n",
      " 1   Standard deviation            394277 non-null  float64\n",
      " 2   Minimum pitch                 394277 non-null  float64\n",
      " 3   Maximum pitch                 394277 non-null  float64\n",
      " 4   Number of pulses              394277 non-null  int64  \n",
      " 5   Number of periods             394277 non-null  int64  \n",
      " 6   Mean period                   394277 non-null  float64\n",
      " 7   Mean intensity                394277 non-null  float64\n",
      " 8   Minimum intensity             394277 non-null  float64\n",
      " 9   Maximum intensity             394277 non-null  float64\n",
      " 10  Standard deviation of period  394277 non-null  float64\n",
      " 11  Fraction of unvoiced          394277 non-null  float64\n",
      " 12  Number of voice breaks        394277 non-null  int64  \n",
      " 13  Degree of voice breaks        394277 non-null  float64\n",
      " 14  Jitter local                  394277 non-null  float64\n",
      " 15  Jitter local absolute         394277 non-null  float64\n",
      " 16  Jitter rap                    394277 non-null  float64\n",
      " 17  Jitter ppq5                   394277 non-null  float64\n",
      " 18  Jitter ddp                    394277 non-null  float64\n",
      " 19  Shimmer local                 394277 non-null  float64\n",
      " 20  Shimmer local dB              394277 non-null  float64\n",
      " 21  Shimmer apq3                  394277 non-null  float64\n",
      " 22  Shimmer apq5                  394277 non-null  float64\n",
      " 23  Shimmer apq11                 394277 non-null  float64\n",
      " 24  Shimmer dda                   394277 non-null  float64\n",
      " 25  Mean autocorrelation          394277 non-null  float64\n",
      " 26  Mean NHR                      394277 non-null  float64\n",
      " 27  Mean HNR                      394277 non-null  float64\n",
      " 28  Audio Length                  394277 non-null  float64\n",
      " 29  Ticker                        394277 non-null  object \n",
      " 30  Date                          394277 non-null  int64  \n",
      " 31  Sentence_num                  394277 non-null  int64  \n",
      " 32  audio_file                    394277 non-null  object \n",
      "dtypes: float64(26), int64(5), object(2)\n",
      "memory usage: 99.3+ MB\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "MAEC_praat_features = MAEC_praat_features.replace(['--undefined--', '--undefined-', '--undefined-- ',  '--'], np.nan)\n",
    "\n",
    "# Convert all columns into float64 except \n",
    "cols_to_convert = MAEC_praat_features.columns.difference(['Ticker', 'Date', 'audio_file'])\n",
    "MAEC_praat_features[cols_to_convert] = MAEC_praat_features[cols_to_convert].progress_apply(pd.to_numeric)\n",
    "# impute median on all NULL (--undefined--)\n",
    "MAEC_praat_features[cols_to_convert] = MAEC_praat_features[cols_to_convert].progress_apply(\n",
    "    lambda col: col.fillna(col.median()) if pd.api.types.is_numeric_dtype(col) else col\n",
    ")\n",
    "MAEC_praat_features.info(verbose=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "1b0a8225",
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "def get_KeFVP(n_days_filename, single_day_filename):\n",
    "    n_days_data = pd.read_csv(f'data/KeFVP_price_data/{n_days_filename}.csv')\n",
    "    n_days_data['Date'] = pd.to_datetime(n_days_data['time'], format='%Y-%m-%d')\n",
    "    n_days_data.drop_duplicates(subset=['Date', 'ticker'], inplace=True)\n",
    "    # drop this row specifically, is no transcript/ udio for this\n",
    "    # n_days_data = n_days_data[~((n_days_data['Date'] == '2017-10-31') & \n",
    "    #                                     (n_days_data['name'] == 'American Tower Corp A'))]\n",
    "    # single day volatility\n",
    "    single_day_data = pd.read_csv(f'data/KeFVP_price_data/{single_day_filename}.csv')\n",
    "    single_day_data['Date'] = pd.to_datetime(single_day_data['time'], format='%Y-%m-%d')\n",
    "    single_day_data.drop_duplicates(subset=['Date', 'ticker'], inplace=True)\n",
    "    single_day_data = single_day_data[['future_Single_3', 'future_Single_7', 'future_Single_15', \n",
    "                                       'future_Single_30', 'Date', 'ticker']].copy()# , 'name'\n",
    "    # merge\n",
    "    target_data = n_days_data[['future_3', 'future_7', 'future_15', 'future_30', 'Date', 'ticker']].copy()# , 'name'\n",
    "    target_data = pd.merge(target_data, single_day_data, how=\"left\",on = ['Date','ticker'])\n",
    "    target_data = target_data.rename(columns={'ticker':'Ticker', \n",
    "                                            'future_3':'3_day',\n",
    "                                            'future_7':'7_day',\n",
    "                                            'future_15':'15_day',\n",
    "                                            'future_30':'30_day',\n",
    "                                            'future_Single_3':'3_day_single',\n",
    "                                            'future_Single_7':'7_day_single',\n",
    "                                            'future_Single_15':'15_day_single',\n",
    "                                            'future_Single_30':'30_day_single'})\n",
    "    print('should all be the same,- ', len(n_days_data), len(single_day_data), len(target_data))\n",
    "    return target_data\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "a931b8df",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "should all be the same,-  535 535 535\n",
      "should all be the same,-  76 76 76\n",
      "should all be the same,-  154 154 154\n",
      "765\n"
     ]
    }
   ],
   "source": [
    "# these will be different for 2016\n",
    "train_n_days_filename = 'maec/15/maec15_train_avg_val'\n",
    "train_single_day_filename = 'maec/15/maec15_train_single_val'\n",
    "val_n_days_filename = 'maec/15/maec15_dev_avg_val'\n",
    "val_single_day_filename = 'maec/15/maec15_dev_single_val'\n",
    "test_n_days_filename = 'maec/15/maec15_test_avg_val'\n",
    "test_single_day_filename = 'maec/15/maec15_test_single_val'\n",
    "\n",
    "train_targets = get_KeFVP(train_n_days_filename, train_single_day_filename)\n",
    "val_targets = get_KeFVP(val_n_days_filename, val_single_day_filename)\n",
    "test_targets = get_KeFVP(test_n_days_filename, test_single_day_filename)\n",
    "\n",
    "# merge all  train/validate/test\n",
    "MAEC_dataset = pd.concat([train_targets, val_targets, test_targets], ignore_index=True)\n",
    "\n",
    "# grab all unique ticker/company\n",
    "KeFVP_ticker_Company = MAEC_dataset[['Ticker','Date']].copy()\n",
    "KeFVP_ticker_Company['Ticker'] = KeFVP_ticker_Company['Ticker'].str.upper()\n",
    "KeFVP_ticker_Company.drop_duplicates(inplace=True)\n",
    "print(len(KeFVP_ticker_Company))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "edfee6ca",
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "max_sent_len = 500 \n",
    "\n",
    "# The maximum sentences per meeting is  \n",
    "# Ensure each meeting has  sentences, add zeros to the end\n",
    "def add_zero_padding(group):\n",
    "    complete_index = pd.Index(np.arange(1, (max_sent_len + 1)), name='Sentence_num')\n",
    "    group = group.set_index('Sentence_num').reindex(complete_index).reset_index()\n",
    "    group['Date'] = group['Date'].ffill().bfill()\n",
    "    group['Ticker'] = group['Ticker'].ffill().bfill()\n",
    "    group.fillna(0.0, inplace=True)\n",
    "    return group\n",
    "\n",
    "def combine_audio_text_targets(audio_features, text_features, num_features, save_directory, \n",
    "                               train_n_days_filename, train_single_day_filename, val_n_days_filename, \n",
    "                               val_single_day_filename, test_n_days_filename, test_single_day_filename ):\n",
    "    # combine & add_zero_padding\n",
    "    features = pd.merge(audio_features, text_features, how=\"left\",on = ['Ticker','Date','Sentence_num'])\n",
    "    ##################################\n",
    "    # add MAEC_emotion2vec\n",
    "    features = pd.merge(features, MAEC_emotion2vec, how=\"left\",on = ['Ticker','Date','Sentence_num'])\n",
    "    num_features += 9\n",
    "    ##################################\n",
    "    features = features.drop(['Shimmer apq11','Audio Length','audio_file'], axis=1)\n",
    "    features = features.groupby(['Ticker', 'Date'], group_keys=False).progress_apply(add_zero_padding).reset_index(drop=True)\n",
    "    features.fillna(0, inplace=True)\n",
    "\n",
    "    # match the targets to the features\n",
    "    # (target values will be duplicated 500 times, for each meeting, but that's okay for now)\n",
    "    # we need the date and company columns to sort\n",
    "    train_targets = get_KeFVP(train_n_days_filename, train_single_day_filename)\n",
    "    val_targets = get_KeFVP(val_n_days_filename, val_single_day_filename)\n",
    "    test_targets = get_KeFVP(test_n_days_filename, test_single_day_filename)\n",
    "\n",
    "    train_targets['Date'] = train_targets['Date'].astype(str).str.replace('-', '').astype('Int64')\n",
    "    val_targets['Date'] = val_targets['Date'].astype(str).str.replace('-', '').astype('Int64')\n",
    "    test_targets['Date'] = test_targets['Date'].astype(str).str.replace('-', '').astype('Int64')\n",
    "\n",
    "    train_dataset = pd.merge(train_targets, features, how=\"left\",on = ['Ticker','Date'])\n",
    "    val_dataset = pd.merge(val_targets, features, how=\"left\",on = ['Ticker','Date'])\n",
    "    test_dataset = pd.merge(test_targets, features, how=\"left\",on = ['Ticker','Date'])\n",
    "\n",
    "    train_dataset.fillna(0, inplace=True)\n",
    "    val_dataset.fillna(0, inplace=True)\n",
    "    test_dataset.fillna(0, inplace=True)\n",
    "\n",
    "    train_dataset = train_dataset.sort_values(by=['Date', 'Ticker', 'Sentence_num'], ascending=[True, True, True])\n",
    "    val_dataset = val_dataset.sort_values(by=['Date', 'Ticker', 'Sentence_num'], ascending=[True, True, True])\n",
    "    test_dataset = test_dataset.sort_values(by=['Date', 'Ticker', 'Sentence_num'], ascending=[True, True, True])\n",
    "\n",
    "    train_dataset['Date'] = pd.to_datetime(train_dataset['Date'].astype(str), format='%Y%m%d')\n",
    "    val_dataset['Date'] = pd.to_datetime(val_dataset['Date'].astype(str), format='%Y%m%d')\n",
    "    test_dataset['Date'] = pd.to_datetime(test_dataset['Date'].astype(str), format='%Y%m%d')\n",
    "\n",
    "    train_features = train_dataset.drop(['Ticker','Date','Sentence_num','3_day','3_day_single','7_day',\n",
    "                                         '7_day_single', '15_day','15_day_single','30_day','30_day_single'], axis=1).to_numpy()\n",
    "    val_features = test_dataset.drop(['Ticker','Date','Sentence_num','3_day','3_day_single','7_day',\n",
    "                                      '7_day_single', '15_day','15_day_single','30_day','30_day_single'], axis=1).to_numpy()\n",
    "    test_features = test_dataset.drop(['Ticker','Date','Sentence_num','3_day','3_day_single','7_day',\n",
    "                                       '7_day_single', '15_day','15_day_single','30_day','30_day_single'], axis=1).to_numpy()\n",
    "\n",
    "    # must include date and ticker, some targets are identical\n",
    "    train_targets = train_dataset[['Date', 'Ticker', '3_day','3_day_single','7_day','7_day_single', '15_day','15_day_single','30_day','30_day_single']].copy()\n",
    "    val_targets = test_dataset[['Date', 'Ticker', '3_day','3_day_single','7_day','7_day_single', '15_day','15_day_single','30_day','30_day_single']].copy()\n",
    "    test_targets = test_dataset[['Date', 'Ticker', '3_day','3_day_single','7_day','7_day_single', '15_day','15_day_single','30_day','30_day_single']].copy()\n",
    "\n",
    "    train_targets.drop_duplicates(inplace=True)\n",
    "    val_targets.drop_duplicates(inplace=True)\n",
    "    test_targets.drop_duplicates(inplace=True)\n",
    "\n",
    "    train_targets = train_targets.drop(['Ticker','Date'], axis=1)\n",
    "    val_targets = val_targets.drop(['Ticker','Date'], axis=1)\n",
    "    test_targets = test_targets.drop(['Ticker','Date'], axis=1)\n",
    "    \n",
    "    #############################  save  ######################################################################\n",
    "    if not os.path.exists(f'data/{save_directory}'):\n",
    "        os.makedirs(f'data/{save_directory}')\n",
    "\n",
    "    # FEATURES- Reshape the NumPy array to have dimensions ( # meetings, 523 sentences (with padding), num_features)\n",
    "    np.save(f'data/{save_directory}/train_features.npy', train_features.reshape(int(len(train_features)/max_sent_len), max_sent_len, num_features))\n",
    "    np.save(f'data/{save_directory}/val_features.npy', val_features.reshape(int(len(val_features)/max_sent_len), max_sent_len, num_features))\n",
    "    np.save(f'data/{save_directory}/test_features.npy', test_features.reshape(int(len(test_features)/max_sent_len), max_sent_len, num_features))\n",
    "    \n",
    "    np.save(f'data/{save_directory}/train_targets_3.npy', train_targets['3_day'].to_numpy())\n",
    "    np.save(f'data/{save_directory}/val_targets_3.npy', val_targets['3_day'].to_numpy())\n",
    "    np.save(f'data/{save_directory}/test_targets_3.npy', test_targets['3_day'].to_numpy())\n",
    "    np.save(f'data/{save_directory}/train_secondary_targets_3.npy', train_targets['3_day_single'].to_numpy())\n",
    "    np.save(f'data/{save_directory}/val_secondary_targets_3.npy', val_targets['3_day_single'].to_numpy())\n",
    "    np.save(f'data/{save_directory}/test_secondary_targets_3.npy', test_targets['3_day_single'].to_numpy())\n",
    "\n",
    "    np.save(f'data/{save_directory}/train_targets_7.npy', train_targets['7_day'].to_numpy())\n",
    "    np.save(f'data/{save_directory}/val_targets_7.npy', val_targets['7_day'].to_numpy())\n",
    "    np.save(f'data/{save_directory}/test_targets_7.npy', test_targets['7_day'].to_numpy())\n",
    "    np.save(f'data/{save_directory}/train_secondary_targets_7.npy', train_targets['7_day_single'].to_numpy())\n",
    "    np.save(f'data/{save_directory}/val_secondary_targets_7.npy', val_targets['7_day_single'].to_numpy())\n",
    "    np.save(f'data/{save_directory}/test_secondary_targets_7.npy', test_targets['7_day_single'].to_numpy())\n",
    "\n",
    "    np.save(f'data/{save_directory}/train_targets_15.npy', train_targets['15_day'].to_numpy())\n",
    "    np.save(f'data/{save_directory}/val_targets_15.npy', val_targets['15_day'].to_numpy())\n",
    "    np.save(f'data/{save_directory}/test_targets_15.npy', test_targets['15_day'].to_numpy())\n",
    "    np.save(f'data/{save_directory}/train_secondary_targets_15.npy', train_targets['15_day_single'].to_numpy())\n",
    "    np.save(f'data/{save_directory}/val_secondary_targets_15.npy', val_targets['15_day_single'].to_numpy())\n",
    "    np.save(f'data/{save_directory}/test_secondary_targets_15.npy', test_targets['15_day_single'].to_numpy())\n",
    "\n",
    "    np.save(f'data/{save_directory}/train_targets_30.npy', train_targets['30_day'].to_numpy())\n",
    "    np.save(f'data/{save_directory}/val_targets_30.npy', val_targets['30_day'].to_numpy())\n",
    "    np.save(f'data/{save_directory}/test_targets_30.npy', test_targets['30_day'].to_numpy())\n",
    "    np.save(f'data/{save_directory}/train_secondary_targets_30.npy', train_targets['30_day_single'].to_numpy())\n",
    "    np.save(f'data/{save_directory}/val_secondary_targets_30.npy', val_targets['30_day_single'].to_numpy())\n",
    "    np.save(f'data/{save_directory}/test_secondary_targets_30.npy', test_targets['30_day_single'].to_numpy())\n",
    "        \n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "d0631218",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 3443/3443 [00:38<00:00, 89.26it/s] \n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "should all be the same,-  535 535 535\n",
      "should all be the same,-  76 76 76\n",
      "should all be the same,-  154 154 154\n"
     ]
    }
   ],
   "source": [
    "MAEC_praat_features['Date'] = MAEC_praat_features['Date'].astype('Int64')\n",
    "\n",
    "combine_audio_text_targets(MAEC_praat_features, MAEC_RoBERTa_features, 1051, 'KeFVP_emotion2vec_MAEC15_RoBERTa',\n",
    "                           train_n_days_filename, train_single_day_filename, val_n_days_filename,\n",
    "                           val_single_day_filename, test_n_days_filename, test_single_day_filename \n",
    ")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "118fc586",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 3443/3443 [00:32<00:00, 106.35it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "should all be the same,-  535 535 535\n",
      "should all be the same,-  76 76 76\n",
      "should all be the same,-  154 154 154\n"
     ]
    }
   ],
   "source": [
    "combine_audio_text_targets(MAEC_praat_features, MAEC_RoBERTa_features2, 1051, 'KeFVP_emotion2vec_MAEC15_RoBERTa2',\n",
    "                           train_n_days_filename, train_single_day_filename, val_n_days_filename,\n",
    "                           val_single_day_filename, test_n_days_filename, test_single_day_filename \n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "999f7e99",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 3443/3443 [00:24<00:00, 139.50it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "should all be the same,-  535 535 535\n",
      "should all be the same,-  76 76 76\n",
      "should all be the same,-  154 154 154\n"
     ]
    }
   ],
   "source": [
    "combine_audio_text_targets(MAEC_praat_features, MAEC_investopedia_features, 795, 'KeFVP_emotion2vec_MAEC15_investopedia',\n",
    "                           train_n_days_filename, train_single_day_filename, val_n_days_filename,\n",
    "                           val_single_day_filename, test_n_days_filename, test_single_day_filename \n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "4e13bb8b",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 3443/3443 [00:32<00:00, 104.76it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "should all be the same,-  535 535 535\n",
      "should all be the same,-  76 76 76\n",
      "should all be the same,-  154 154 154\n"
     ]
    }
   ],
   "source": [
    "combine_audio_text_targets(MAEC_praat_features, MAEC_bge_features, 1051, 'KeFVP_emotion2vec_MAEC15_bge',\n",
    "                           train_n_days_filename, train_single_day_filename, val_n_days_filename,\n",
    "                           val_single_day_filename, test_n_days_filename, test_single_day_filename \n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "c8dad44b",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 3443/3443 [00:25<00:00, 137.64it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "should all be the same,-  535 535 535\n",
      "should all be the same,-  76 76 76\n",
      "should all be the same,-  154 154 154\n"
     ]
    }
   ],
   "source": [
    "combine_audio_text_targets(MAEC_praat_features, MAEC_bge_base_features, 795, 'KeFVP_emotion2vec_MAEC15_bge_base',\n",
    "                           train_n_days_filename, train_single_day_filename, val_n_days_filename,\n",
    "                           val_single_day_filename, test_n_days_filename, test_single_day_filename \n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "17714587",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 3443/3443 [00:13<00:00, 258.07it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "should all be the same,-  535 535 535\n",
      "should all be the same,-  76 76 76\n",
      "should all be the same,-  154 154 154\n"
     ]
    }
   ],
   "source": [
    "combine_audio_text_targets(MAEC_praat_features, MAEC_glove_features, 327, 'KeFVP_emotion2vec_MAEC15_glove',\n",
    "                           train_n_days_filename, train_single_day_filename, val_n_days_filename,\n",
    "                           val_single_day_filename, test_n_days_filename, test_single_day_filename \n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6cfc04ed",
   "metadata": {},
   "source": [
    "# 2016"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "a6369734",
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_KeFVP(n_days_filename, single_day_filename):\n",
    "    n_days_data = pd.read_csv(f'data/KeFVP_price_data/{n_days_filename}.csv')\n",
    "    ######################\n",
    "    n_days_data = n_days_data.loc[:, ~n_days_data.columns.str.contains('^Unnamed')]  # Drop unnamed columns\n",
    "    n_days_data.replace([float('inf'), float('-inf')], 0, inplace=True)  # Replace infinity with 0\n",
    "    ######################\n",
    "    n_days_data['Date'] = pd.to_datetime(n_days_data['time'], format='%Y-%m-%d')\n",
    "    n_days_data.drop_duplicates(subset=['Date', 'ticker'], inplace=True)\n",
    "    # drop this row specifically, is no transcript/ udio for this\n",
    "    # n_days_data = n_days_data[~((n_days_data['Date'] == '2017-10-31') & \n",
    "    #                                     (n_days_data['name'] == 'American Tower Corp A'))]\n",
    "    # single day volatility\n",
    "    single_day_data = pd.read_csv(f'data/KeFVP_price_data/{single_day_filename}.csv')\n",
    "    ######################\n",
    "    single_day_data = single_day_data.loc[:, ~single_day_data.columns.str.contains('^Unnamed')]  # Drop unnamed columns\n",
    "    single_day_data.replace([float('inf'), float('-inf')], 0, inplace=True)  # Replace infinity with 0\n",
    "    ######################\n",
    "    single_day_data['Date'] = pd.to_datetime(single_day_data['time'], format='%Y-%m-%d')\n",
    "    single_day_data.drop_duplicates(subset=['Date', 'ticker'], inplace=True)\n",
    "    single_day_data = single_day_data[['future_Single_3', 'future_Single_7', 'future_Single_15', \n",
    "                                       'future_Single_30', 'Date', 'ticker']].copy()# , 'name'\n",
    "    # merge\n",
    "    target_data = n_days_data[['future_3', 'future_7', 'future_15', 'future_30', 'Date', 'ticker']].copy()# , 'name'\n",
    "    target_data = pd.merge(target_data, single_day_data, how=\"left\",on = ['Date','ticker'])\n",
    "    target_data = target_data.rename(columns={'ticker':'Ticker', \n",
    "                                            'future_3':'3_day',\n",
    "                                            'future_7':'7_day',\n",
    "                                            'future_15':'15_day',\n",
    "                                            'future_30':'30_day',\n",
    "                                            'future_Single_3':'3_day_single',\n",
    "                                            'future_Single_7':'7_day_single',\n",
    "                                            'future_Single_15':'15_day_single',\n",
    "                                            'future_Single_30':'30_day_single'})\n",
    "    print('should all be the same,- ', len(n_days_data), len(single_day_data), len(target_data))\n",
    "    return target_data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "cd912028-cb72-4bdf-b82e-09c7c604a164",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "should all be the same,-  980 980 980\n",
      "should all be the same,-  140 140 140\n",
      "should all be the same,-  280 280 280\n",
      "1400\n"
     ]
    }
   ],
   "source": [
    "\n",
    "train_n_days_filename = 'maec/16/maec16_train_avg_val'\n",
    "train_single_day_filename = 'maec/16/maec16_train_single_val'\n",
    "val_n_days_filename = 'maec/16/maec16_dev_avg_val'\n",
    "val_single_day_filename = 'maec/16/maec16_dev_single_val'\n",
    "test_n_days_filename = 'maec/16/maec16_test_avg_val'\n",
    "test_single_day_filename = 'maec/16/maec16_test_single_val'\n",
    "\n",
    "train_targets = get_KeFVP(train_n_days_filename, train_single_day_filename)\n",
    "val_targets = get_KeFVP(val_n_days_filename, val_single_day_filename)\n",
    "test_targets = get_KeFVP(test_n_days_filename, test_single_day_filename)\n",
    "\n",
    "# merge all  train/validate/test\n",
    "MAEC_dataset = pd.concat([train_targets, val_targets, test_targets], ignore_index=True)\n",
    "\n",
    "# grab all unique ticker/company\n",
    "KeFVP_ticker_Company = MAEC_dataset[['Ticker','Date']].copy()\n",
    "KeFVP_ticker_Company['Ticker'] = KeFVP_ticker_Company['Ticker'].str.upper()\n",
    "KeFVP_ticker_Company.drop_duplicates(inplace=True)\n",
    "print(len(KeFVP_ticker_Company))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "1583e99e",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 3443/3443 [00:32<00:00, 106.32it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "should all be the same,-  980 980 980\n",
      "should all be the same,-  140 140 140\n",
      "should all be the same,-  280 280 280\n"
     ]
    }
   ],
   "source": [
    "combine_audio_text_targets(MAEC_praat_features, MAEC_RoBERTa_features, 1051, 'KeFVP_emotion2vec_MAEC16_RoBERTa',\n",
    "                           train_n_days_filename, train_single_day_filename, val_n_days_filename,\n",
    "                           val_single_day_filename, test_n_days_filename, test_single_day_filename \n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "f4a8ea10",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 3443/3443 [00:32<00:00, 107.39it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "should all be the same,-  980 980 980\n",
      "should all be the same,-  140 140 140\n",
      "should all be the same,-  280 280 280\n"
     ]
    }
   ],
   "source": [
    "combine_audio_text_targets(MAEC_praat_features, MAEC_RoBERTa_features2, 1051, 'KeFVP_emotion2vec_MAEC16_RoBERTa2',\n",
    "                           train_n_days_filename, train_single_day_filename, val_n_days_filename,\n",
    "                           val_single_day_filename, test_n_days_filename, test_single_day_filename \n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "6b02791f",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 3443/3443 [00:25<00:00, 137.37it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "should all be the same,-  980 980 980\n",
      "should all be the same,-  140 140 140\n",
      "should all be the same,-  280 280 280\n"
     ]
    }
   ],
   "source": [
    "combine_audio_text_targets(MAEC_praat_features, MAEC_investopedia_features, 795, 'KeFVP_emotion2vec_MAEC16_investopedia',\n",
    "                           train_n_days_filename, train_single_day_filename, val_n_days_filename,\n",
    "                           val_single_day_filename, test_n_days_filename, test_single_day_filename \n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "c9ae8d55",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 3443/3443 [00:33<00:00, 104.30it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "should all be the same,-  980 980 980\n",
      "should all be the same,-  140 140 140\n",
      "should all be the same,-  280 280 280\n"
     ]
    }
   ],
   "source": [
    "combine_audio_text_targets(MAEC_praat_features, MAEC_bge_features, 1051, 'KeFVP_emotion2vec_MAEC16_bge',\n",
    "                           train_n_days_filename, train_single_day_filename, val_n_days_filename,\n",
    "                           val_single_day_filename, test_n_days_filename, test_single_day_filename \n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "95c2b6c7",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 3443/3443 [00:25<00:00, 134.07it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "should all be the same,-  980 980 980\n",
      "should all be the same,-  140 140 140\n",
      "should all be the same,-  280 280 280\n"
     ]
    }
   ],
   "source": [
    "combine_audio_text_targets(MAEC_praat_features, MAEC_bge_base_features, 795, 'KeFVP_emotion2vec_MAEC16_bge_base',\n",
    "                           train_n_days_filename, train_single_day_filename, val_n_days_filename,\n",
    "                           val_single_day_filename, test_n_days_filename, test_single_day_filename \n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "72d1bccc",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 3443/3443 [00:13<00:00, 261.88it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "should all be the same,-  980 980 980\n",
      "should all be the same,-  140 140 140\n",
      "should all be the same,-  280 280 280\n"
     ]
    }
   ],
   "source": [
    "combine_audio_text_targets(MAEC_praat_features, MAEC_glove_features, 327, 'KeFVP_emotion2vec_MAEC16_glove',\n",
    "                           train_n_days_filename, train_single_day_filename, val_n_days_filename,\n",
    "                           val_single_day_filename, test_n_days_filename, test_single_day_filename \n",
    ")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "transformers",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.15"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}