09. Tensorflow使用LR和GBDT和DNN实现银行营销二分类.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "rNdWfPXCjTjY"
   },
   "source": [
    "## Tensorflow2使用LR和GBDT和DNN实现二分类预估"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### CTR预估用于推荐系统的排序阶段：\n",
    "* 本身是一个二分类问题，收集的样本数据1代表点击是正样本、0代表未点击是负样本\n",
    "* 输入数据为用户user的信息、召回阶段产出的候选item信息列表\n",
    "* 输出为用户点积这些item的概率，按照概率排序取TOP N即为推荐结果\n",
    "\n",
    "#### CTR预估常用的排序算法\n",
    "* LR：逻辑回归，最简单的做法，常用于推荐系统初期，性能高、可解释\n",
    "* GBDT：集成树模型，用N棵树的集成结果进行预测，效果比LR好\n",
    "* DNN：深度学习，使用多层layer实现非线性变换，需要大量的数据集和调参技巧"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 演示流程：\n",
    "\n",
    "1. 用Pandas导入 CSV 文件。\n",
    "2. 用tf.dataset读取数据，进行分批（batch）、随机排序（shuffle）处理\n",
    "3. 用feature_column将 CSV 中的列映射到用于训练模型的特征\n",
    "4. 训练LR/GBDT/DNN对比AUC的效果"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "K1y4OHpGgss7"
   },
   "source": [
    "#### 银行营销数据集\n",
    "\n",
    "数据来自葡萄牙银行机构电话营销活动的记录。\n",
    "\n",
    "每行是同一个客户的电话沟通记录，以及最后一列是这客户最后明确购买还是不购买产品，是一个二分类问题。\n",
    "\n",
    "数据列如下：\n",
    "01. 年龄，数字\n",
    "02. 工作类型：分类，比如管理员、企业家\n",
    "03. 婚姻状况：分类，比如已婚、离婚、单身\n",
    "04. 教育：分类，小学、中学等\n",
    "05. 是否有信用：分类、是/否\n",
    "06. 余额：数字、年均余额\n",
    "07. 是否有住房贷款：分类、是/否\n",
    "08. 是否有贷款：分类，是/否\n",
    "09. 联系方式：分类、位置、手机号码、固定号码\n",
    "10. 天：数字、最后一次联系日\n",
    "11. 月：数字、最后一次联系的月份\n",
    "12. 持续时间：数字、上次联系时间\n",
    "13. 广告系列：数字，在这个广告和用户联系的次数，包括其他人\n",
    "14. pdays：数字，和这个客户上次联系的间隔天数\n",
    "15. 以前：数字，这个广告此前和此客户的联系次数\n",
    "16. 以前的结果：分类、成功、失败\n",
    "17. 最终是否订阅了定期存款：二分类、是、否"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "VxyBFc_kKazA"
   },
   "source": [
    "### 1. 导入库"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "9dEreb4QKizj"
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "import tensorflow as tf\n",
    "\n",
    "from tensorflow import feature_column\n",
    "from sklearn.model_selection import train_test_split"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "KCEhSZcULZ9n"
   },
   "source": [
    "### 2. 使用 Pandas读取CSV"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "REZ57BXCLdfG"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>age</th>\n",
       "      <th>job</th>\n",
       "      <th>marital</th>\n",
       "      <th>education</th>\n",
       "      <th>default</th>\n",
       "      <th>balance</th>\n",
       "      <th>housing</th>\n",
       "      <th>loan</th>\n",
       "      <th>contact</th>\n",
       "      <th>day</th>\n",
       "      <th>month</th>\n",
       "      <th>duration</th>\n",
       "      <th>campaign</th>\n",
       "      <th>pdays</th>\n",
       "      <th>previous</th>\n",
       "      <th>poutcome</th>\n",
       "      <th>y</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>58</td>\n",
       "      <td>management</td>\n",
       "      <td>married</td>\n",
       "      <td>tertiary</td>\n",
       "      <td>no</td>\n",
       "      <td>2143</td>\n",
       "      <td>yes</td>\n",
       "      <td>no</td>\n",
       "      <td>unknown</td>\n",
       "      <td>5</td>\n",
       "      <td>may</td>\n",
       "      <td>261</td>\n",
       "      <td>1</td>\n",
       "      <td>-1</td>\n",
       "      <td>0</td>\n",
       "      <td>unknown</td>\n",
       "      <td>no</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>44</td>\n",
       "      <td>technician</td>\n",
       "      <td>single</td>\n",
       "      <td>secondary</td>\n",
       "      <td>no</td>\n",
       "      <td>29</td>\n",
       "      <td>yes</td>\n",
       "      <td>no</td>\n",
       "      <td>unknown</td>\n",
       "      <td>5</td>\n",
       "      <td>may</td>\n",
       "      <td>151</td>\n",
       "      <td>1</td>\n",
       "      <td>-1</td>\n",
       "      <td>0</td>\n",
       "      <td>unknown</td>\n",
       "      <td>no</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>33</td>\n",
       "      <td>entrepreneur</td>\n",
       "      <td>married</td>\n",
       "      <td>secondary</td>\n",
       "      <td>no</td>\n",
       "      <td>2</td>\n",
       "      <td>yes</td>\n",
       "      <td>yes</td>\n",
       "      <td>unknown</td>\n",
       "      <td>5</td>\n",
       "      <td>may</td>\n",
       "      <td>76</td>\n",
       "      <td>1</td>\n",
       "      <td>-1</td>\n",
       "      <td>0</td>\n",
       "      <td>unknown</td>\n",
       "      <td>no</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>47</td>\n",
       "      <td>blue-collar</td>\n",
       "      <td>married</td>\n",
       "      <td>unknown</td>\n",
       "      <td>no</td>\n",
       "      <td>1506</td>\n",
       "      <td>yes</td>\n",
       "      <td>no</td>\n",
       "      <td>unknown</td>\n",
       "      <td>5</td>\n",
       "      <td>may</td>\n",
       "      <td>92</td>\n",
       "      <td>1</td>\n",
       "      <td>-1</td>\n",
       "      <td>0</td>\n",
       "      <td>unknown</td>\n",
       "      <td>no</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>33</td>\n",
       "      <td>unknown</td>\n",
       "      <td>single</td>\n",
       "      <td>unknown</td>\n",
       "      <td>no</td>\n",
       "      <td>1</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>unknown</td>\n",
       "      <td>5</td>\n",
       "      <td>may</td>\n",
       "      <td>198</td>\n",
       "      <td>1</td>\n",
       "      <td>-1</td>\n",
       "      <td>0</td>\n",
       "      <td>unknown</td>\n",
       "      <td>no</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   age           job  marital  education default  balance housing loan  \\\n",
       "0   58    management  married   tertiary      no     2143     yes   no   \n",
       "1   44    technician   single  secondary      no       29     yes   no   \n",
       "2   33  entrepreneur  married  secondary      no        2     yes  yes   \n",
       "3   47   blue-collar  married    unknown      no     1506     yes   no   \n",
       "4   33       unknown   single    unknown      no        1      no   no   \n",
       "\n",
       "   contact  day month  duration  campaign  pdays  previous poutcome   y  \n",
       "0  unknown    5   may       261         1     -1         0  unknown  no  \n",
       "1  unknown    5   may       151         1     -1         0  unknown  no  \n",
       "2  unknown    5   may        76         1     -1         0  unknown  no  \n",
       "3  unknown    5   may        92         1     -1         0  unknown  no  \n",
       "4  unknown    5   may       198         1     -1         0  unknown  no  "
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.read_csv(\"./datas/bank/bank-full.csv\", sep=\";\")\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(45211, 17)"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['no', 'yes'], dtype=object)"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df[\"y\"].unique()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>age</th>\n",
       "      <th>job</th>\n",
       "      <th>marital</th>\n",
       "      <th>education</th>\n",
       "      <th>default</th>\n",
       "      <th>balance</th>\n",
       "      <th>housing</th>\n",
       "      <th>loan</th>\n",
       "      <th>contact</th>\n",
       "      <th>day</th>\n",
       "      <th>month</th>\n",
       "      <th>duration</th>\n",
       "      <th>campaign</th>\n",
       "      <th>pdays</th>\n",
       "      <th>previous</th>\n",
       "      <th>poutcome</th>\n",
       "      <th>y</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>58</td>\n",
       "      <td>management</td>\n",
       "      <td>married</td>\n",
       "      <td>tertiary</td>\n",
       "      <td>no</td>\n",
       "      <td>2143</td>\n",
       "      <td>yes</td>\n",
       "      <td>no</td>\n",
       "      <td>unknown</td>\n",
       "      <td>5</td>\n",
       "      <td>may</td>\n",
       "      <td>261</td>\n",
       "      <td>1</td>\n",
       "      <td>-1</td>\n",
       "      <td>0</td>\n",
       "      <td>unknown</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>44</td>\n",
       "      <td>technician</td>\n",
       "      <td>single</td>\n",
       "      <td>secondary</td>\n",
       "      <td>no</td>\n",
       "      <td>29</td>\n",
       "      <td>yes</td>\n",
       "      <td>no</td>\n",
       "      <td>unknown</td>\n",
       "      <td>5</td>\n",
       "      <td>may</td>\n",
       "      <td>151</td>\n",
       "      <td>1</td>\n",
       "      <td>-1</td>\n",
       "      <td>0</td>\n",
       "      <td>unknown</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>33</td>\n",
       "      <td>entrepreneur</td>\n",
       "      <td>married</td>\n",
       "      <td>secondary</td>\n",
       "      <td>no</td>\n",
       "      <td>2</td>\n",
       "      <td>yes</td>\n",
       "      <td>yes</td>\n",
       "      <td>unknown</td>\n",
       "      <td>5</td>\n",
       "      <td>may</td>\n",
       "      <td>76</td>\n",
       "      <td>1</td>\n",
       "      <td>-1</td>\n",
       "      <td>0</td>\n",
       "      <td>unknown</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>47</td>\n",
       "      <td>blue-collar</td>\n",
       "      <td>married</td>\n",
       "      <td>unknown</td>\n",
       "      <td>no</td>\n",
       "      <td>1506</td>\n",
       "      <td>yes</td>\n",
       "      <td>no</td>\n",
       "      <td>unknown</td>\n",
       "      <td>5</td>\n",
       "      <td>may</td>\n",
       "      <td>92</td>\n",
       "      <td>1</td>\n",
       "      <td>-1</td>\n",
       "      <td>0</td>\n",
       "      <td>unknown</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>33</td>\n",
       "      <td>unknown</td>\n",
       "      <td>single</td>\n",
       "      <td>unknown</td>\n",
       "      <td>no</td>\n",
       "      <td>1</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>unknown</td>\n",
       "      <td>5</td>\n",
       "      <td>may</td>\n",
       "      <td>198</td>\n",
       "      <td>1</td>\n",
       "      <td>-1</td>\n",
       "      <td>0</td>\n",
       "      <td>unknown</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   age           job  marital  education default  balance housing loan  \\\n",
       "0   58    management  married   tertiary      no     2143     yes   no   \n",
       "1   44    technician   single  secondary      no       29     yes   no   \n",
       "2   33  entrepreneur  married  secondary      no        2     yes  yes   \n",
       "3   47   blue-collar  married    unknown      no     1506     yes   no   \n",
       "4   33       unknown   single    unknown      no        1      no   no   \n",
       "\n",
       "   contact  day month  duration  campaign  pdays  previous poutcome  y  \n",
       "0  unknown    5   may       261         1     -1         0  unknown  0  \n",
       "1  unknown    5   may       151         1     -1         0  unknown  0  \n",
       "2  unknown    5   may        76         1     -1         0  unknown  0  \n",
       "3  unknown    5   may        92         1     -1         0  unknown  0  \n",
       "4  unknown    5   may       198         1     -1         0  unknown  0  "
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.loc[df[\"y\"]==\"yes\", \"y\"] = 1\n",
    "df.loc[df[\"y\"]==\"no\", \"y\"] = 0\n",
    "df[\"y\"] = df[\"y\"].astype(int)\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "u0zhLtQqMPem"
   },
   "source": [
    "### 3. 将 dataframe 拆分为训练集和测试集"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "YEOpw7LhMYsI"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((36168, 16), (9043, 16), (36168,), (9043,))"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train, test = train_test_split(df, test_size=0.2)\n",
    "y_train = train.pop(\"y\")\n",
    "y_test = test.pop(\"y\")\n",
    "\n",
    "train.shape, test.shape, y_train.shape, y_test.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "84ef46LXMfvu"
   },
   "source": [
    "### 4. 用tf.data.Dataset读取数据"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "def make_input_fn(data_df, label_df, num_epochs=50, shuffle=True, batch_size=32):\n",
    "    def input_function():\n",
    "        ds = tf.data.Dataset.from_tensor_slices((dict(data_df), label_df))\n",
    "        if shuffle:\n",
    "            ds = ds.shuffle(1000)\n",
    "        ds = ds.batch(batch_size).repeat(num_epochs)\n",
    "        return ds\n",
    "    return input_function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_input_fn = make_input_fn(train, y_train)\n",
    "test_input_fn = make_input_fn(test, y_test, num_epochs=1, shuffle=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "ypkI9zx6Rj1q"
   },
   "source": [
    "### 5. feature_column特征处理\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 分类列名称列表\n",
    "category_names = [\"job\", \"marital\", \"education\", \"default\", \"housing\", \"loan\", \"contact\", \"month\", \"poutcome\"]\n",
    "\n",
    "# 数值列名称列表\n",
    "numeric_names = [\"balance\", \"duration\", \"campaign\", \"pdays\", \"previous\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'job': ['management', 'technician', 'entrepreneur', 'blue-collar', 'unknown', 'retired', 'admin.', 'services', 'self-employed', 'unemployed', 'housemaid', 'student'], 'marital': ['married', 'single', 'divorced'], 'education': ['tertiary', 'secondary', 'unknown', 'primary'], 'default': ['no', 'yes'], 'housing': ['yes', 'no'], 'loan': ['no', 'yes'], 'contact': ['unknown', 'cellular', 'telephone'], 'month': ['may', 'jun', 'jul', 'aug', 'oct', 'nov', 'dec', 'jan', 'feb', 'mar', 'apr', 'sep'], 'poutcome': ['unknown', 'failure', 'other', 'success']}\n"
     ]
    }
   ],
   "source": [
    "# 数据统计，分类列的枚举列表\n",
    "category_vocabulary_list = {}\n",
    "\n",
    "for feature_name in category_names:\n",
    "    category_vocabulary_list[feature_name] = list(df[feature_name].unique())\n",
    "\n",
    "print(category_vocabulary_list)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "4PlLY7fORuzA"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[IndicatorColumn(categorical_column=VocabularyListCategoricalColumn(key='job', vocabulary_list=('management', 'technician', 'entrepreneur', 'blue-collar', 'unknown', 'retired', 'admin.', 'services', 'self-employed', 'unemployed', 'housemaid', 'student'), dtype=tf.string, default_value=-1, num_oov_buckets=0)),\n",
       " IndicatorColumn(categorical_column=VocabularyListCategoricalColumn(key='marital', vocabulary_list=('married', 'single', 'divorced'), dtype=tf.string, default_value=-1, num_oov_buckets=0)),\n",
       " IndicatorColumn(categorical_column=VocabularyListCategoricalColumn(key='education', vocabulary_list=('tertiary', 'secondary', 'unknown', 'primary'), dtype=tf.string, default_value=-1, num_oov_buckets=0)),\n",
       " IndicatorColumn(categorical_column=VocabularyListCategoricalColumn(key='default', vocabulary_list=('no', 'yes'), dtype=tf.string, default_value=-1, num_oov_buckets=0)),\n",
       " IndicatorColumn(categorical_column=VocabularyListCategoricalColumn(key='housing', vocabulary_list=('yes', 'no'), dtype=tf.string, default_value=-1, num_oov_buckets=0)),\n",
       " IndicatorColumn(categorical_column=VocabularyListCategoricalColumn(key='loan', vocabulary_list=('no', 'yes'), dtype=tf.string, default_value=-1, num_oov_buckets=0)),\n",
       " IndicatorColumn(categorical_column=VocabularyListCategoricalColumn(key='contact', vocabulary_list=('unknown', 'cellular', 'telephone'), dtype=tf.string, default_value=-1, num_oov_buckets=0)),\n",
       " IndicatorColumn(categorical_column=VocabularyListCategoricalColumn(key='month', vocabulary_list=('may', 'jun', 'jul', 'aug', 'oct', 'nov', 'dec', 'jan', 'feb', 'mar', 'apr', 'sep'), dtype=tf.string, default_value=-1, num_oov_buckets=0)),\n",
       " IndicatorColumn(categorical_column=VocabularyListCategoricalColumn(key='poutcome', vocabulary_list=('unknown', 'failure', 'other', 'success'), dtype=tf.string, default_value=-1, num_oov_buckets=0)),\n",
       " NumericColumn(key='balance', shape=(1,), default_value=None, dtype=tf.float32, normalizer_fn=None),\n",
       " NumericColumn(key='duration', shape=(1,), default_value=None, dtype=tf.float32, normalizer_fn=None),\n",
       " NumericColumn(key='campaign', shape=(1,), default_value=None, dtype=tf.float32, normalizer_fn=None),\n",
       " NumericColumn(key='pdays', shape=(1,), default_value=None, dtype=tf.float32, normalizer_fn=None),\n",
       " NumericColumn(key='previous', shape=(1,), default_value=None, dtype=tf.float32, normalizer_fn=None)]"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "feature_columns = []\n",
    "\n",
    "for feature_name in category_names:\n",
    "    feature_columns.append(\n",
    "        feature_column.indicator_column(\n",
    "            feature_column.categorical_column_with_vocabulary_list(feature_name, category_vocabulary_list[feature_name])\n",
    "        )\n",
    "    )\n",
    "    \n",
    "for feature_name in numeric_names:\n",
    "    feature_columns.append(feature_column.numeric_column(feature_name))\n",
    "\n",
    "feature_columns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "bBx4Xu0eTXWq"
   },
   "source": [
    "### 6.1 使用LR训练和评估模型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_input_fn = make_input_fn(train, y_train)\n",
    "test_input_fn = make_input_fn(test, y_test, num_epochs=1, shuffle=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "_YJPPb3xTPeZ",
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Using default config.\n",
      "WARNING:tensorflow:Using temporary folder as model directory: /tmp/tmppa3dx8bs\n",
      "INFO:tensorflow:Using config: {'_model_dir': '/tmp/tmppa3dx8bs', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': allow_soft_placement: true\n",
      "graph_options {\n",
      "  rewrite_options {\n",
      "    meta_optimizer_iterations: ONE\n",
      "  }\n",
      "}\n",
      ", '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_service': None, '_cluster_spec': ClusterSpec({}), '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}\n",
      "WARNING:tensorflow:From /home/pss/anaconda3/lib/python3.7/site-packages/tensorflow/python/ops/resource_variable_ops.py:1666: calling BaseResourceVariable.__init__ (from tensorflow.python.ops.resource_variable_ops) with constraint is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "If using Keras pass *_constraint arguments to layers.\n",
      "WARNING:tensorflow:From /home/pss/anaconda3/lib/python3.7/site-packages/tensorflow/python/training/training_util.py:236: Variable.initialized_value (from tensorflow.python.ops.variables) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use Variable.read_value. Variables in 2.X are initialized automatically both in eager and graph (inside tf.defun) contexts.\n",
      "INFO:tensorflow:Calling model_fn.\n",
      "WARNING:tensorflow:From /home/pss/anaconda3/lib/python3.7/site-packages/tensorflow/python/feature_column/feature_column_v2.py:540: Layer.add_variable (from tensorflow.python.keras.engine.base_layer_v1) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Please use `layer.add_weight` method instead.\n",
      "WARNING:tensorflow:From /home/pss/anaconda3/lib/python3.7/site-packages/tensorflow/python/keras/optimizer_v2/ftrl.py:144: calling Constant.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Call initializer instance with the dtype argument instead of passing it to the constructor\n",
      "INFO:tensorflow:Done calling model_fn.\n",
      "INFO:tensorflow:Create CheckpointSaverHook.\n",
      "INFO:tensorflow:Graph was finalized.\n",
      "INFO:tensorflow:Running local_init_op.\n",
      "INFO:tensorflow:Done running local_init_op.\n",
      "INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 0...\n",
      "INFO:tensorflow:Saving checkpoints for 0 into /tmp/tmppa3dx8bs/model.ckpt.\n",
      "INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 0...\n",
      "INFO:tensorflow:loss = 0.6931472, step = 0\n",
      "INFO:tensorflow:global_step/sec: 270.537\n",
      "INFO:tensorflow:loss = 0.2017379, step = 100 (0.371 sec)\n",
      "INFO:tensorflow:global_step/sec: 574.369\n",
      "INFO:tensorflow:loss = 0.41973132, step = 200 (0.174 sec)\n",
      "INFO:tensorflow:global_step/sec: 656.657\n",
      "INFO:tensorflow:loss = 5.4152613, step = 300 (0.152 sec)\n",
      "INFO:tensorflow:global_step/sec: 647.443\n",
      "INFO:tensorflow:loss = 7.493124, step = 400 (0.155 sec)\n",
      "INFO:tensorflow:global_step/sec: 623.811\n",
      "INFO:tensorflow:loss = 0.020828264, step = 500 (0.160 sec)\n",
      "INFO:tensorflow:global_step/sec: 628.05\n",
      "INFO:tensorflow:loss = 0.05077633, step = 600 (0.160 sec)\n",
      "INFO:tensorflow:global_step/sec: 581.593\n",
      "INFO:tensorflow:loss = 0.5082642, step = 700 (0.172 sec)\n",
      "INFO:tensorflow:global_step/sec: 574.923\n",
      "INFO:tensorflow:loss = 0.53245765, step = 800 (0.173 sec)\n",
      "INFO:tensorflow:global_step/sec: 748.036\n",
      "INFO:tensorflow:loss = 2.1250887, step = 900 (0.135 sec)\n",
      "INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 1000...\n",
      "INFO:tensorflow:Saving checkpoints for 1000 into /tmp/tmppa3dx8bs/model.ckpt.\n",
      "INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 1000...\n",
      "INFO:tensorflow:Loss for final step: 0.047768787.\n",
      "INFO:tensorflow:Calling model_fn.\n",
      "INFO:tensorflow:Done calling model_fn.\n",
      "INFO:tensorflow:Starting evaluation at 2020-08-05T16:10:43Z\n",
      "INFO:tensorflow:Graph was finalized.\n",
      "INFO:tensorflow:Restoring parameters from /tmp/tmppa3dx8bs/model.ckpt-1000\n",
      "INFO:tensorflow:Running local_init_op.\n",
      "INFO:tensorflow:Done running local_init_op.\n",
      "INFO:tensorflow:Inference Time : 1.31960s\n",
      "INFO:tensorflow:Finished evaluation at 2020-08-05-16:10:44\n",
      "INFO:tensorflow:Saving dict for global step 1000: accuracy = 0.89815325, accuracy_baseline = 0.88809025, auc = 0.8428245, auc_precision_recall = 0.47382975, average_loss = 0.42300984, global_step = 1000, label/mean = 0.11190976, loss = 0.42291445, precision = 0.56565654, prediction/mean = 0.08622836, recall = 0.38735178\n",
      "INFO:tensorflow:Saving 'checkpoint_path' summary for global step 1000: /tmp/tmppa3dx8bs/model.ckpt-1000\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'accuracy': 0.89815325,\n",
       " 'accuracy_baseline': 0.88809025,\n",
       " 'auc': 0.8428245,\n",
       " 'auc_precision_recall': 0.47382975,\n",
       " 'average_loss': 0.42300984,\n",
       " 'label/mean': 0.11190976,\n",
       " 'loss': 0.42291445,\n",
       " 'precision': 0.56565654,\n",
       " 'prediction/mean': 0.08622836,\n",
       " 'recall': 0.38735178,\n",
       " 'global_step': 1000}"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "linear_est = tf.estimator.LinearClassifier(\n",
    "    feature_columns=feature_columns)\n",
    "\n",
    "linear_est.train(train_input_fn, max_steps=1000)\n",
    "result = linear_est.evaluate(test_input_fn)\n",
    "result"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "bBx4Xu0eTXWq"
   },
   "source": [
    "### 6.2 使用GBDT训练和评估模型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_input_fn = make_input_fn(train, y_train)\n",
    "test_input_fn = make_input_fn(test, y_test, num_epochs=1, shuffle=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "_YJPPb3xTPeZ",
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Using default config.\n",
      "WARNING:tensorflow:Using temporary folder as model directory: /tmp/tmppjeurnyu\n",
      "INFO:tensorflow:Using config: {'_model_dir': '/tmp/tmppjeurnyu', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': allow_soft_placement: true\n",
      "graph_options {\n",
      "  rewrite_options {\n",
      "    meta_optimizer_iterations: ONE\n",
      "  }\n",
      "}\n",
      ", '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_service': None, '_cluster_spec': ClusterSpec({}), '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}\n",
      "WARNING:tensorflow:From /home/pss/anaconda3/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/canned/boosted_trees.py:398: VocabularyListCategoricalColumn._num_buckets (from tensorflow.python.feature_column.feature_column_v2) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "The old _FeatureColumn APIs are being deprecated. Please use the new FeatureColumn APIs instead.\n",
      "INFO:tensorflow:Calling model_fn.\n",
      "INFO:tensorflow:Done calling model_fn.\n",
      "INFO:tensorflow:Create CheckpointSaverHook.\n",
      "WARNING:tensorflow:Issue encountered when serializing resources.\n",
      "Type is unsupported, or the types of the items don't match field type in CollectionDef. Note this is a warning and probably safe to ignore.\n",
      "'_Resource' object has no attribute 'name'\n",
      "INFO:tensorflow:Graph was finalized.\n",
      "INFO:tensorflow:Running local_init_op.\n",
      "INFO:tensorflow:Done running local_init_op.\n",
      "WARNING:tensorflow:Issue encountered when serializing resources.\n",
      "Type is unsupported, or the types of the items don't match field type in CollectionDef. Note this is a warning and probably safe to ignore.\n",
      "'_Resource' object has no attribute 'name'\n",
      "INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 0...\n",
      "INFO:tensorflow:Saving checkpoints for 0 into /tmp/tmppjeurnyu/model.ckpt.\n",
      "WARNING:tensorflow:Issue encountered when serializing resources.\n",
      "Type is unsupported, or the types of the items don't match field type in CollectionDef. Note this is a warning and probably safe to ignore.\n",
      "'_Resource' object has no attribute 'name'\n",
      "INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 0...\n",
      "INFO:tensorflow:loss = 0.6931472, step = 0\n",
      "WARNING:tensorflow:It seems that global step (tf.train.get_global_step) has not been increased. Current value (could be stable): 0 vs previous value: 0. You could increase the global step by passing tf.train.get_global_step() to Optimizer.apply_gradients or Optimizer.minimize.\n",
      "WARNING:tensorflow:It seems that global step (tf.train.get_global_step) has not been increased. Current value (could be stable): 0 vs previous value: 0. You could increase the global step by passing tf.train.get_global_step() to Optimizer.apply_gradients or Optimizer.minimize.\n",
      "WARNING:tensorflow:It seems that global step (tf.train.get_global_step) has not been increased. Current value (could be stable): 0 vs previous value: 0. You could increase the global step by passing tf.train.get_global_step() to Optimizer.apply_gradients or Optimizer.minimize.\n",
      "WARNING:tensorflow:It seems that global step (tf.train.get_global_step) has not been increased. Current value (could be stable): 0 vs previous value: 0. You could increase the global step by passing tf.train.get_global_step() to Optimizer.apply_gradients or Optimizer.minimize.\n",
      "WARNING:tensorflow:It seems that global step (tf.train.get_global_step) has not been increased. Current value (could be stable): 0 vs previous value: 0. You could increase the global step by passing tf.train.get_global_step() to Optimizer.apply_gradients or Optimizer.minimize.\n",
      "INFO:tensorflow:loss = 0.6931472, step = 0 (0.648 sec)\n",
      "INFO:tensorflow:global_step/sec: 90.6729\n",
      "INFO:tensorflow:loss = 0.6289942, step = 100 (0.459 sec)\n",
      "INFO:tensorflow:global_step/sec: 475.085\n",
      "INFO:tensorflow:loss = 0.56080306, step = 200 (0.211 sec)\n",
      "INFO:tensorflow:global_step/sec: 468.533\n",
      "INFO:tensorflow:loss = 0.516282, step = 300 (0.214 sec)\n",
      "INFO:tensorflow:global_step/sec: 468.899\n",
      "INFO:tensorflow:loss = 0.4749595, step = 400 (0.211 sec)\n",
      "INFO:tensorflow:global_step/sec: 493.921\n",
      "INFO:tensorflow:loss = 0.48430163, step = 500 (0.203 sec)\n",
      "INFO:tensorflow:global_step/sec: 438.592\n",
      "INFO:tensorflow:loss = 0.42261285, step = 600 (0.229 sec)\n",
      "INFO:tensorflow:global_step/sec: 471.244\n",
      "INFO:tensorflow:loss = 0.45923734, step = 700 (0.212 sec)\n",
      "INFO:tensorflow:global_step/sec: 503.138\n",
      "INFO:tensorflow:loss = 0.35785902, step = 800 (0.199 sec)\n",
      "INFO:tensorflow:global_step/sec: 519.062\n",
      "INFO:tensorflow:loss = 0.37546015, step = 900 (0.193 sec)\n",
      "INFO:tensorflow:global_step/sec: 129.691\n",
      "INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 1000...\n",
      "INFO:tensorflow:Saving checkpoints for 1000 into /tmp/tmppjeurnyu/model.ckpt.\n",
      "WARNING:tensorflow:Issue encountered when serializing resources.\n",
      "Type is unsupported, or the types of the items don't match field type in CollectionDef. Note this is a warning and probably safe to ignore.\n",
      "'_Resource' object has no attribute 'name'\n",
      "INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 1000...\n",
      "INFO:tensorflow:Loss for final step: 0.3721273.\n",
      "INFO:tensorflow:Calling model_fn.\n",
      "WARNING:tensorflow:From /home/pss/anaconda3/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/canned/head.py:642: auc (from tensorflow.python.ops.metrics_impl) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "The value of AUC returned by this may race with the update so this is deprecated. Please use tf.keras.metrics.AUC instead.\n",
      "WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to \"careful_interpolation\" instead.\n",
      "WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to \"careful_interpolation\" instead.\n",
      "INFO:tensorflow:Done calling model_fn.\n",
      "INFO:tensorflow:Starting evaluation at 2020-08-05T16:11:37Z\n",
      "INFO:tensorflow:Graph was finalized.\n",
      "INFO:tensorflow:Restoring parameters from /tmp/tmppjeurnyu/model.ckpt-1000\n",
      "INFO:tensorflow:Running local_init_op.\n",
      "INFO:tensorflow:Done running local_init_op.\n",
      "INFO:tensorflow:Inference Time : 1.73508s\n",
      "INFO:tensorflow:Finished evaluation at 2020-08-05-16:11:38\n",
      "INFO:tensorflow:Saving dict for global step 1000: accuracy = 0.90014374, accuracy_baseline = 0.88809025, auc = 0.8591649, auc_precision_recall = 0.48814416, average_loss = 0.3558857, global_step = 1000, label/mean = 0.11190976, loss = 0.35584077, precision = 0.62471396, prediction/mean = 0.253982, recall = 0.26976284\n",
      "WARNING:tensorflow:Issue encountered when serializing resources.\n",
      "Type is unsupported, or the types of the items don't match field type in CollectionDef. Note this is a warning and probably safe to ignore.\n",
      "'_Resource' object has no attribute 'name'\n",
      "INFO:tensorflow:Saving 'checkpoint_path' summary for global step 1000: /tmp/tmppjeurnyu/model.ckpt-1000\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'accuracy': 0.90014374,\n",
       " 'accuracy_baseline': 0.88809025,\n",
       " 'auc': 0.8591649,\n",
       " 'auc_precision_recall': 0.48814416,\n",
       " 'average_loss': 0.3558857,\n",
       " 'label/mean': 0.11190976,\n",
       " 'loss': 0.35584077,\n",
       " 'precision': 0.62471396,\n",
       " 'prediction/mean': 0.253982,\n",
       " 'recall': 0.26976284,\n",
       " 'global_step': 1000}"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gbdt = tf.estimator.BoostedTreesClassifier(\n",
    "    feature_columns=feature_columns, n_batches_per_layer=100)\n",
    "\n",
    "gbdt.train(train_input_fn, max_steps=1000)\n",
    "result = gbdt.evaluate(test_input_fn)\n",
    "result"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "bBx4Xu0eTXWq"
   },
   "source": [
    "### 6.3 使用DNN训练和评估模型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_input_fn = make_input_fn(train, y_train)\n",
    "test_input_fn = make_input_fn(test, y_test, num_epochs=1, shuffle=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "_YJPPb3xTPeZ",
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Using default config.\n",
      "WARNING:tensorflow:Using temporary folder as model directory: /tmp/tmpqpwcr_05\n",
      "INFO:tensorflow:Using config: {'_model_dir': '/tmp/tmpqpwcr_05', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': allow_soft_placement: true\n",
      "graph_options {\n",
      "  rewrite_options {\n",
      "    meta_optimizer_iterations: ONE\n",
      "  }\n",
      "}\n",
      ", '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_service': None, '_cluster_spec': ClusterSpec({}), '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}\n",
      "INFO:tensorflow:Calling model_fn.\n",
      "INFO:tensorflow:Done calling model_fn.\n",
      "INFO:tensorflow:Create CheckpointSaverHook.\n",
      "INFO:tensorflow:Graph was finalized.\n",
      "INFO:tensorflow:Running local_init_op.\n",
      "INFO:tensorflow:Done running local_init_op.\n",
      "INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 0...\n",
      "INFO:tensorflow:Saving checkpoints for 0 into /tmp/tmpqpwcr_05/model.ckpt.\n",
      "INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 0...\n",
      "INFO:tensorflow:loss = 6.940135, step = 0\n",
      "INFO:tensorflow:global_step/sec: 308.616\n",
      "INFO:tensorflow:loss = 0.85257244, step = 100 (0.325 sec)\n",
      "INFO:tensorflow:global_step/sec: 761.239\n",
      "INFO:tensorflow:loss = 0.6160759, step = 200 (0.131 sec)\n",
      "INFO:tensorflow:global_step/sec: 726.537\n",
      "INFO:tensorflow:loss = 0.6673007, step = 300 (0.137 sec)\n",
      "INFO:tensorflow:global_step/sec: 646.311\n",
      "INFO:tensorflow:loss = 0.5841639, step = 400 (0.155 sec)\n",
      "INFO:tensorflow:global_step/sec: 668.258\n",
      "INFO:tensorflow:loss = 0.8640286, step = 500 (0.150 sec)\n",
      "INFO:tensorflow:global_step/sec: 541.437\n",
      "INFO:tensorflow:loss = 0.58984846, step = 600 (0.185 sec)\n",
      "INFO:tensorflow:global_step/sec: 661.079\n",
      "INFO:tensorflow:loss = 0.49384272, step = 700 (0.151 sec)\n",
      "INFO:tensorflow:global_step/sec: 682.129\n",
      "INFO:tensorflow:loss = 0.48415846, step = 800 (0.147 sec)\n",
      "INFO:tensorflow:global_step/sec: 728.075\n",
      "INFO:tensorflow:loss = 0.5839883, step = 900 (0.140 sec)\n",
      "INFO:tensorflow:global_step/sec: 633.863\n",
      "INFO:tensorflow:loss = 0.5536402, step = 1000 (0.156 sec)\n",
      "INFO:tensorflow:global_step/sec: 809.31\n",
      "INFO:tensorflow:loss = 0.57881415, step = 1100 (0.125 sec)\n",
      "INFO:tensorflow:global_step/sec: 741.718\n",
      "INFO:tensorflow:loss = 0.57170314, step = 1200 (0.132 sec)\n",
      "INFO:tensorflow:global_step/sec: 832.876\n",
      "INFO:tensorflow:loss = 0.5218694, step = 1300 (0.120 sec)\n",
      "INFO:tensorflow:global_step/sec: 756.198\n",
      "INFO:tensorflow:loss = 0.34184575, step = 1400 (0.132 sec)\n",
      "INFO:tensorflow:global_step/sec: 623.019\n",
      "INFO:tensorflow:loss = 0.40819386, step = 1500 (0.162 sec)\n",
      "INFO:tensorflow:global_step/sec: 747.158\n",
      "INFO:tensorflow:loss = 1.1620795, step = 1600 (0.132 sec)\n",
      "INFO:tensorflow:global_step/sec: 664.967\n",
      "INFO:tensorflow:loss = 0.53267473, step = 1700 (0.152 sec)\n",
      "INFO:tensorflow:global_step/sec: 627.971\n",
      "INFO:tensorflow:loss = 0.39495078, step = 1800 (0.158 sec)\n",
      "INFO:tensorflow:global_step/sec: 766.481\n",
      "INFO:tensorflow:loss = 0.6052793, step = 1900 (0.131 sec)\n",
      "INFO:tensorflow:global_step/sec: 756.63\n",
      "INFO:tensorflow:loss = 0.77632153, step = 2000 (0.133 sec)\n",
      "INFO:tensorflow:global_step/sec: 615.725\n",
      "INFO:tensorflow:loss = 0.8536211, step = 2100 (0.163 sec)\n",
      "INFO:tensorflow:global_step/sec: 760.193\n",
      "INFO:tensorflow:loss = 0.54129523, step = 2200 (0.130 sec)\n",
      "INFO:tensorflow:global_step/sec: 599.017\n",
      "INFO:tensorflow:loss = 0.6218288, step = 2300 (0.168 sec)\n",
      "INFO:tensorflow:global_step/sec: 486.052\n",
      "INFO:tensorflow:loss = 0.4102406, step = 2400 (0.205 sec)\n",
      "INFO:tensorflow:global_step/sec: 797.551\n",
      "INFO:tensorflow:loss = 0.5401689, step = 2500 (0.125 sec)\n",
      "INFO:tensorflow:global_step/sec: 613.293\n",
      "INFO:tensorflow:loss = 0.4659374, step = 2600 (0.162 sec)\n",
      "INFO:tensorflow:global_step/sec: 632.637\n",
      "INFO:tensorflow:loss = 0.8490702, step = 2700 (0.158 sec)\n",
      "INFO:tensorflow:global_step/sec: 680.495\n",
      "INFO:tensorflow:loss = 0.35606387, step = 2800 (0.147 sec)\n",
      "INFO:tensorflow:global_step/sec: 677.439\n",
      "INFO:tensorflow:loss = 0.35035092, step = 2900 (0.148 sec)\n",
      "INFO:tensorflow:global_step/sec: 622.563\n",
      "INFO:tensorflow:loss = 0.42017668, step = 3000 (0.160 sec)\n",
      "INFO:tensorflow:global_step/sec: 612.329\n",
      "INFO:tensorflow:loss = 0.37329862, step = 3100 (0.165 sec)\n",
      "INFO:tensorflow:global_step/sec: 692.713\n",
      "INFO:tensorflow:loss = 0.42661282, step = 3200 (0.143 sec)\n",
      "INFO:tensorflow:global_step/sec: 698.338\n",
      "INFO:tensorflow:loss = 0.3283816, step = 3300 (0.143 sec)\n",
      "INFO:tensorflow:global_step/sec: 740.027\n",
      "INFO:tensorflow:loss = 0.31534225, step = 3400 (0.136 sec)\n",
      "INFO:tensorflow:global_step/sec: 703.132\n",
      "INFO:tensorflow:loss = 0.7215259, step = 3500 (0.141 sec)\n",
      "INFO:tensorflow:global_step/sec: 723.832\n",
      "INFO:tensorflow:loss = 0.35926887, step = 3600 (0.139 sec)\n",
      "INFO:tensorflow:global_step/sec: 567.262\n",
      "INFO:tensorflow:loss = 0.46332362, step = 3700 (0.177 sec)\n",
      "INFO:tensorflow:global_step/sec: 656.294\n",
      "INFO:tensorflow:loss = 0.3631253, step = 3800 (0.151 sec)\n",
      "INFO:tensorflow:global_step/sec: 154.935\n",
      "INFO:tensorflow:loss = 0.3349234, step = 3900 (0.645 sec)\n",
      "INFO:tensorflow:global_step/sec: 668.197\n",
      "INFO:tensorflow:loss = 0.29228905, step = 4000 (0.151 sec)\n",
      "INFO:tensorflow:global_step/sec: 598.769\n",
      "INFO:tensorflow:loss = 0.25082293, step = 4100 (0.167 sec)\n",
      "INFO:tensorflow:global_step/sec: 676.129\n",
      "INFO:tensorflow:loss = 0.30674836, step = 4200 (0.146 sec)\n",
      "INFO:tensorflow:global_step/sec: 692.863\n",
      "INFO:tensorflow:loss = 0.51635873, step = 4300 (0.144 sec)\n",
      "INFO:tensorflow:global_step/sec: 684.939\n",
      "INFO:tensorflow:loss = 0.42645568, step = 4400 (0.146 sec)\n",
      "INFO:tensorflow:global_step/sec: 762.319\n",
      "INFO:tensorflow:loss = 0.29751444, step = 4500 (0.131 sec)\n",
      "INFO:tensorflow:global_step/sec: 685.046\n",
      "INFO:tensorflow:loss = 0.7011014, step = 4600 (0.146 sec)\n",
      "INFO:tensorflow:global_step/sec: 741.743\n",
      "INFO:tensorflow:loss = 0.48851538, step = 4700 (0.136 sec)\n",
      "INFO:tensorflow:global_step/sec: 652.895\n",
      "INFO:tensorflow:loss = 0.9379805, step = 4800 (0.152 sec)\n",
      "INFO:tensorflow:global_step/sec: 621.762\n",
      "INFO:tensorflow:loss = 0.34004694, step = 4900 (0.162 sec)\n",
      "INFO:tensorflow:global_step/sec: 676.429\n",
      "INFO:tensorflow:loss = 0.9120659, step = 5000 (0.149 sec)\n",
      "INFO:tensorflow:global_step/sec: 603.053\n",
      "INFO:tensorflow:loss = 0.2791518, step = 5100 (0.163 sec)\n",
      "INFO:tensorflow:global_step/sec: 721.963\n",
      "INFO:tensorflow:loss = 0.50930357, step = 5200 (0.139 sec)\n",
      "INFO:tensorflow:global_step/sec: 666.78\n",
      "INFO:tensorflow:loss = 0.3572204, step = 5300 (0.150 sec)\n",
      "INFO:tensorflow:global_step/sec: 709.711\n",
      "INFO:tensorflow:loss = 0.34792233, step = 5400 (0.141 sec)\n",
      "INFO:tensorflow:global_step/sec: 620.28\n",
      "INFO:tensorflow:loss = 0.45399022, step = 5500 (0.162 sec)\n",
      "INFO:tensorflow:global_step/sec: 522.49\n",
      "INFO:tensorflow:loss = 0.31871036, step = 5600 (0.191 sec)\n",
      "INFO:tensorflow:global_step/sec: 677.262\n",
      "INFO:tensorflow:loss = 0.335261, step = 5700 (0.149 sec)\n",
      "INFO:tensorflow:global_step/sec: 761.847\n",
      "INFO:tensorflow:loss = 0.6656867, step = 5800 (0.131 sec)\n",
      "INFO:tensorflow:global_step/sec: 745.706\n",
      "INFO:tensorflow:loss = 0.33416766, step = 5900 (0.134 sec)\n",
      "INFO:tensorflow:global_step/sec: 779.581\n",
      "INFO:tensorflow:loss = 0.40265596, step = 6000 (0.128 sec)\n",
      "INFO:tensorflow:global_step/sec: 749.313\n",
      "INFO:tensorflow:loss = 0.40928322, step = 6100 (0.133 sec)\n",
      "INFO:tensorflow:global_step/sec: 704.562\n",
      "INFO:tensorflow:loss = 0.4051754, step = 6200 (0.142 sec)\n",
      "INFO:tensorflow:global_step/sec: 583.138\n",
      "INFO:tensorflow:loss = 0.26573443, step = 6300 (0.172 sec)\n",
      "INFO:tensorflow:global_step/sec: 650.521\n",
      "INFO:tensorflow:loss = 0.21339291, step = 6400 (0.153 sec)\n",
      "INFO:tensorflow:global_step/sec: 680.171\n",
      "INFO:tensorflow:loss = 0.23495217, step = 6500 (0.147 sec)\n",
      "INFO:tensorflow:global_step/sec: 774.361\n",
      "INFO:tensorflow:loss = 0.3293721, step = 6600 (0.129 sec)\n",
      "INFO:tensorflow:global_step/sec: 722.264\n",
      "INFO:tensorflow:loss = 0.62516314, step = 6700 (0.139 sec)\n",
      "INFO:tensorflow:global_step/sec: 713.252\n",
      "INFO:tensorflow:loss = 0.39132473, step = 6800 (0.140 sec)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:global_step/sec: 848.486\n",
      "INFO:tensorflow:loss = 0.40685356, step = 6900 (0.118 sec)\n",
      "INFO:tensorflow:global_step/sec: 855.584\n",
      "INFO:tensorflow:loss = 0.38497496, step = 7000 (0.117 sec)\n",
      "INFO:tensorflow:global_step/sec: 653.319\n",
      "INFO:tensorflow:loss = 0.3524521, step = 7100 (0.153 sec)\n",
      "INFO:tensorflow:global_step/sec: 698.997\n",
      "INFO:tensorflow:loss = 0.3569974, step = 7200 (0.142 sec)\n",
      "INFO:tensorflow:global_step/sec: 689.407\n",
      "INFO:tensorflow:loss = 0.37861946, step = 7300 (0.145 sec)\n",
      "INFO:tensorflow:global_step/sec: 724.54\n",
      "INFO:tensorflow:loss = 0.371001, step = 7400 (0.138 sec)\n",
      "INFO:tensorflow:global_step/sec: 643.304\n",
      "INFO:tensorflow:loss = 0.42895234, step = 7500 (0.155 sec)\n",
      "INFO:tensorflow:global_step/sec: 691.365\n",
      "INFO:tensorflow:loss = 0.38287464, step = 7600 (0.145 sec)\n",
      "INFO:tensorflow:global_step/sec: 724.676\n",
      "INFO:tensorflow:loss = 0.3309633, step = 7700 (0.138 sec)\n",
      "INFO:tensorflow:global_step/sec: 627.989\n",
      "INFO:tensorflow:loss = 0.22624245, step = 7800 (0.162 sec)\n",
      "INFO:tensorflow:global_step/sec: 677.592\n",
      "INFO:tensorflow:loss = 0.2507131, step = 7900 (0.147 sec)\n",
      "INFO:tensorflow:global_step/sec: 744.032\n",
      "INFO:tensorflow:loss = 0.25926965, step = 8000 (0.133 sec)\n",
      "INFO:tensorflow:global_step/sec: 751.607\n",
      "INFO:tensorflow:loss = 0.2192243, step = 8100 (0.134 sec)\n",
      "INFO:tensorflow:global_step/sec: 652.066\n",
      "INFO:tensorflow:loss = 0.55547285, step = 8200 (0.153 sec)\n",
      "INFO:tensorflow:global_step/sec: 637.072\n",
      "INFO:tensorflow:loss = 0.1896804, step = 8300 (0.156 sec)\n",
      "INFO:tensorflow:global_step/sec: 754.103\n",
      "INFO:tensorflow:loss = 0.4603298, step = 8400 (0.133 sec)\n",
      "INFO:tensorflow:global_step/sec: 152.26\n",
      "INFO:tensorflow:loss = 0.39936733, step = 8500 (0.656 sec)\n",
      "INFO:tensorflow:global_step/sec: 681.011\n",
      "INFO:tensorflow:loss = 0.23406655, step = 8600 (0.147 sec)\n",
      "INFO:tensorflow:global_step/sec: 796.391\n",
      "INFO:tensorflow:loss = 0.4344553, step = 8700 (0.127 sec)\n",
      "INFO:tensorflow:global_step/sec: 688.678\n",
      "INFO:tensorflow:loss = 0.5266614, step = 8800 (0.143 sec)\n",
      "INFO:tensorflow:global_step/sec: 692.255\n",
      "INFO:tensorflow:loss = 0.32180628, step = 8900 (0.145 sec)\n",
      "INFO:tensorflow:global_step/sec: 735.5\n",
      "INFO:tensorflow:loss = 0.34035486, step = 9000 (0.136 sec)\n",
      "INFO:tensorflow:global_step/sec: 659.728\n",
      "INFO:tensorflow:loss = 0.4173571, step = 9100 (0.152 sec)\n",
      "INFO:tensorflow:global_step/sec: 704.133\n",
      "INFO:tensorflow:loss = 0.5337366, step = 9200 (0.142 sec)\n",
      "INFO:tensorflow:global_step/sec: 721.643\n",
      "INFO:tensorflow:loss = 0.44501668, step = 9300 (0.139 sec)\n",
      "INFO:tensorflow:global_step/sec: 660.833\n",
      "INFO:tensorflow:loss = 0.37879056, step = 9400 (0.153 sec)\n",
      "INFO:tensorflow:global_step/sec: 729.089\n",
      "INFO:tensorflow:loss = 0.49887902, step = 9500 (0.136 sec)\n",
      "INFO:tensorflow:global_step/sec: 654.087\n",
      "INFO:tensorflow:loss = 0.3467123, step = 9600 (0.153 sec)\n",
      "INFO:tensorflow:global_step/sec: 616.626\n",
      "INFO:tensorflow:loss = 0.40506393, step = 9700 (0.162 sec)\n",
      "INFO:tensorflow:global_step/sec: 636.13\n",
      "INFO:tensorflow:loss = 0.25415385, step = 9800 (0.157 sec)\n",
      "INFO:tensorflow:global_step/sec: 747.099\n",
      "INFO:tensorflow:loss = 0.48888522, step = 9900 (0.134 sec)\n",
      "INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 10000...\n",
      "INFO:tensorflow:Saving checkpoints for 10000 into /tmp/tmpqpwcr_05/model.ckpt.\n",
      "INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 10000...\n",
      "INFO:tensorflow:Loss for final step: 0.27114332.\n",
      "INFO:tensorflow:Calling model_fn.\n",
      "INFO:tensorflow:Done calling model_fn.\n",
      "INFO:tensorflow:Starting evaluation at 2020-08-05T16:12:53Z\n",
      "INFO:tensorflow:Graph was finalized.\n",
      "INFO:tensorflow:Restoring parameters from /tmp/tmpqpwcr_05/model.ckpt-10000\n",
      "INFO:tensorflow:Running local_init_op.\n",
      "INFO:tensorflow:Done running local_init_op.\n",
      "INFO:tensorflow:Inference Time : 1.73152s\n",
      "INFO:tensorflow:Finished evaluation at 2020-08-05-16:12:55\n",
      "INFO:tensorflow:Saving dict for global step 10000: accuracy = 0.8915183, accuracy_baseline = 0.88809025, auc = 0.74716, auc_precision_recall = 0.33547643, average_loss = 0.33921474, global_step = 10000, label/mean = 0.11190976, loss = 0.3391198, precision = 0.5536332, prediction/mean = 0.15207805, recall = 0.15810277\n",
      "INFO:tensorflow:Saving 'checkpoint_path' summary for global step 10000: /tmp/tmpqpwcr_05/model.ckpt-10000\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'accuracy': 0.8915183,\n",
       " 'accuracy_baseline': 0.88809025,\n",
       " 'auc': 0.74716,\n",
       " 'auc_precision_recall': 0.33547643,\n",
       " 'average_loss': 0.33921474,\n",
       " 'label/mean': 0.11190976,\n",
       " 'loss': 0.3391198,\n",
       " 'precision': 0.5536332,\n",
       " 'prediction/mean': 0.15207805,\n",
       " 'recall': 0.15810277,\n",
       " 'global_step': 10000}"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dnn = tf.estimator.DNNClassifier(\n",
    "    hidden_units=[64, 32, 16], \n",
    "    feature_columns=feature_columns)\n",
    "\n",
    "dnn.train(train_input_fn, max_steps=10000)\n",
    "result = dnn.evaluate(test_input_fn)\n",
    "result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [],
   "name": "feature_columns.ipynb",
   "private_outputs": true,
   "provenance": [],
   "toc_visible": true,
   "version": "0.3.2"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}