mindspore_main.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 目标检测：口罩佩戴检测  \n",
    "\n",
    "<br>\n",
    "<hr>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "toc-hr-collapsed": true
   },
   "source": [
    "## 1.实验介绍"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.1 实验背景  \n",
    "\n",
    "今年一场席卷全球的新型冠状病毒给人们带来了沉重的生命财产的损失。  \n",
    "有效防御这种传染病毒的方法就是积极佩戴口罩。  \n",
    "我国对此也采取了严肃的措施，在公共场合要求人们必须佩戴口罩。  \n",
    "在本次实验中，我们要建立一个目标检测的模型，可以识别图中的人是否佩戴了口罩。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.2 实验要求\n",
    "\n",
    "1. 建立深度学习模型，检测出图中的人是否佩戴了口罩，并将其尽可能调整到最佳状态。 \n",
    "2. 学习OpenCV dnn的使用方法，以及经典模型 MobileNetV2 的结构。\n",
    "3. 学习训练时的方法。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.3 实验环境\n",
    "\n",
    "可以使用基于 Python 的 OpenCV 、PIL 库进行图像相关处理，使用 Numpy 库进行相关数值运算，使用 MindSpore 等深度学习框架训练模型等。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.4 注意事项  \n",
    "+ Python 与 Python Package 的使用方式，可在右侧 `API文档` 中查阅。\n",
    "+ 当右上角的『Python 3』长时间指示为运行中的时候，造成代码无法执行时，可以重新启动 Kernel 解决（左上角『Kernel』-『Restart Kernel』）。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.5 参考资料\n",
    "+ 论文 Joint Face Detection and Alignment using Multi-task Cascaded Convolutional Networks：https://kpzhang93.github.io/MTCNN_face_detection_alignment/\n",
    "+ OpenCV：https://opencv-python-tutroals.readthedocs.io/en/latest/py_tutorials/py_tutorials.html\n",
    "+ PIL：https://pillow.readthedocs.io/en/stable/\n",
    "+ Numpy：https://www.numpy.org/\n",
    "+ Scikit-learn： https://scikit-learn.org/\n",
    "+ mindspore：https://www.mindspore.cn/tutorial/training/zh-CN/master/index.html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.6 实验思路\n",
    "\n",
    "针对目标检测的任务，可以分为两个部分：目标识别和位置检测。  \n",
    "通常情况下，特征提取需要由特有的特征提取神经网络来完成，如 VGG、MobileNet、ResNet 等，这些特征提取网络往往被称为 Backbone 。而在 BackBone 后面接全连接层(FC)就可以执行分类任务。  \n",
    "但 FC 对目标的位置识别乏力。经过算法的发展，当前主要以特定的功能网络来代替 FC 的作用，如 Mask-Rcnn、SSD、YOLO 等。  \n",
    "我们选择充分使用已有的人脸检测的模型，再训练一个识别口罩的模型，从而提高训练的开支、增强模型的准确率。\n",
    "\n",
    "**常规目标检测：**  \n",
    "\n",
    "<img src=\"https://imgbed.momodel.cn/20200914162156.png\" width=500px/>\n",
    "\n",
    "\n",
    "\n",
    "**本次案例：**   \n",
    "\n",
    "\n",
    "<img src=\"https://imgbed.momodel.cn/20200918102630.png\" width=500px/>\n",
    "\n",
    "<br>\n",
    "<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "toc-hr-collapsed": false
   },
   "source": [
    "## 2. OpenCV 人脸检测"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "数据信息存放在 `/datasets/5f680a696ec9b83bb0037081-momodel/data` 文件夹下。    \n",
    "该文件夹主要有文件夹 `image`、文件 `train.txt` 、文件夹 `keras_model_data` 和文件夹 `mindspore_model_data`共四部分：\n",
    "+ **image 文件夹**：图片分成两类，戴口罩的和没有戴口罩的  \n",
    "+ **train.txt**：  存放的是 image 文件夹下对应图片的标签  （keras 框架专用文件）\n",
    "+ **keras_model_data** 文件夹：存放 keras 框架相关预训练好的模型 （keras 框架专用文件夹）\n",
    "+ **mindspore_model_data** 文件夹：存放 mindspore 框架相关预训练好的模型（mindspore 框架专用文件）\n",
    "\n",
    "opencv 人脸检测模型在数据集 **mindspore_model_data/opencv_dnn** 文件夹中"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "select": true
   },
   "outputs": [],
   "source": [
    "import os\n",
    "# 数据集路径\n",
    "basic_path = \"./datasets/5f680a696ec9b83bb0037081-momodel/data/\"\n",
    "# opencv 人脸检测模型在数据集 mindspore_model_data/opencv_dnn 文件夹中\n",
    "opencv_dnn_path = basic_path + 'mindspore_model_data/opencv_dnn'\n",
    "print(opencv_dnn_path)\n",
    "# 查看文件夹里面文件\n",
    "os.listdir(opencv_dnn_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "发现该文件夹下有我们需要的路径，所以\n",
    "+ **依赖文件夹的路径为 opencv_dnn_path** =           \n",
    "`/datasets/5f680a696ec9b83bb0037081-momodel/data/mindspore_model_data/opencv_dnn`\n",
    "\n",
    "+ **deploy.prototxt 文件的路径**：       \n",
    "`opencv_dnn_path + '/' + 'deploy.prototxt'`\n",
    "+ **res10_300x300_ssd_iter_140000_fp16.caffemodel 文件的路径**：        \n",
    "`opencv_dnn_path + '/' + 'res10_300x300_ssd_iter_140000_fp16.caffemodel'`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "select": true
   },
   "outputs": [],
   "source": [
    "import os\n",
    "import numpy as np\n",
    "import cv2\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "class FaceDet():\n",
    "    def __init__(self):\n",
    "        self.opencv_dnn_path = 'datasets/5f680a696ec9b83bb0037081-momodel/data/mindspore_model_data/opencv_dnn/'\n",
    "        self.threshold = 0.15\n",
    "        self.caffe_model = self.opencv_dnn_path + \"deploy.prototxt\"\n",
    "        self.caffe_param = self.opencv_dnn_path + \"res10_300x300_ssd_iter_140000_fp16.caffemodel\"\n",
    "        \n",
    "    def draw_detections(self, image, detections):\n",
    "        h, w, c = image.shape\n",
    "        for i in range(0, detections.shape[2]):\n",
    "            confidence = detections[0, 0, i, 2]\n",
    "            if confidence > self.threshold:\n",
    "                box = detections[0, 0, i, 3:7] * np.array([w, h, w, h])\n",
    "                (startX, startY, endX, endY) = box.astype(\"int\")\n",
    "                text = \"{:.2f}%\".format(confidence * 100)\n",
    "                y = startY - 10 if startY - 10 > 10 else startY + 10\n",
    "                cv2.rectangle(image, (startX, startY), (endX, endY),\n",
    "                              (0, 255, 0), 1)\n",
    "                cv2.putText(image, text, (startX, y),\n",
    "                            cv2.FONT_HERSHEY_SIMPLEX, 0.45, (0, 255, 0), 2)\n",
    "        return image\n",
    "\n",
    "    def detect(self, image):\n",
    "        net = cv2.dnn.readNetFromCaffe(self.caffe_model, self.caffe_param)\n",
    "\n",
    "        # def blobFromImage(image, scalefactor=None, size=None, mean=None, swapRB=None, crop=None, ddepth=None)\n",
    "        # image：输入图像\n",
    "        # mean：对每个通道像素值减去对应的均值，这里用(104.0, 177.0, 123.0)，和模型训练时的值一致\n",
    "        # scalefactor：对像素值的缩放比例\n",
    "        # size：模型输入图片的尺寸\n",
    "        # swapRB：OpenCV默认的图片通道顺序是BGR，如果需要交换R和G，则设为True\n",
    "        # crop: 调整图片大小后，是否裁剪\n",
    "        blob = cv2.dnn.blobFromImage(image, 1.0, (300, 300), (104.0, 177.0, 123.0), False, False)\n",
    "        net.setInput(blob)\n",
    "        detections = net.forward()\n",
    "        return detections"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "验证一下人脸检测的效果，其中人脸框上方的`xx%`为置信度"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "img = cv2.imread(\"test.jpg\")\n",
    "detect = FaceDet()\n",
    "detections = detect.detect(img)\n",
    "drawed_img = detect.draw_detections(img, detections)\n",
    "\n",
    "# OpenCV reads image to BGR format. Transform images before showing it.\n",
    "drawed_img = cv2.cvtColor(drawed_img, cv2.COLOR_BGR2RGB)\n",
    "plt.figure(figsize = (8,8))\n",
    "plt.imshow(drawed_img)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "toc-hr-collapsed": false
   },
   "source": [
    "## 3 口罩识别"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "导入标准库、第三方库，已及MindSpore的模块。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "select": true
   },
   "outputs": [],
   "source": [
    "import math\n",
    "import os\n",
    "import numpy as np\n",
    "import cv2\n",
    "import matplotlib.pyplot as plt\n",
    "from easydict import EasyDict\n",
    "from PIL import Image\n",
    "\n",
    "from mindspore import context\n",
    "from mindspore import nn\n",
    "from mindspore import Tensor\n",
    "from mindspore.train.model import Model\n",
    "from mindspore.train.serialization import load_checkpoint\n",
    "from mindspore.train.callback import Callback\n",
    "from mindspore.train.callback import LossMonitor\n",
    "from mindspore.train.callback import ModelCheckpoint\n",
    "from mindspore.train.callback import CheckpointConfig\n",
    "\n",
    "# 模型定义脚本以及数据处理脚本\n",
    "from mindspore_py.mobilenetV2 import MobileNetV2Backbone \n",
    "from mindspore_py.mobilenetV2 import MobileNetV2Head \n",
    "from mindspore_py.mobilenetV2 import mobilenet_v2 \n",
    "from mindspore_py.dataset import create_dataset\n",
    "# Log Level = Error\n",
    "os.environ['GLOG_v'] = '3' \n",
    "# 设置采用图模式执行，设备为CPU/GPU\n",
    "context.set_context(mode=context.GRAPH_MODE, device_target=\"CPU\") "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.1 数据集介绍\n",
    "\n",
    "数据信息存放在 `/datasets/5f680a696ec9b83bb0037081-momodel/data/image` 文件夹下。              \n",
    "收集的图片分成 mask 和 nomask 两类，戴口罩的和没有戴口罩的。        \n",
    "现在我们尝试读取数据集中的一张戴口罩的图片并显示图片名称。  \n",
    "现在我们尝试读取数据集中戴口罩的图片及其名称，以下是训练集中的正样本："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mask_num = 4\n",
    "fig = plt.figure(figsize = (15,15))\n",
    "basic_path = \"./datasets/5f680a696ec9b83bb0037081-momodel/data/\"\n",
    "for i in range(mask_num):\n",
    "    sub_img = cv2.imread(basic_path + \"/image/mask/mask_\" + str(i + 101) + \".jpg\")\n",
    "    sub_img = cv2.cvtColor(sub_img, cv2.COLOR_RGB2BGR)\n",
    "    ax = fig.add_subplot(4, 4, (i + 1))\n",
    "    ax.set_xticks([])\n",
    "    ax.set_yticks([])\n",
    "    ax.set_title(\"mask_\" + str(i + 1))\n",
    "    ax.imshow(sub_img)\n",
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "以下是训练集中的负样本："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "nomask_num = 4\n",
    "fig1 = plt.figure(figsize=(15, 15))\n",
    "for i in range(nomask_num):\n",
    "    sub_img = cv2.imread(basic_path + \"/image/nomask/nomask_\" + str(i + 130) + \".jpg\")\n",
    "    sub_img = cv2.cvtColor(sub_img, cv2.COLOR_RGB2BGR)\n",
    "    ax = fig1.add_subplot(4, 4, (i + 1))\n",
    "    ax.set_xticks([])\n",
    "    ax.set_yticks([])\n",
    "    ax.set_title(\"nomask_\" + str(i + 1))\n",
    "    ax.imshow(sub_img)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.2 模型训练Tips"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "配置后续训练、验证、推理用到的参数。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "select": true
   },
   "outputs": [],
   "source": [
    "basic_path = \"./datasets/5f680a696ec9b83bb0037081-momodel/data/\"\n",
    "config = EasyDict({\n",
    "    \"num_classes\": 2,\n",
    "    \"image_height\": 224,\n",
    "    \"image_width\": 224,\n",
    "    \"data_split\": [0.9, 0.1],\n",
    "    \"backbone_out_channels\":1280,\n",
    "    \"batch_size\": 16,\n",
    "    \"eval_batch_size\": 8,\n",
    "    \"epochs\": 3,\n",
    "    \"lr_max\": 0.01,\n",
    "    \"momentum\": 0.9,\n",
    "    \"weight_decay\": 1e-4,\n",
    "    \"save_checkpoint\": True,\n",
    "    \"save_checkpoint_epochs\": 1,\n",
    "    \"save_checkpoint_path\": \"./results\",\n",
    "    \"dataset_path\": basic_path + \"image\",\n",
    "    \"pretrained_ckpt\": basic_path + \"mindspore_model_data/mobilenetV2-200_1067.ckpt\"\n",
    "})\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 3.2.1 动态学习率\n",
    "\n",
    "一般情况下，模型训练时采用静态学习率，如0.01。随着训练步数的增加，模型逐渐趋于收敛，对权重参数的更新幅度应该逐渐降低，以减小模型训练后期的抖动。所以，模型训练时可以采用动态下降的学习率，常见的学习率下降策略有：\n",
    "\n",
    "- polynomial decay/square decay\n",
    "- cosine decay\n",
    "- exponential decay\n",
    "- stage decay\n",
    "\n",
    "这里实现 cosine decay 下降策略。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "select": true
   },
   "outputs": [],
   "source": [
    "def cosine_decay(total_steps, lr_init=0.0, lr_end=0.0, lr_max=0.1, warmup_steps=0):\n",
    "    \"\"\"\n",
    "    Applies cosine decay to generate learning rate array.\n",
    "\n",
    "    Args:\n",
    "       total_steps(int): all steps in training.\n",
    "       lr_init(float): init learning rate.\n",
    "       lr_end(float): end learning rate\n",
    "       lr_max(float): max learning rate.\n",
    "       warmup_steps(int): all steps in warmup epochs.\n",
    "\n",
    "    Returns:\n",
    "       list, learning rate array.\n",
    "    \"\"\"\n",
    "    lr_init, lr_end, lr_max = float(lr_init), float(lr_end), float(lr_max)\n",
    "    decay_steps = total_steps - warmup_steps\n",
    "    lr_all_steps = []\n",
    "    inc_per_step = (lr_max - lr_init) / warmup_steps if warmup_steps else 0\n",
    "    for i in range(total_steps):\n",
    "        if i < warmup_steps:\n",
    "            lr = lr_init + lr_inc * (i + 1)\n",
    "        else:\n",
    "            cosine_decay = 0.5 * (1 + math.cos(math.pi * (i - warmup_steps) / decay_steps))\n",
    "            lr = (lr_max - lr_end) * cosine_decay + lr_end\n",
    "        lr_all_steps.append(lr)\n",
    "\n",
    "    return lr_all_steps"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 3.2.2 边训练边验证\n",
    "\n",
    "在面对复杂网络时，往往需要进行几十甚至几百次的epoch训练。在训练之前，很难掌握在训练到第几个epoch时，模型的精度能达到满足要求的程度，所以经常会采用一边训练的同时，在相隔固定epoch的位置对模型进行精度验证，并保存相应的模型，等训练完毕后，通过查看对应模型精度的变化就能迅速地挑选出相对最优的模型。流程如下：\n",
    "\n",
    "- 定义回调函数EvalCallback，实现同步进行训练和验证。\n",
    "- 定义训练网络并执行。\n",
    "- 将不同epoch下的模型精度绘制出折线图并挑选最优模型Checkpoint。\n",
    "\n",
    "当我们训练深度学习神经网络的时候通常希望能获得最好的泛化性能。但是深度学习神经网络很容易过拟合。当网络在训练集上表现越来越好，错误率越来越低的时候，就极有可能出现了过拟合。我们可以设计一种早停法，比如验证精度连续5次不在上升就停止训练，这样能避免继续训练导致过拟合的问题。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "select": true
   },
   "outputs": [],
   "source": [
    "class EvalCallback(Callback):\n",
    "    def __init__(self, model, eval_dataset, history, eval_epochs=1):\n",
    "        self.model = model\n",
    "        self.eval_dataset = eval_dataset\n",
    "        self.eval_epochs = eval_epochs\n",
    "        self.history = history\n",
    "        self.acc_max = 0\n",
    "        # acc连续5次<=过程中的最大值，则停止训练\n",
    "        self.count_max = 5\n",
    "        self.count = 0\n",
    "    \n",
    "    def epoch_begin(self, run_context):\n",
    "        self.losses = []\n",
    "    \n",
    "    def step_end(self, run_context):\n",
    "        cb_param = run_context.original_args()\n",
    "        loss = cb_param.net_outputs\n",
    "        self.losses.append(loss.asnumpy())\n",
    "    \n",
    "    def epoch_end(self, run_context):\n",
    "        cb_param = run_context.original_args()\n",
    "        cur_epoch = cb_param.cur_epoch_num\n",
    "        train_loss = np.mean(self.losses)\n",
    "        \n",
    "        if cur_epoch % self.eval_epochs == 0:\n",
    "            metric = self.model.eval(self.eval_dataset, dataset_sink_mode=False)\n",
    "            self.history[\"epoch\"].append(cur_epoch)\n",
    "            self.history[\"eval_acc\"].append(metric[\"acc\"])\n",
    "            self.history[\"eval_loss\"].append(metric[\"loss\"])\n",
    "            self.history[\"train_loss\"].append(train_loss)\n",
    "            if self.acc_max < metric[\"acc\"]:\n",
    "                self.count = 0\n",
    "                self.acc_max = metric[\"acc\"]\n",
    "            else:\n",
    "                self.count += 1\n",
    "                if self.count == self.count_max:\n",
    "                    run_context.request_stop()\n",
    "            print(\"epoch: %d, train_loss: %f, eval_loss: %f, eval_acc: %f\" %(cur_epoch, train_loss, metric[\"loss\"], metric[\"acc\"]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.3 模型训练\n",
    "\n",
    "在模型训练过程中，可以添加检查点（Checkpoint）用于保存模型的参数，以便进行推理及中断后再训练使用。使用场景如下：\n",
    "\n",
    "- 训练后推理场景\n",
    "    - 模型训练完毕后保存模型的参数，用于推理或预测操作。\n",
    "    - 训练过程中，通过实时验证精度，把精度最高的模型参数保存下来，用于预测操作。\n",
    "- 再训练场景\n",
    "    - 进行长时间训练任务时，保存训练过程中的Checkpoint文件，防止任务异常退出后从初始状态开始训练。\n",
    "    - Fine-tuning（微调）场景，即训练一个模型并保存参数，基于该模型，面向第二个类似任务进行模型训练。\n",
    "\n",
    "这里加载 ImageNet 数据上预训练的 MobileNetv2 进行 Fine-tuning，**只训练最后修改的 FC 层**，并在训练过程中保存 Checkpoint。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "select": true
   },
   "outputs": [],
   "source": [
    "# 运行该 cell 代码训练模型时，请清理 results 文件夹中 mindspore 框架之前训练好的模型，否则 model_path 格式会发生一定的变化，造成推理时可能找不到模型的报错\n",
    "def train():\n",
    "    train_dataset, eval_dataset = create_dataset(dataset_path=config.dataset_path, config=config)\n",
    "    step_size = train_dataset.get_dataset_size()\n",
    "    \n",
    "    backbone = MobileNetV2Backbone() #last_channel=config.backbone_out_channels\n",
    "    # Freeze parameters of backbone. You can comment these two lines.\n",
    "    for param in backbone.get_parameters():\n",
    "       param.requires_grad = False\n",
    "    # load parameters from pretrained model\n",
    "    load_checkpoint(config.pretrained_ckpt, backbone)\n",
    "\n",
    "    # head = MobileNetV2Head(num_classes=config.num_classes, last_channel=config.backbone_out_channels)\n",
    "    head = MobileNetV2Head(input_channel=backbone.out_channels, num_classes=config.num_classes)\n",
    "    network = mobilenet_v2(backbone, head)\n",
    "\n",
    "    # define loss, optimizer, and model\n",
    "    loss = nn.SoftmaxCrossEntropyWithLogits(sparse=True, reduction='mean')\n",
    "    lrs = cosine_decay(config.epochs * step_size, lr_max=config.lr_max)\n",
    "    opt = nn.Momentum(network.trainable_params(), lrs, config.momentum, config.weight_decay)\n",
    "    model = Model(network, loss, opt, metrics={'acc', 'loss'})\n",
    "\n",
    "    history = {'epoch': [], 'train_loss': [], 'eval_loss': [], 'eval_acc': []}\n",
    "    eval_cb = EvalCallback(model, eval_dataset, history)\n",
    "    cb = [eval_cb]\n",
    "    if config.save_checkpoint:\n",
    "        ckpt_cfg = CheckpointConfig(save_checkpoint_steps=config.save_checkpoint_epochs * step_size, keep_checkpoint_max=config.epochs)\n",
    "        ckpt_cb = ModelCheckpoint(prefix=\"mobilenetv2_mask\", directory=config.save_checkpoint_path, config=ckpt_cfg)\n",
    "        cb.append(ckpt_cb)\n",
    "    model.train(config.epochs, train_dataset, callbacks=cb, dataset_sink_mode=False)\n",
    "    \n",
    "    return history"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "将不同 epoch 下的模型精度绘制出折线图并挑选最优模型 Checkpoint。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "select": true
   },
   "outputs": [],
   "source": [
    "history = train()\n",
    "\n",
    "plt.plot(history['epoch'], history['train_loss'], label='train_loss')\n",
    "plt.plot(history['epoch'], history['eval_loss'], 'r', label='val_loss')\n",
    "plt.legend()\n",
    "plt.show()\n",
    "\n",
    "plt.plot(history['epoch'], history['eval_acc'], 'r', label = 'val_acc')\n",
    "plt.legend()\n",
    "plt.show()\n",
    "\n",
    "model_path = 'results/mobilenetv2_mask-%d_39.ckpt' % (np.argmax(history['eval_acc']) + 1) # 挑选出最优模型Checkpoint\n",
    "print(\"the path of best model checkpoint is :\", model_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.4 模型推理\n",
    "\n",
    "加载模型 Checkpoint 进行推理。           \n",
    "使用 load_checkpoint 接口加载数据时，需要把数据传入给原始网络，而不能传递给带有优化器和损失函数的训练网络。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "select": true
   },
   "outputs": [],
   "source": [
    "def image_process(image):\n",
    "    \"\"\"Precess one image per time.\n",
    "    \n",
    "    Args:\n",
    "        image: shape (H, W, C)\n",
    "    \"\"\"\n",
    "    mean=[0.485*255, 0.456*255, 0.406*255]\n",
    "    std=[0.229*255, 0.224*255, 0.225*255]\n",
    "    image = (np.array(image) - mean) / std\n",
    "    image = image.transpose((2,0,1))\n",
    "    img_tensor = Tensor(np.array([image], np.float32))\n",
    "    return img_tensor\n",
    "\n",
    "def infer_one(network, image_path):\n",
    "    image = Image.open(image_path).resize((config.image_height, config.image_width))\n",
    "    logits = network(image_process(image))\n",
    "    pred = np.argmax(logits.asnumpy(), axis=1)[0]\n",
    "    print(\"图片路径：\", image_path, \"图片预测类别\",pred)\n",
    "\n",
    "def infer(basic_path, model_path):\n",
    "    backbone = MobileNetV2Backbone(last_channel=config.backbone_out_channels)\n",
    "    head = MobileNetV2Head(input_channel=backbone.out_channels, num_classes=config.num_classes)\n",
    "    network = mobilenet_v2(backbone, head)\n",
    "    load_checkpoint(model_path, network)\n",
    "    for i in range(250, 258):\n",
    "        infer_one(network, basic_path + 'image/mask/mask_%s.jpg' % i)\n",
    "    for i in range(371, 378):\n",
    "        infer_one(network, basic_path + 'image/nomask/nomask_%s.jpg' % i)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "infer(basic_path, model_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.5 口罩识别"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "select": true
   },
   "outputs": [],
   "source": [
    "class MaskRec():\n",
    "    def __init__(self, model_path):\n",
    "        self.face_det = FaceDet()\n",
    "        self.mask_model_input_size = (160, 160)\n",
    "        self.class_names = ['YES', 'NO']\n",
    "        # 初始化MobileNetv2\n",
    "        backbone = MobileNetV2Backbone(last_channel=config.backbone_out_channels)\n",
    "        head = MobileNetV2Head(input_channel=backbone.out_channels, num_classes=config.num_classes)\n",
    "        self.mask_model = mobilenet_v2(backbone, head)\n",
    "        load_checkpoint(model_path, self.mask_model)\n",
    "    \n",
    "    def to_small_square(self, startX, startY, endX, endY):\n",
    "        w = endX - startX\n",
    "        h = endY - startY\n",
    "        l = min(w, h)\n",
    "        \n",
    "        startX = int(startX + (w - l) / 2)\n",
    "        endX = startX + l\n",
    "        startY = int(startY + (h - l) / 2)\n",
    "        endY = startY + l\n",
    "        return startX, startY, endX, endY\n",
    "    \n",
    "    def recognize(self, image):\n",
    "        # 人脸检测\n",
    "        detections = self.face_det.detect(image)\n",
    "        h, w, c = image.shape\n",
    "        predict_labels = []\n",
    "        for i in range(0, detections.shape[2]):\n",
    "            confidence = detections[0, 0, i, 2]\n",
    "            if confidence > self.face_det.threshold:\n",
    "                box = detections[0, 0, i, 3:7] * np.array([w, h, w, h])\n",
    "                startX, startY, endX, endY = box.astype(\"int\")\n",
    "                # 截取图像\n",
    "                startX, startY, endX, endY = self.to_small_square(startX, startY, endX, endY)\n",
    "                crop_img = image[startY:endY, startX:endX]\n",
    "                # 图像预处理, mask_model accept image with RGB\n",
    "                resized = cv2.resize(crop_img, (config.image_height, config.image_width))\n",
    "                img_tensor = image_process(cv2.cvtColor(crop_img, cv2.COLOR_BGR2RGB))\n",
    "                # 预测\n",
    "                logits = self.mask_model(img_tensor)\n",
    "                predict_label = np.argmax(logits.asnumpy(), axis=1)[0]\n",
    "                predict_labels.append(predict_label)\n",
    "                # 画图\n",
    "                y = startY - 10 if startY - 10 > 10 else startY + 10\n",
    "                cv2.rectangle(image, (startX, startY), (endX, endY),\n",
    "                              (0, 255, 0), 1)\n",
    "                cv2.putText(image, self.class_names[predict_label], (startX, y),\n",
    "                            cv2.FONT_HERSHEY_SIMPLEX, 0.45, (0, 255, 0), 2)\n",
    "        return image, len(predict_labels), predict_labels.count(0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "img = cv2.imread(\"./test1.jpg\")\n",
    "img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)\n",
    "\n",
    "detect = MaskRec(model_path)\n",
    "img, all_num, mask_num = detect.recognize(img)\n",
    "\n",
    "# 展示图片口罩识别结果\n",
    "fig = plt.figure(figsize=(8, 8))\n",
    "ax1 = fig.add_subplot(111)\n",
    "ax1.set_xticks([])\n",
    "ax1.set_yticks([])\n",
    "ax1.set_title('test_mask')\n",
    "ax1.imshow(img)\n",
    "print(\"图中的人数有：\" + str(all_num) + \"个\")\n",
    "print(\"戴口罩的人数有：\" + str(mask_num) + \"个\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "select": true
   },
   "source": [
    "## 4 作业\n",
    "**作业要求及注意事项**：    \n",
    "\n",
    "1.使用上述学到的方法，训练自己的口罩识别模型，尽可能提高准确度。将训练好的模型保存在 results 文件夹下。             \n",
    "2.点击左侧栏提交作业后点击【生成文件】则需要勾选与预测 predict() 函数的 cell相关的其它cell ，并将其转化成为 main.py 文件。                       \n",
    "3.请导入必要的包和第三方库以及该模型所依赖的 py 文件 (包括此文件中曾经导入过的)。             \n",
    "4.请加载你认为训练最佳的模型，即请按要求填写模型路径。              \n",
    "5.predict() 函数的输入输出及函数名称请不要改动。\n",
    "\n",
    "\n",
    "===========================================  **模型预测代码答题区域**  ===========================================  \n",
    "在下方的代码块中编写 **模型预测** 部分的代码，请勿在别的位置作答"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "select": true
   },
   "outputs": [],
   "source": [
    "# -------------------------- 请加载您最满意的模型 -----------------------------\n",
    "# 加载模型(请加载你认为的最佳模型)\n",
    "# 加载模型,加载请注意 model_path 是相对路径, 与当前文件同级。\n",
    "# 如果你的模型是在 results 文件夹下的 mobilenetv2_mask_1-1_39.ckpt 模型，则 model_path = 'results/mobilenetv2_mask-1_39.ckpt'\n",
    "model_path = None\n",
    "# ---------------------------------------------------------------------------\n",
    "\n",
    "def predict(img):\n",
    "    \"\"\"\n",
    "    加载模型和模型预测\n",
    "    :param img: cv2.imread 图像\n",
    "    :return: 预测的图片中的总人数、其中佩戴口罩的人数\n",
    "    \"\"\"\n",
    "    # -------------------------- 实现模型预测部分的代码 ---------------------------\n",
    "    \n",
    "    detect = MaskRec(model_path)\n",
    "    img, all_num, mask_num = detect.recognize(img)\n",
    "    \n",
    "    # -------------------------------------------------------------------------\n",
    "    \n",
    "    return all_num,mask_num"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 输入图片路径和名称\n",
    "img = cv2.imread(\"test1.jpg\")\n",
    "img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)\n",
    "all_num,mask_num = predict(img)\n",
    "# 打印预测该张图片中总人数以及戴口罩的人数\n",
    "print(all_num, mask_num)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}