From 36aa83c92b27d7b88773c9f4bf2cb70a7c2c580c Mon Sep 17 00:00:00 2001 From: ya Date: Tue, 31 Jul 2018 09:17:31 +0800 Subject: [PATCH] Group 4 submits task1 task2 and additional translation, fix #66, fix #22, fix #4 --- blog-cn/pouch_with_kata.md | 83 ++++++ ...20\350\241\214\347\216\257\345\242\203.md" | 96 +++++++ ...47\277\273\350\257\221pouch_with_lxcfs.md" | 93 ++++++ blog-en/PouchContainer-tutorial.md | 95 +++++++ ...50\351\207\217\345\256\236\350\267\265.md" | 268 ++++++++++++++++++ 5 files changed, 635 insertions(+) create mode 100644 blog-cn/pouch_with_kata.md create mode 100644 "blog-cn/\345\237\272\344\272\216Ubuntu\346\220\255\345\273\272PouchContainer\350\277\220\350\241\214\347\216\257\345\242\203.md" create mode 100644 "blog-cn/\347\277\273\350\257\221pouch_with_lxcfs.md" create mode 100644 blog-en/PouchContainer-tutorial.md create mode 100644 "blog-en/\347\277\273\350\257\221_PouchContainer \345\267\245\347\250\213\350\264\250\351\207\217\345\256\236\350\267\265.md" diff --git a/blog-cn/pouch_with_kata.md b/blog-cn/pouch_with_kata.md new file mode 100644 index 0000000..c5aa83f --- /dev/null +++ b/blog-cn/pouch_with_kata.md @@ -0,0 +1,83 @@ + + +# PouchContainer with kata + +## 介绍 +Kata Container整合了Intel® Clear Containers以及Hyper runV的技术特点，在保证containers运行速度的同时也保证了虚拟机的安全性，其核心技术与runV相同。有关虚拟机container的详细信息，可以参阅 [runV doc](https://github.com/alibaba/pouch/blob/master/docs/features/pouch_with_runV.md). + +## 安装准备工作 +由于kata不提供安装选项，所以我们从[clear container project](https://github.com/clearcontainers) 上总结了一些安装方法。如果需要了解细节，可以参阅[kata-containers](https://github.com/kata-containers/community#users)。 + + +### 安装步骤 + +1. 安装qemu + +在运行[QEMU](https://www.qemu.org)之前，我们需要先运行VM。通过执行以下代码，我们可以很容易地安装QEMU相关的工具包。 + +如果宿主\物理机是 Ubuntu OS 系统，执行: + +``` +sudo apt-get install -y qemu qemu-kvm +``` + +如果宿主\物理机是 Red Hat series OS 系统，执行: + +``` +sudo yum install -y qemu qemu-kvm +``` + +2. 安装guest kernel以及guest image + +[kata-containers/osbuilder](https://github.com/kata-containers/osbuilder) 提供了一个工具用于创建guest image,详情可参考[detail steps](https://github.com/kata-containers/osbuilder#usage)。由于这个工具不能用于创建guest kernel，创建guest kernel的详细步骤可参考[clearcontainers/osbuilder](https://github.com/clearcontainers/osbuilder#build-guest-kernel). + +3. 安装 kata-runtime + +在这一部分，我们需要安装三个可执行文件：[kata-runtime](https://github.com/kata-containers/runtime), [kata-proxy](https://github.com/kata-containers/proxy) 以及 [kata-shim](https://github.com/kata-containers/shim)。 在一个运行的kata container中，kata-runtime会调用kata-proxy 以及 kata-shim。从源码中获取这些可执行文件其实十分简单，只需要先从github中clone下代码，然后输入make命令。下述代码以获取kata runtime为例： + +```shell +git clone https://github.com/kata-containers/runtime.git +cd runtime +make +``` + +### 配置 kata runtime +Kata runtime从配置文件中直接读取配置参数，配置文件的默认路径为 +`/etc/kata-containers/configuration.toml`. +生成默认配置文件的命令为: + +```shell +git clone https://github.com/kata-containers/runtime.git +cd runtime +make +``` + +文件会生成在 `cli/config/configuration.toml`，将这个文件复制到默认路径中即可。 + +```shell +cp cli/config/configuration.toml /etc/kata-containers/configuration.toml +``` + +需要注意的是，你可能需要修改这个文件，确保所有可执行文件在系统中的路径都正确。 + +### 启动kata container + +当上述所有步骤全部完成, 你就可以使用kata container了。 + +```shell +$ pouch run -d --runtime=kata-runtime 8ac48589692a top +00d1f38250fc76b5e66e7fa05a41d342d1b48202d24e2dbf06b20a113b2a008c + +$ pouch ps +Name ID Status Created Image Runtime +00d1f3 00d1f3 Up 5 seconds 7 seconds ago docker.io/library/busybox:latest kata-runtime +``` + +进入kata container. + +```shell +$ pouch exec -it 00d1f3 sh +/ # uname -r +4.9.47-77.container +``` +link: https://github.com/alibaba/pouch/blob/master/docs/features/pouch_with_kata.md \ No newline at end of file diff --git "a/blog-cn/\345\237\272\344\272\216Ubuntu\346\220\255\345\273\272PouchContainer\350\277\220\350\241\214\347\216\257\345\242\203.md" "b/blog-cn/\345\237\272\344\272\216Ubuntu\346\220\255\345\273\272PouchContainer\350\277\220\350\241\214\347\216\257\345\242\203.md" new file mode 100644 index 0000000..ba45fc2 --- /dev/null +++ "b/blog-cn/\345\237\272\344\272\216Ubuntu\346\220\255\345\273\272PouchContainer\350\277\220\350\241\214\347\216\257\345\242\203.md" @@ -0,0 +1,96 @@ +## PouchContainer简介 + +PouchContainer是阿里巴巴集团开源的高效、企业级容器引擎技术，拥有隔离性强、可移植性高、资源占用少等特点。可以帮助企业快速实现存量业务容器化，同时提高超大规模下数据中心的物理资源利用率。 + +本文将给大家介绍PouchContainer的搭建和上手过程。 + + + +## 环境准备 + +- VirtualBox + +- 包含pouch的Ubuntu镜像 + + + +## 搭建过程 + +1. 打开安装好的VirtualBox，首先点击标题栏的New按钮，新建一个操作系统，name可以自定义，type选择Linux，Version选择Red Hat（64-bit)。 + + ![](http://oyrpkn4bk.bkt.clouddn.com/%E5%B1%8F%E5%B9%95%E5%BF%AB%E7%85%A7%202018-07-30%20%E4%B8%8B%E5%8D%885.33.31.png) + + + +2. 点击继续按钮进入内存的选择页面。内存选择1024MB，当然也可以根据需要加大内存。 + + ![](http://oyrpkn4bk.bkt.clouddn.com/%E5%B1%8F%E5%B9%95%E5%BF%AB%E7%85%A7%202018-07-30%20%E4%B8%8B%E5%8D%885.25.41.png) + + + +3. 点击继续按钮进入硬盘选择页面，选中"使用已有的虚拟硬盘文件"，选择包含pouch的Ubuntu镜像`ubuntuPouch.vdi`，点击创建，在VirtualBox左边栏可以看到新建的虚拟机。 + + ![](http://oyrpkn4bk.bkt.clouddn.com/%E5%B1%8F%E5%B9%95%E5%BF%AB%E7%85%A7%202018-07-30%20%E4%B8%8B%E5%8D%885.34.02.png) + + + +4. 启动虚拟机，等待进入登录阶段，用户名`pouch`，密码`123456` + +5. 切换到root用户下：`sudo su root` + +6. 检查网络是否通畅：`ping www.alibaba-inc.com` + +7. 启动pouch服务：`systemctl start pouch` + +8. 输入`pouch`命令判断pouch是否启动 + + ![](http://oyrpkn4bk.bkt.clouddn.com/%E5%B1%8F%E5%B9%95%E5%BF%AB%E7%85%A7%202018-07-30%20%E4%B8%8B%E5%8D%885.43.46.png) + + + +9. 执行`pouch run -t -d busybox sh`启动名为busybox的容器 + +10. 使用`pouch ps -a`可以查看所有创建的容器 + + ![](http://oyrpkn4bk.bkt.clouddn.com/%E5%B1%8F%E5%B9%95%E5%BF%AB%E7%85%A7%202018-07-30%20%E4%B8%8B%E5%8D%886.05.59.png) + + + +11. 执行`pouch exec -it {id} sh`进入容器 + +![](http://oyrpkn4bk.bkt.clouddn.com/%E5%B1%8F%E5%B9%95%E5%BF%AB%E7%85%A7%202018-07-30%20%E4%B8%8B%E5%8D%886.06.23.png) + +到此为止，PouchContainer在Ubuntu虚拟机上的搭建过程已经完成，Centos与此类似。 + +为了本地开发方便，我们将继续配置虚拟机的ssh连接。 + + + +## SSH连接虚拟机 + +1. 在VirtualBox界面点击设置-->网络，连接方式选择网络地址转换 + +2. 在该界面下点击高级，端口转换，配置端口映射。主机端口可以任选，子系统端口必须选择22。 + + + + ![](http://oyrpkn4bk.bkt.clouddn.com/%E5%B1%8F%E5%B9%95%E5%BF%AB%E7%85%A7%202018-07-30%20%E4%B8%8B%E5%8D%885.56.06.png) + + + +3. 配置完成，打开本地terminal，使用命令`ssh -p 2233 pouch@127.0.0.1`连接虚拟机。 + + ![](http://oyrpkn4bk.bkt.clouddn.com/%E5%B1%8F%E5%B9%95%E5%BF%AB%E7%85%A7%202018-07-30%20%E4%B8%8B%E5%8D%885.58.24.png) + + + +## 总结 + +通过上面的教程，我们可以很轻松的在非Linux电脑上体验PouchContainer。 + + + + + + + diff --git "a/blog-cn/\347\277\273\350\257\221pouch_with_lxcfs.md" "b/blog-cn/\347\277\273\350\257\221pouch_with_lxcfs.md" new file mode 100644 index 0000000..56b9437 --- /dev/null +++ "b/blog-cn/\347\277\273\350\257\221pouch_with_lxcfs.md" @@ -0,0 +1,93 @@ +# PouchContainer与LXCFS + +容器技术为不同的隔离环境提供传统的虚拟化技术,如VMware，KVM。普遍的Linux容器以削弱隔离为代价换来了更快的容器打包以及设定速度。容器的资源视图是Linux容器面临的最广为人知的隔离问题之一。 + +容器为用户提供了一种限制运行容器的资源使用的大致方法，资源包括内存，cpu, blkio等。但容器中的步骤无法使之获取额外的相应资源。但是，如果容器内的步骤用命令 `free`, `cat /proc/meminfo`, `cat /proc/cpuinfo`, `cat /proc/uptime`查出资源上限，这些命令会得到对宿主机而言正确的信息，而不是容器本身。 + +举个例子来说，如果我们创建一个200MB主机内存，总内存2GB的容器，我们可以发现通过命令`free`得出的资源上限是不正确的，结果是主机的总内存： + +``` shell +$ pouch run -m 200m registry.hub.docker.com/library/ubuntu:16.04 free -h + total used free shared buff/cache available +Mem: 2.0G 103M 1.2G 3.3M 684M 1.7G +Swap: 2.0G 0B 2.0G +``` + +## 资源隔离情景 + +由于缺乏资源隔离，一些应用可能会在容器提供的环境中运行异常。应用可能发觉它的运行时间与在正常物理机或虚拟机上的运行时间不同。以下是一些由于隔离匮乏而影响应用运行的例子: + +> 对于很多基于JVM的Java应用，应用的启动脚本多依赖于系统资源对于JVM分配堆栈大小的能力。因此，一个在2GB内存主机上创建出的200MB内存的容器，会认为它能够使用2GB的内存，所以启动脚本会让Java运行时根据与200MB上限相差甚远的2GB内存分配堆栈大小。因此应用程序当然会启动失败。对于Java应用，一些库会根据它们的资源视图分配堆栈大小，这也会暴露出潜在的安全问题。 + +除了内存资源视图的问题，还有弱CPU资源视图隔离的问题。 + +> 大多数中间软件会根据它认为的处理器信息去设置默认的线程数量。用户会配置在cgroup文件生效的容器cpu是不无道理的事情。但是，容器中的步骤通过使用`/proc/cpuinfo` 总是得到整个内核数量，并一定会导致不稳定。 + +资源隔离会影响容器里系统级的应用程序。 + +> 容器还可以用来打包系统级的应用程序，并且系统级应用程序会时常需要通过虚拟文件系统`/proc`获取系统信息。如果使用的是主机的正常运行时间，而不是容器的正常运行时间，系统级应用程序会失去控制或者不按预期运行。`cpuinfo` 和 `meminfo`，以及一些其它系统资源视图也是这些应用程序需要考虑的方面。 + +## 什么是LXCFS + +[LXCFS](https://github.com/lxc/lxcfs)是一个小型的 [FUSE文件系统](https://en.wikipedia.org/wiki/Filesystem_in_Userspace) ，开发它的意图是让Linux容器更像一个虚拟机。最开始它是LXC的副产品，但其实它可以在任何运行环境被使用。 LXCFS 兼容Linux kernel 2.6+。LXCFS可以处理 `procfs`里重要文档提供的信息，包括: + +* /proc/cpuinfo +* /proc/diskstats +* /proc/meminfo +* /proc/stat +* /proc/swaps +* /proc/uptime + +PouchContainer在之前的版本中一直支持LXCFS，并且一直很稳定。 换句话来说，如果用户启动LXCFS,LXCFS在主机上会运行守护进程。通常来说，当创建了一个有资源上限的容器，一些映射这个容器的虚拟文件会在cgroup文件系统中被创建。然后，LXCFS会动态地读取这些文件里的数值，比如`memory.limit_in_bytes`，并在主机上产生一个新的分支虚拟文件（如 `/var/lib/lxc/lxcfs/proc/meminfo`），然后把这个文件绑定在容器上。最后，容器里的步骤会通过类似`/proc/meminfo`的命令读取文件，获取到真正的资源视图。 + +下面是LXCFS和容器的结构： + +![pouch_with_lxcfs](../static_files/pouch_with_lxcfs.png) + +## 开始使用 + +利用LXCFS，用户达到资源隔离是很容易的事情。事实上，如果LXCFS软件在$PATH中不存在的话，它会自动与PouchContainer一起安装在主机上。 + +在体验LXCFS确保的资源视图隔离之前, 用户需要确认LXCFS模式在pouchd中是被允许的。如果LXCFS模式还没有被设置好，用户需要中止pouchd，再通过 `pouchd --enable-lxcfs`命令开启pouchd。只有在pouchd开启LXCFS模式，用户才能在容器中使用LXCFS功能。 + +当pouchd中开启着LXCFS模式时，pouchd有额外能力可以创建有隔离的资源视图的容器。除此之外，pouchd还可以创建没有资源视图隔离的普通容器。 + +最后，命令`pouch run`中的 `--enableLxcfs` 标志是在已经允许LXCFS模式的pouchd中使得LXCFS可以创建容器的唯一方法。在这里我们在2GB内存的主机上创建一个内存为200MB的容器。 + +### 启动条件 + +如果是在Centos中，首先要确保你的lxcfs服务在运行。 如果是其他的系统可能会使用不同的启动方式。 + + +``` +$ systemctl start lxcfs +$ ps -aux|grep lxcfs +root 1465765 0.0 0.0 95368 1844 ? Ssl 11:55 0:00 /usr/bin/lxcfs /var/lib/lxcfs/ +root 1465971 0.0 0.0 112736 2408 pts/0 S+ 11:55 0:00 grep --color=auto lxcfs +``` + +通过更改--enable-lxcfs标记启动 pouchd lxcfs + +``` +$ cat /usr/lib/systemd/system/pouch.service +[Unit] +Description=pouch + +[Service] +ExecStart=/usr/local/bin/pouchd --enable-lxcfs +... + +$ systemctl daemon-reload && systemctl restart pouch +``` + +``` shell +$ pouch run -m 200m --enableLxcfs registry.hub.docker.com/library/ubuntu:16.04 free -h + total used free shared buff/cache available +Mem: 200M 876K 199M 3.3M 12K 199M +Swap: 2.0G 0B 2.0G +``` + +我们可以发现返回的总内存大小就是容器的内存上限。 +在执行上述命令过后，我们同时也可以发现在容器中的进程获取的资源也是它实际的资源上限。换句话说，在容器中的应用会更加的安全。这也是PouchContainer重要的特性之一。 + +link: https://github.com/alibaba/pouch/blob/master/docs/features/pouch_with_lxcfs.md diff --git a/blog-en/PouchContainer-tutorial.md b/blog-en/PouchContainer-tutorial.md new file mode 100644 index 0000000..03f04ae --- /dev/null +++ b/blog-en/PouchContainer-tutorial.md @@ -0,0 +1,95 @@ +## Introduction of PouchContainer +PouchContainer is an open-source container engine technology in enterprise level created by Alibaba. It can provide strong isolation with high protability and occupy limited resource at the same time. PouchContainer can not only help enterprises achieve containerization of their stock business,but also improve the physical resource usage of data center in large scale. + +In this passage, the installation and a tutorial of PouchContainer will be introduced. + + +## Prepare Environment + +- VirtualBox + +- Ubuntu image with pouch + + + +## Building steps + +1. Open the already installed VirtualBox. First, click the button NEW on the menu to create a new operation system. Configure as below: name can be defined freely; choose type as Linux; choose Version as Red Hat(64-bit). + + ![](http://oyrpkn4bk.bkt.clouddn.com/%E5%B1%8F%E5%B9%95%E5%BF%AB%E7%85%A7%202018-07-30%20%E4%B8%8B%E5%8D%885.33.31.png) + + + +2. Click NEXT, entering the page with RAM otpions. Choose RAM as 1024 MB. You can choose larger RAM regarding to your need. + + ![](http://oyrpkn4bk.bkt.clouddn.com/%E5%B1%8F%E5%B9%95%E5%BF%AB%E7%85%A7%202018-07-30%20%E4%B8%8B%E5%8D%885.25.41.png) + + + +3. Click NEXT, engtering the page with hard disk options. Choose 'Using existing virtual hard disk file', then find the Ubuntu Image containing pouch `ubuntuPouch.vdi`. Click create. Finally a new created virtual machine is showed on the left bar. + + ![](http://oyrpkn4bk.bkt.clouddn.com/%E5%B1%8F%E5%B9%95%E5%BF%AB%E7%85%A7%202018-07-30%20%E4%B8%8B%E5%8D%885.34.02.png) + + + +4. Activate virtual machine, wating for the log in message. Enter the user name `pouch` and the code`123456` + +5. Switch to the root user：`sudo su root` + +6. Check Internet：`ping www.alibaba-inc.com` + +7. Activate pouch：`systemctl start pouch` + +8. Type `pouch` order to see whether it has been activated + + ![](http://oyrpkn4bk.bkt.clouddn.com/%E5%B1%8F%E5%B9%95%E5%BF%AB%E7%85%A7%202018-07-30%20%E4%B8%8B%E5%8D%885.43.46.png) + + + +9. Execute `pouch run -t -d busybox sh` to start a container named as busybox + +10. Execute `pouch ps -a` to see all of the containers that has been created + + ![](http://oyrpkn4bk.bkt.clouddn.com/%E5%B1%8F%E5%B9%95%E5%BF%AB%E7%85%A7%202018-07-30%20%E4%B8%8B%E5%8D%886.05.59.png) + + + +11. Execute `pouch exec -it {id} sh` to enter the container where {id} is the first six characters of the result got from the last command. + +![](http://oyrpkn4bk.bkt.clouddn.com/%E5%B1%8F%E5%B9%95%E5%BF%AB%E7%85%A7%202018-07-30%20%E4%B8%8B%E5%8D%886.06.23.png) + +So far, PouchContainer has been installed in the Ubuntu virtual machine. You can follow the same instruction to set up Centos, since the installation process of Centos is similar. + +For the conveinence of local program development, we will show how to configure the ssh connection of the virtual machine. + + + + +## Connecting virtual machine via SSH + +1. In the interface of VirtualBox, click Settings -> Network. Check ‘Enable Network Adapter’. Choose 'NAT' in 'Attached to' options. + +2. Click Advance -> Port Forwarding. Then you can configure the ip address and port number of host and guest. The Host Port can be chosen randomly while the Guest Port must be 22. + + + + ![](http://oyrpkn4bk.bkt.clouddn.com/%E5%B1%8F%E5%B9%95%E5%BF%AB%E7%85%A7%202018-07-30%20%E4%B8%8B%E5%8D%885.56.06.png) + + + +3. After configuration，open local terminal，type command `ssh -p 2233 pouch@127.0.0.1` to connect to the virtual machine. + + ![](http://oyrpkn4bk.bkt.clouddn.com/%E5%B1%8F%E5%B9%95%E5%BF%AB%E7%85%A7%202018-07-30%20%E4%B8%8B%E5%8D%885.58.24.png) + + + +## Conclusion + +Basing on the tutorial showed above, we could use PouchContainer easily in any non-Linux computer. + + + + + + + diff --git "a/blog-en/\347\277\273\350\257\221_PouchContainer \345\267\245\347\250\213\350\264\250\351\207\217\345\256\236\350\267\265.md" "b/blog-en/\347\277\273\350\257\221_PouchContainer \345\267\245\347\250\213\350\264\250\351\207\217\345\256\236\350\267\265.md" new file mode 100644 index 0000000..3ac79c8 --- /dev/null +++ "b/blog-en/\347\277\273\350\257\221_PouchContainer \345\267\245\347\250\213\350\264\250\351\207\217\345\256\236\350\267\265.md" @@ -0,0 +1,268 @@ +# 0. Preface + +With the development of [PouchContainer](https://github.com/alibaba/pouch), the functions are getting abundant and the project is getting vast, therefore many outside developer are attracted to join the developing of the project. Since the coding style of different person can be diverse while the unified coding style is the prerequisite of the code maintenance, the reviewer has the responsibility of not only checking the correctness and the performance of the code but also unifying the style of the code. What's more, the coverage and the stability of the test cases are also important points requiring more attention. You can simply imagine how the project lacking the regression test case ensure that the update of the code won't affect the current function every time. + +In this passage, the coding style regulation exercise and the unit test case instance of golang will be explained. + +# 1. Unified coding style regulation +PouchContainer is a project developed in golang where shell script is used to complete some automated operation, such as compilation and packaging. Besides golang and shell script, PouchContainer also contains loads of Markdown documents which are the entrance for users getting to know PouchContainer so that the regulated typesetting and the corrected spelling are also important aspects that need more focus. In the following chapter, the coding style regulation and the application scenario of PouchContainer will be introduced. + +## 1.1 Golinter - the unified coding style +The grammar of golong is designed simply and the community is full of [CodeReview](https://github.com/golang/go/wiki/CodeReviewComments) instruction so that most of the golang projects have the similar coding style and few of them will get stuck with the __religion__ fight. Basing on the community, PouchContainer also defines some specific rules to regulate developer to ensure the readability of their code. The details of the rules can be found [there](https://github.com/alibaba/pouch/blob/master/docs/contributions/code_styles.md#additional-style-rules). + +But it's hard to ensure the unification of the project coding style simply relying on the written agreement. Therefore, like other languages, golang has official fundamental tool chain such as [golint](https://github.com/golang/lint), [gofmt](https://golang.org/cmd/gofmt)，[goimports](https://github.com/golang/tools/blob/master/cmd/goimports/doc.go) and [go vet](https://golang.org/cmd/vet),etc. Those tools can be used to check and unify the coding style before compilation to make the automated coding review possible. For now, PouchContainer will run above checking tools in CircleCI before __every__ pull request raised by developer. If error is detected by those tools, the code reviewer has right to __refuse__ examination or even refuse to merge those code. + +Besides the provided official toolkit, we can use the code chekcing toolkit provided by the third party in the open source community as well, such as [errcheck](https://github.com/kisielk/errcheck) can check whether developer has eliminated all of the errors returned by functions. But those tools don't have the same output format, which make it difficult to merge the outputs produced by different tools. The good news is that someone in the community achieves the unified interface of this layer, which is [gometalinter](https://github.com/alecthomas/gometalinter). It can integrate various code checking toolkit, the recommended combinations are: + +* [golint](https://github.com/golang/lint) - Google's (mostly stylistic) linter. +* [gofmt -s](https://golang.org/cmd/gofmt/) - Checks if the code is properly formatted and could not be further simplified. +* [goimports](https://godoc.org/golang.org/x/tools/cmd/goimports) - Checks missing or unreferenced package imports. +* [go vet](https://golang.org/cmd/vet/) - Reports potential errors that otherwise compile. +* [varcheck](https://github.com/opennota/check) - Find unused global variables and constants. +* [structcheck](https://github.com/opennota/check) - Find unused struct fields +* [errcheck](https://github.com/kisielk/errcheck) - Check that error return values are used. +* [misspell](https://github.com/client9/misspell) - Finds commonly misspelled English words. + +Every project can order specific gometalinter combination basing on their needs. + +## 1.2 Shellcheck - reduce potential problem in shell script + +Although shell script has powerful functions, it still needs grammar check to avoid some potential and unpredictable error, such as the definition of the unused variable, which won't affect the normal function of the script but its existence will become the burden of the maintenance. + +```powershell +#!/usr/bin/env bash + +pouch_version=0.5.x + +dosomething() { + echo "do something" +} + +dosomething +``` + +PouchContainer will use [shellcheck](https://github.com/koalaman/shellcheck) to check the shell scripts in the current project. Taking the above code as an example, shellcheck will raise the warning that there is unused variable. This toll can detect the potential problems in shell script to reduce the error probability in running. + +```plain +In test.sh line 3: +pouch_version=0.5.x +^-- SC2034: pouch_version appears unused. Verify it or export it. +``` + +The current lasting integrating mission of PouchContainer will scan every `.sh` script in the project and use shellcheck to examine them one by one. The details can be found [there](https://github.com/alibaba/pouch/blob/master/.circleci/config.yml#L21-L24)。 + + +> NOTE: When the examination of shellcheck is too strict, the project can use comment to avoid it, or shutdown some specific examination in the project. The detailed checking rule can be found [there](https://github.com/koalaman/shellcheck/wiki)。 + +## 1.3 Markdownlint - Unified document format + +As an open-source project, documents have the same significance as code to PouchContainer. Because documents are the best methods for users get to know PouchContainer. Documents are written in markdown so that its formatting and spelling are mainly focused. + +Same as the code, there will be error missing merely relying on written aggrement. Therefore, PouchContainer applies [markdownlint](https://github.com/markdownlint/markdownlint) and [misspell](https://github.com/client9/misspell) to check document format and spelling. These examinations have the same reputation as `golint` which will be called in every Pull Request. Once an error is raised, the code reviewer has right to __refuse __ inspect or merge those code. + +The current lasting integration mission in PouchContainer will examine the markdown document formatting and document spelling. The specific configuration is showed [there](https://github.com/alibaba/pouch/blob/master/.circleci/config.yml#L13-L20). + +> NOTE: +> When the examination of mardownlint is too strict, the project can use comment to avoid it, or shutdown some specific examination in the project. The detailed checking rule can be found [there](https://github.com/markdownlint/markdownlint/blob/master/docs/RULES.md)。 + +## 1.4 Summary + +The above content is all about the coding style problem. PouchContainer can check coding grammar automatically which is integrated in every time code review to help code reviewer find potential problems. + +# 2. How to make golang unit testing + +Unit testing is used to guarantee the correctness of single module. In the testing pyramid, the broader the coverage of unit testing gets, the more general the covered functions become and the less testing cost generated by integration testing and end-to-end testing. In the complicated system, the longer the dealing chain of a mission is, the higher the cost of problem targeting gets, specially the problem caused by small modules. In the following chapters, the summary of how to develop golang unit testing will be introduced. + +## 2.1 Table-Driven Test - DRY +We can understand unit testing simply, that is giving a fixed input to some function and determine whether the output is expected. When the tested functions have lots of input scenarios, we could use Table-Driven to organize our test case, which is showed below. Table-Driven applies array to organize test case and uses loop to test the correctness of the function. + +```go +// from https://golang.org/doc/code.html#Testing +package stringutil + +import "testing" + +func TestReverse(t *testing.T) { + cases := []struct { + in, want string + }{ + {"Hello, world", "dlrow ,olleH"}, + {"Hello, 世界", "界世 ,olleH"}, + {"", ""}, + } + for _, c := range cases { + got := Reverse(c.in) + if got != c.want { + t.Errorf("Reverse(%q) == %q, want %q", c.in, got, c.want) + } + } +} +``` +For the convenience of testing and maintenance, we can add some aiding information to describe the current testing. For example, [reference](https://github.com/alibaba/pouch/blob/master/pkg/reference/parse_test.go#L54) wants to test the input of [punycode](https://en.wikipedia.org/wiki/Punycode), if there is no `punycode` like word, the code reviewer or project maintainer might not know the difference between `xn--bcher-kva.tld/redis:3` and `docker.io/library/redis:3` + +```go +{ + name: "Normal", + input: "docker.io/library/nginx:alpine", + expected: taggedReference{ + Named: namedReference{"docker.io/library/nginx"}, + tag: "alpine", + }, + err: nil, +}, { + name: "Punycode", + input: "xn--bcher-kva.tld/redis:3", + expected: taggedReference{ + Named: namedReference{"xn--bcher-kva.tld/redis"}, + tag: "3", + }, + err: nil, +} +``` +But some fucntions have complicated behaviors, one input can't be taken as a completed test case. Take [TestTeeReader](https://github.com/golang/go/blob/release-branch.go1.9/src/io/io_test.go#L284) as an instance. After TeeReader reads hello, world from buffer, data reading is finished. If it tries to read it again, an error end-of-file will be raised. Test case like this needs a separate case to finish with no need to make Table-Drive format on purpose. + + + +Simply speaking, if you want to test some function with lots of code copied, those testing code can be withdrawn and use Table-Drive to organize test case.Don`t Repeat Yourself is the rule we obey. + +> NOTE: The organizing method of Table-Driven is recommended by golang community. The details can be found [there](https://github.com/golang/go/wiki/TableDrivenTests)。 +## 2.2 Mock - Mock External Dependency + + During test procedure, we will always encounter dependency issues. For example, PouchContainer client needs HTTP server but it may over fit for units, and this belongs to integration test. So how can we complete this part of integration test? + +In the world of golang, the realization of interface belongs to [Duck Type](https://en.wikipedia.org/wiki/Duck_typing). As long as the realization corresponds to the definition of the interface, there can be different realization at one interface. If external dependency constrains through interface, then the unit testing will mock these dependent behaviors. Here two common testing scenarios will be discussed. + +### 2.2.1 RoundTripper + +Take PouchContainer client testing as an example. PouchContainer client uses [http.Client](https://golang.org/pkg/net/http/#Client). http.Client [RoundTripper](https://golang.org/pkg/net/http/#RoundTripper) is used to execute HTTP request， and it allows developer to customize the logic of sending HTTP request, and this is the reason why golang can support HTTP 2 contracts perfectly. + +```plain +http.Client -> http.RoundTripper [http.DefaultTransport] +``` + +For PouchContainer client, the focus of testing is whether the target address is correct, whether incoming query is reasonable and whether the result can be returned normally and so on. Thus before testing, developer should have prepared corresponding RoundTripper realization, and this realization is not responsible for the real business logic. Instead it is only used to see if the input meets the expectation or not. + +As the following codes shown, PouchContainer `newMockClient` can accept customized request processing logic. In the example of testing deleting mirror, developer will decide the target address and if the HTTP Method is DELETE in customized logic, and in this way the functional testing can be completed without HTTP Server being enabled. + +```go +// https://github.com/alibaba/pouch/blob/master/client/client_mock_test.go#L12-L22 +type transportFunc func(*http.Request) (*http.Response, error) + +func (transFunc transportFunc) RoundTrip(req *http.Request) (*http.Response, error) { + return transFunc(req) +} + +func newMockClient(handler func(*http.Request) (*http.Response, error)) *http.Client { + return &http.Client{ + Transport: transportFunc(handler), + } +} + +// https://github.com/alibaba/pouch/blob/master/client/image_remove_test.go +func TestImageRemove(t *testing.T) { + expectedURL := "/images/image_id" + + httpClient := newMockClient(func(req *http.Request) (*http.Response, error) { + if !strings.HasPrefix(req.URL.Path, expectedURL) { + return nil, fmt.Errorf("expected URL '%s', got '%s'", expectedURL, req.URL) + } + if req.Method != "DELETE" { + return nil, fmt.Errorf("expected DELETE method, got %s", req.Method) + } + + return &http.Response{ + StatusCode: http.StatusNoContent, + Body: ioutil.NopCloser(bytes.NewReader([]byte(""))), + }, nil + }) + + client := &APIClient{ + HTTPCli: httpClient, + } + + err := client.ImageRemove(context.Background(), "image_id", false) + if err != nil { + t.Fatal(err) + } +} +``` + +### 2.2.2 MockImageManager + +For dependency between internal packages (such as PouchContainer Image API Bridge is dependent on PouchContainer Daemon ImageManager), the dependent behavior is decided by interface. If we want to test the logic of Image Bridge, we do not need to enable containerd. Instead we just need to realize corresponding Daemon ImageManager, just like RoundTripper. + +```go +// https://github.com/alibaba/pouch/blob/master/apis/server/image_bridge_test.go +type mockImgePull struct { + mgr.ImageMgr + handler func(ctx context.Context, imageRef string, authConfig *types.AuthConfig, out io.Writer) error +} + +func (m *mockImgePull) PullImage(ctx context.Context, imageRef string, authConfig *types.AuthConfig, out io.Writer) error { + return m.handler(ctx, imageRef, authConfig, out) +} + +func Test_pullImage_without_tag(t *testing.T) { + var s Server + + s.ImageMgr = &mockImgePull{ + ImageMgr: &mgr.ImageManager{}, + handler: func(ctx context.Context, imageRef string, authConfig *types.AuthConfig, out io.Writer) error { + assert.Equal(t, "reg.abc.com/base/os:7.2", imageRef) + return nil + }, + } + req := &http.Request{ + Form: map[string][]string{"fromImage": {"reg.abc.com/base/os:7.2"}}, + Header: map[string][]string{}, + } + s.pullImage(context.Background(), nil, req) +} +``` + +### 2.2.3 summary + +ImageManager and RoundTripper mock in the same way except that the number of functions defined by the interface is different. In general, developer can define a structure in which the method is used as field, as the codes shown below. + +```go +type Do interface { + Add(x int, y int) int + Sub(x int, y int) int +} + +type mockDo struct { + addFunc func(x int, y int) int + subFunc func(x int, y int) int +} + +// Add implements Do.Add function. +type (m *mockDo) Add(x int, y int) int { + return m.addFunc(x, y) +} + +// Sub implements Do.Sub function. +type (m *mockDo) Sub(x int, y int) int { + return m.subFunc(x, y) +} +``` + +When the interface is large and complex, a manual way will bring a burden to the testing, so our community provide tools that can automatically generate mocks for interfaces to release burden of developers, such as [mockery](https://github.com/vektra/mockery). + +## 2.3 others + +Sometimes this relies on third party service, and a typical example would be PouchContainer client. Duck Type that we introduced above can be used to do this testing. Besides, we can also sign up for http.Handler and enable mockHTTPServer to complete the request of processing. But this testing method will over fit, and thus it is suggested that we use it only when Duck Type cannot achieve the goal or we use it during integration testing. + +> NOTE: In golang community some people complete [monkeypatch](https://github.com/bouk/monkey) by adjusting binary codes. This tool is not suggested, and developers are suggested to write codes which could be tested. + +## 2.4 summary + +PouchContainer integrate unit testing case to code reviewing process, and reviewers can check the code performance of test case at any time. + +# 3. summary + +During code reviewing process, we should check codes, and run unit testing and integration testing to help reviewers to make the right decision. Now, PouchContainer mainly checks code and runs test through TravisCI/CircleCI 和 [pouchrobot](https://github.com/pouchcontainer/pouchrobot). + + +link: https://github.com/pouchcontainer/blog/blob/master/blog-cn/PouchContainer%20%E5%B7%A5%E7%A8%8B%E8%B4%A8%E9%87%8F%E5%AE%9E%E8%B7%B5.md + +