c45_Bagging_Pasting_OOB_随机森林.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* **每个子模型不需要太高的准确率, 假设每个子模型只有51%的准确率**\n",
    "\n",
    "**1个子模型, 整体准确率: 51%**\n",
    "\n",
    "**3个子模型, 整体准确率: $0.51^3+C_3^2*0.51^2*0.49=51.5$%**\n",
    "\n",
    "**500个子模型, 整体准确率: $\\sum_{i=251}^{500}C_{500}^i*0.51^i*0.49^{500-i}=65.5$%**\n",
    "(251开始表示得有一半以上)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "----\n",
    "### 如何创建差异性? 每个子模型只看样本数据的一部分\n",
    "####  放回取样 : Bagging (常用, 统计学中叫 Bootstrap)\n",
    "#### 不放回取样 : Pasting"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX8AAAD8CAYAAACfF6SlAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAIABJREFUeJztnXucHNV153+nWz1SD2Rn9HKERpKlZFkcCEKyHsErrW3AvIwRA5gRJk7sxLbwYptYzgrE2iuE4th67Ab8Yh0Z8wEn2GgwYhgMfAQGEiI52IysF8LW8gpoZsAgxMhI09I8+u4f1dVTXXXvrVuP7q7uPt/PRx9NV1dX3arqPvfc8yQhBBiGYZjGIlXtATAMwzCVh4U/wzBMA8LCn2EYpgFh4c8wDNOAsPBnGIZpQFj4MwzDNCAs/BmGYRoQFv4MwzANCAt/hmGYBmRctQegYsqUKWL27NnVHgbDMExNsXPnzkNCiKl++yVW+M+ePRs9PT3VHgbDMExNQUSvmuzHZh+GYZgGhIU/wzBMA8LCn2EYpgFh4c8wDNOAsPBnGIZpQFj4MwzDNCCJDfVk4qFrVx82bTuA/oEcprdmserC09A+v63aw2IYpsqw8K9junb14aat+5AbHgUA9A3kcNPWfQDAEwDDNDhs9qljNm07UBT8NrnhUWzadqBKI2IYJimw8K9j+gdygbYzDNM4sPCvY6a3ZgNtZximcWDhX8esuvA0ZDPpkm3ZTBqrLjytSiNiGCYpsMO3jrGduhztwzCMGxb+dU77/DYW9gzDeGDhzzAB4LwJpl5g4c8whnDeBFNPsMOXYQzhvAmmnmDNn2E0OM08QrEP500wtQgLfyaRJMG27jbzqOC8CaYWYeHPJI64bOtRJxCZmccN500wtQrb/JnEEYdt3Z5A+grmGnsC6drVZ3wMnTmHALS1ZvHNK85kZy9Tk7DmzySOOGoS6SYQU2E9vTWLPsk521qz2LH6XOOxMEwSYc2fSRxx1CSKYwLh8hhMPROL8CeiO4noTSJ6TvH+h4noCBHtLvxbE8d5mfokDqEbxwTSPr8NVy5oQ5oIAJAmwpULOGOaqQ/i0vzvAnCRzz7/JoSYV/i3LqbzMnVI+/w2fPOKM9HWmg1tW49jAuna1Yf7d/ZhVFhBnqNC4P6dfYH8BgyTVGKx+Qshniai2XEci2GA6DWJ4ihqF4ffgGGSSiUdvh8goj0A+gH8DyHEfvcORLQCwAoAmDVrVgWHxtQjfhOIXygoN8Nh6plKCf9fA3ivEOIoEX0UQBeAU907CSE2A9gMAAsXLlQlVDI1ShISt5xj8cslUEX7cFIXUw9UJNpHCPF7IcTRwt+PAMgQ0ZRKnJtJBnHE3Xft6sOS9U9izuqHsWT9k5Fs7ya5BBztw9QzFdH8iWgagN8JIQQRLYY16bxdiXMz4YhbS49qP+/a1YdVP92D4VFrQdg3kMOqn+4BEK6ipolJh5vhMPVMLMKfiH4C4MMAphBRL4CbAWQAQAjxfQAfB/DfiWgEQA7A1UIINusklHKULo5qP7/lof1FwW8zPCpwy0P7Q43J1KTDzXCYeiWuaJ9P+Lz/XQDfjeNcTPkpR5RLVPv5O4PDgbb7serC0zxF29ikwzQSnOHLeChHlEvS7Odx5BLUInH6TZjahmv7MB7KEeUS1X7ems1gIOfV8luzmUhjqndh74Q7kTFOWPgzHsplEokibNcuOwOr7tuD4fyY3T+TIqxddkakMTUSnLTGOGHhX0NUKk4+iVEufmOKem+SlINQLjhpjXFCSQ26Wbhwoejp6an2MBKDrKtUNpOuazu1qUCW3RsCIGDZ8v0EeaPc2yXrn+QS1Q0AEe0UQiz02481/zITl0bZaEt2P/u0876miIrF12zsVyZ27Ua5txzhxDhh4V9G4nSw1cqSvRKTHYCS++oW/G78BHmt3NuoJNGcx1QPFv5lJE6NshbqzFRqsjPprWt6PKA27m1cNFqEE6OG4/zLSJwaZdLi5AFvzPja7v2Re+/atDbLQzhbmzOh7p9OkCfx3jJMuWHNv4zEqVGWc8kexlQj0/JVhBHWKkuOEOr7mi7Y/m1nr42fIGdzCNOIsPAvI3E72MqxZA9rqgliegkz2R2RJHTZ29cuO0MbnRNmMmNzCNNosPAvI7WgUYb1S5hq82EnO92qye++RhHkjRDvzzAAC/+yk3SN0sQvIROIKuE8sTmD5qZxkYWn36rJfV9t/0OU83L5A6aRYOHf4Pj5JVQC8coFbbh/Z59HON986RmxCMogq6awQts9qR07MdIQ8f4MA3CGb8Pjl92qywpddeFpiTCRhMlclV23jrbWbNWvk2FM4Axfxgg/DVtnFopievGzrQexvasijXR+iSAOa3Kcg01BTL3Awp/R+iVMw1WDmF5MSje431+5ZTd6Xj2Mr7ef6TmWO7RTNUYnpg5r2bHZFMTUA5zkxZTgTtw6531TjRKgTBqim+4re18AuOeZ1zzNRzZtOyAV/ARoo4x0E0OaqNjgRWUUrbfSD0zjwcK/jojapcnWuPsGchCwNO77d/bhygVtvh2vgmQzq/a1Vxiq9wXgmUxUJh8/T5ZuYsgLgVfWX4Idq8/FRE2mMcPUMmz2qRPiCFNUaeRP/fYt35K/QbKZVfsCwNe69qG1OaPszeueGNKSip42uutvn9+Gtd37pd3BnGPWZRozTC3Dmn+dEMTsoiKM49QmSH2cVReeBlIc555nXsNxjSPWPZnoKnrmhkfxt517iisg98roY2ed4jtmXaYxw9QyLPzrhKhF5GzHqQyT8gxBGqK3z29TmmUEgNxwXnkeWzDbgtyPUSFw09Z9+FrXPqVJy9kHeEKm9CehuvZ6rPjJNBYs/OuEqEIqrOPUJkjXrXm3PGY0Jjet2UxJNJCumJyT3PAofvLLg9KV0cN7X8eJkbHJ5p3BYdy0dV9xtcAVP5l6JRbhT0R3EtGbRPSc4n0iom8T0YtEtJeI3h/HeZkxogopnZPVtMKnU6t2ClD3fjI7u5OJzRnptdjN2sPU81eZh94ZHNaay4KsaJiA7O0Ebv1TYG2r9f/ezuoco0GJy+F7F4DvAviR4v2LAZxa+PdnAP5v4X8mJqIWkdOVSe7a1Re4wqcsFj6I0L5yQRue+u1bgRLP7PH6dfYywXmOpNdnqkn2dgJd1wH5giJw5KD1GgDmdpgf46HrgeHc2DEeuj7YMRqYWIS/EOJpIpqt2eUyAD8SVi2JZ4iolYhOEUK8Hsf5GYsoQkpWSA0Ys5nbx5dh6m8w9T+8MziM+3f2KTXslmxGunpozWak5Z5VZDNpjB+X8o34ccOVP2Pg0RvHBL9Nftjabiq4n1g3JvhthnPWdhb+vlQq1LMNwEHH697CNhb+CcEWXn/bucejOdtRMzamFT7dAlQX4ulGl0VLCs80UekKSHcuuzYRAG31UDdxFZGLMmHUxeSTOxxsu4wjvcG2MyVUyuEr+7l61uZEtIKIeoio56233qrAsBgn7fPbkFeYTEaFwKqf7sGq+/Z4bPumWcAyv4QO1UphQJED8M7gcDECaMfqc9Gm0N7tgm/2SimITT9MSK3KJ/K1rn2Bk/JM/SsNQcuMYNuZEiql+fcCmOl4PQNAv3snIcRmAJsBq6pnZYZWHmpVO9Np58Oj3kdiJ4F984ozfa/X7Zfwe8Aq04vK7AOUauKmndT8zGXOZxmm3INqwrjnmdeKx4vSQa0maw1lJ8m1/Owk82Oct6bU5g8Amay1nfGlUsK/G8AXieheWI7eI/Vs74+abVutiaNrVx8Gh0YCf05W4VOFc795tzymFOI604vK7GNjC0M7KznKvTQt/RymiFyQgnH2dyJIIl6iFZCLNwAPfgEYHRrblm6ytpti2/WfWGeZelpmWIKf7f1GxCL8iegnAD4MYAoR9QK4GUAGAIQQ3wfwCICPAngRwCCAv4rjvEklinYW1abcN5ArRry0BfjBB61v7yRswtPaZWdg1X17MJwvFYMTmzPapjAqs48TWxhGjdQxiVDyC6kN4utQCXG/Z5MiwpzVDxeFPIDI5T7KSlyCe24HC/uQxBXt8wmf9wWAL8RxrlogSrZtmInDLRxsh22QH7yJkMukCRAoEdZRG9Lb5w6inZoI07gycP2emckEu+rC06STnAzZuE2ejfuZjx+XSr55iAV3VeHCbmUgSJEzoDw2ZRvTH7yfkLO1cft8cZkSVJq5zmShCku1iTMDVzfREOApeKca9y0P7VcWq/Mbd9C8htzwqPLecClqBXs7G858xMLfkCD2U1NHo33cctmUTd+3j6/Tpo8X6u0ENaOEsTv7mb7cK4aWbAZEljmotTkDIYCVW3Zj07YDkSenVReehpVbdhs1i9GNW2eqosKxVGNVPRu7tWQQuCaRhAZNFms44V8OYeQmiDmjEjZltz3Yro/jHN8575vqacjuxL2CMLmPYfwXXbv6lLkGzvPbk4BzHK3NGRw9PlI0r8Rh526f34aeVw+XROYAwRva6AS4rs+w7cdxdxSzz69yAk9szuD4cN44f6GhiSNZrAZXDg3VwN2vWbmKMA3CTZmz+mGlqcdPI9QJBxXZTBpXLmjzCPpsJo33z2rBL146rB3PrcvnSevg28d1lmQYHBqRmjpasxnsvvkC6fXoVkEE4JX1lxjvb+P3nLp29ZWYZexM4aA9hVXP0r5vQb57smuzn3GbaxJXHReI10RXt6xthfzXQ8DaAf/Pu1cOgBVyeum3qzIBcAN3CWGjcKKWS9YRRiMEvMJBYEw4pAhQ+RbtCpcyzfqZl9/RTiAt2YxS2OaGR/HPz7xWfK1biQzkhqX1gm55aL9WkLtNFqa1gnRj6drVh1U/3VOSwzCQG8aq+6yMZvdKQ3UMVVVUe9xBnduqVpbu74XfcVnYG9AywzL1yLabUKNlJhpK+IcV4kEduEEI4h9wohIOJzWlkRfQCkVV4TNdQbRsJg0i/XGD4J5wv9a1T+sQld0T08k3rUkM2LTtgDR5bTgvQoXmunGOO4i/JMh3NYzTnHEQNVmsRstMNFQ9/7A178tZ0z1syWCVcDg2pI70sFEJQ932b15xplF8vZNlqe3Y3nQ9Xh5/DbY3XY9lqe3F95zj79rVh3scqwbV+d33xHTyHRVCWf5AN4GEDc114m4OY0rU/gxcBiIAczssE03LTABk/e9nsnGWkibFM054mYmG0vzDatlRyyX7IdPc/LS2IIlDTnQ2f9V2W/D6FUtzsiy1Heszd6CZrAzOGXQI6zN3AMNAd35piRDTmUwA4P90nCW9134hn05Ujl/dfTQRtH4ThN0cRnZuGSZOXhPqpgxEpQiSc+C28QvJ968Gykw0lOYfpTFH+/w27Fh9Ll5Zf0mxKFi5MNHawqw6WrMZTMikcM8zr2FCJoXWbKbkPny9/UzcMmc/doy3tPUd46/HLXP2l8TXywqzpSQLhhvGdRYFv00zDeGGcZ2e8esEqN29S4b7ebZmM1YimgRV8bVVF56m/MyxEyO+mrLJBGHaS9ndocz24wDBm8iU00/V8Mhs/ABAaRivHBJAQ2n+QG005jDR2trnt0mjbgBINUa3Vv/O4DCymTRuXT6veMxnu/8Rl766HtmC0G7DIVz66no8230yFi27VrkCWrllt2cM0+mQ9Nqm09vF8Re3KbRvAordu1S4n2fXrj58WTIeQG0vByBNwhrISbR2V0jfbad/CX/57Ht9Vx9hTUgyJ68JJn4q9gmERGXLF3mz6KCE0FCaf61gqrV97KxTPLWys5k0/vzsWZ7VzVO/fcu3FHHbrzcWBX/xeDSEmb/eVHwtWwHJtN9+MUV+DWIyJjZnig3Y56x+GMdOjHi0bwLw52fPCiyM2ue3KUs5q7T09vlt2LXmAunnSu6Rvdw/chCAAI4cxKJ9N+OWOfu1TmXduZ3Eqa37+anYJxCBOiklzcI/gZg4+7p29eH+nX0lGj7Ban/49fYzPQLaT7B07erDNCHX1t+j2G4jEzS34WqcoPEl2wZFEzaOdOD0U/6gRPAM5IYBYSUm2RPWrcvn4evtVqx60D6tKvPU4JDejOMrfBUhfUtevd03UsrETBfVyevEz8QZtC+Bc7I27T0QiST35j1vjWXTd1IDNn43DWf2KQdxL59NHNMqE8FTv5U3wfEzA2zadgALxRTMkJhr3qQpmKYZr8wctPTC6/DNh8fhs0P/jOn0NvrFZGwc6UB3fina/+MhPJ7egunjD6FfTClub24ah11rXMlfezsx8uCXMG70uPX6yEHrNaC0qdrjcZvF/JyvvqYSxXL/FLzt2ZYmQl6IQN+HsAEJKnQmziCrjKglyj34ZcMmvdxCnZSSZuEfkdh/GDCLLgpqIpAJFiqM185g3pjqKInQASxt/eCCVVrhb4/Zfb0rtyzGXVhcsm1Zaju+MU4eBfTQwFLPcQcfXYNmW/AXGDd63Nqu+bHZ0Ulun4gu4sVX+CqSgfrFZM+2vBAl2cgq3IqDnSVtl+Z2auNx2uOD5K7EGjlkIthrIWmqDiqSstknImHa+pngF10U1ETgNAMApU5hO6ywO78Uq4c/i978FOQFoTc/BX9Hn8eiZdeGugbZWHRRQLL9J+TekB5btb3I3k5sGfycb46BE99oMMlyP4fx2DjiFQImppquXX2etphbfnWw2BbTXaY5TlNLkNyVWCOHdILdpkaTpmoN1vwjoooRDxODH4QwJgJbO5fVKrIngu78UnQPWRp4JkXYdNVZsY7RjvZxM53exjnvm+rZ3p+fjBkprymqPz8ZSvdaQbuckbKuUZdj4EYbDSZZ7j/3x1/C48++F8gHN9Ws7d7vqfE/nBeeInKApVB82adSaRDzY5DclVgz3E0Ee9RyC4wRLPwjIqunbm+Piu7HHCXxzFhji3gJsjEep2loznk7ePaLyVJ/xR1Nn8QNw7d7TFF3NH0Sa6G4R//i1S7t1cXj4kPRMrPndqBrdIl1zt/lMP1EFlcumFpS0O6c903Fpm0HsHLLbu1zUbWw1CW8qcyKYcyPpmHPsfoiTAQ79+atCA1V1bMczF79sPK925bPk9acNxHUqpoxfi0OTVBVKZURR+XSEvZ2YvD+L3iE+erhz+Kh/FKPnbxrVx+2P3A7vox7i47j23A1ll5+HQBIhdLz6atBEhGaB6H7sv36e+dwRg5mp2Hj8HLcfXSxsj2ifU7bPBSkcqzuu+OH+7mUs/IsEGNQg2kFzFopkZzAcXJVzwrRplgST2wurYDp1PJMtDJVzZig5QJkBCmLEHtG6NwObOzeL40CksXZW9d4HZZvO88jeJasf1Lqb+kjuakolZ3oL/gdgqk59zpuELfjcGoI3QNLcdPWfZiQ0bdHDOIcndickRazMynO1zeQK6mMWu6M3tiSI00jZWyHqi1ct66w/k+AcC2S9KgkH1j4R0S1JBY+P16/aAndjzZqjRaZOUZVe78cnZ/mXbIC52/9AHJDZmYEleBR3aONIx3435nNaKKR0jdOvGv9YFU/TIkz0jYXdQ8tNWqPGEQI33zpGZ5y0pk04e8vH6vFr1uhOZWAclaeDY1KKzaNlEm6cK2FqCQNHO0TEVWEyBGFPdeJTsD7/WijanTuaKJL5sqzhcvR+SlKjSUnqnvUnV+Kd8UE7xv54WJUiSxpSSickSontWwsqjG1Nmc852uf34ZNHz+r5D5s+vhZxclux+pzcdvyedKENaA0qkwXvVPxBC1Amg2Nh64PlqxlEhlUTWo8KqnuNP9q1CuRaaYmFTBlgsK0O1ecGp0uW7hcdeLjMCPozFcT6aj8Q0d6lc7RRanJaJMkuREEtjddj40jHXh6/Dk4MaJujygbUyZNOHp8bGWl6kksw37Pr16RKgAAQOx5KEbEoRUnXbjWeFRSXWn+SapXoioxYCPTqmVVHU0/K8NU4wuSLVzVe+xK+W9P7yjJXXCiqi2ElhlKu/yG4Q4MiibPR4iAGalD2JC5Az+Y/4p21SJb1ZzUNM4T0hkkF8S0XpEsN6RceSi+xCG4k15Dp8bLPMSi+RPRRQC+BSAN4A4hxHrX+58GsAmALSG+K4S4I45zO0lSDXO3JmYS7aNy8k5szqC5aVygUMIgoX9B7NSVusfu1cVtp7+ARftu9th/2y/9NtpXd3iud+NIBzZk7igtVFf4Yfb/WH693fmlwLCViNZGh+CO1s3SEBa99J2SCqcy3Nr8HEVUj5/pznkPWgrlqp3+ARMloGqlnePQipMe8lnjZR4iC38iSgP4HoDzAfQCeJaIuoUQz7t23SKE+GLU8+lIWg3zoKYN1TgHBoeLNW9MhXoQIR3EWViJeyy7xuk7NwKkNiO4J9ud/+l8PHf6bCx66TulYZs/PgkpRW4GMJbk9vL4a+RpDiFMDmGcse57MJAbRiZFmNicMQ4XDnvuWIhDcNeCcK3hMg9xaP6LAbwohHgZAIjoXgCXAXAL/7KTyIiHAKjGnyLCnNUPY3prFgODQ0ZCXSWM7Vo+TsFxzvumerJKVVplJe6xbOI6BYrKog5hXDLZ7u0EnhgT/GuOXYmfDll1hnQVOG36FUXuwpgcbD/A+aP/apWxoEN4HVPQf/oNAEpj8J0+HzfDeSEtfqfzwaj8Iu8cO1H8TpXFLxaX4A4jXBMYe59E4rD5twFwru96C9vcXElEe4nop0Q0M4bzeihnr91KoPITjApRtK8fG9KHGtrohLHTTh/U2VuJeyybuHQ2fA+uSJPm3OtYR5tL6vsA+izsjSNe+38O40OZHNrnt+FHi17FhqYfYkbqEFIEtNEhy4zliH5x+3xkuO+Nnw+mfX4brlzQ5lnFDA7ny++zmdsBrHzOanCy8rnKCOAwUUZJLh9dRuIQ/rJfkFu1egjAbCHEXAA/B3C39EBEK4ioh4h63npLXppYR1whhNXCPf4gJSLcwt7P4WyvFoKWhja+xxF+ULKJa+NIhyV8najMCJp4fSd5IXDb8nnSNo6yInerhz4TWoAteuk7yOJE6UZX2KJfM3jAe29MHLpP/fYtbcmI2B3A1RSmQcND4whJrVHiMPv0AnBq8jMA9Dt3EEI4A6V/AGCD7EBCiM0ANgNWeYcwg4ktE7FKOMevchTKcGveThu4SpPU2el17/ne44jJOTJTxePpD+Ev3j9mw9cu5xV2+TY6hJfHX1PsIbDzP51fvI6VnbvhtgY5i9wBUEbcGGEQ/eLnN5GtsEyerYk/JjafTbUTs4JGGdV4olYU4hD+zwI4lYjmwIrmuRrANc4diOgUIYRdzWsZgN/EcN6KU+4cAvfxW7IZZfEvJ63ZDACrvkv/QA6tzRkIARzJWY5BVRkBXWZviqikfEAgIv6gVDHri+ZfBMCgvLQi0oTIWqbOICts87nTZwOwQiJ7Xj0sraZZ/Cy8E6wT3++GQfSLyp8CWBOP+5hdu/qUuSDOFYLuuLL9tfjZ06stTE2ijJzXoHriScklKCORhb8QYoSIvghgG6xQzzuFEPuJaB2AHiFEN4DriWgZgBEAhwF8Oup5K02Q0Mkwk4Ts+Jk0IZMiT4y4k0yKMDyaL0kCcgrzvoEcMimShgme876p2PIryQ8Flp8hdDJQDDHekVZwskgTF3bYJnCt1O/hRkB9H4y+GwbRL6pSISrT5aZtB6Rjdk9UMoe+E2OfjYlWX+3ELL/7LCssJyMpuQRlJJY4fyHEIwAecW1b4/j7JgA3xXGuamEaOuknCFQTg+z4w6NCGuNvlw9uyWZwbGhE6QQuHicv0JrN4KTx40rOu2nbAe3EEjp+v9qZj+5IEx/tzsTW7jT5uJ/h4NCI/3ejMKbBR9dgQu4N9Ocn4w7xScwbXYL2wmfsFchPfnkQo0IgTaR0vANqU41zolJNbM2ZFHLD+WArWBOtPmnP3r06kV2DmyTlEpSRuivvUC5M49t1k0TPq4fxz8+8VtzeN5DDqvv2aI/vjPF3s2T9k0ZmIcAyAe2+ufQ4KxUlA5yEsgVXOznHbZoYOgbkDnv3KwikILZ22eSuwhOZM7oENx29bez7MQRkXYrB/Tv7iqGoo8Jq7AJgrJm9A5U5xzlRKRMHTxqP54OWejbR6iv17HXmJ114qGxiKkINFRpaV+Udyolp20RdfL1T8NsM5wXWdu8P3JbRPqYpsuOY2HntfYyLg9k/yuEcQIVoo5aZ3nrt5UIWvTF0FEhlSvdzCCTdfXBHM5msEmyCRuaoIq/ueeY16f02CbuNMylvMCvv5FyyfW6H9axbZsISpmV49mEjdPZ2QtmhqGVmZUNSEwALf0NM49tVgkQXtjmQGw4cP287+0xQHccvHNS2HRvX8yn5UQIQo2NCNsgPKkqooGxZPzpkVfS0yU4qEUiqe3/b8nme/smmQlN2z/0Esc6MIwvFNAm7DaNUqNg4vNyT+zAomrBxeHnpjuWO7w9b7fOJdZCbAKkhzDxuWPgbYhrfrhIkJlmlQXIUVM4+wLLntmYzvsexr2lic8bzHgH487NnBSsOFkcJ3qhJOtplfYGR0jHaiVD2BK2ztTuF5rLUdmxvuh4vj78Gv5hwPT598q+091wlcFsK0Vo6gaxrOu8u5mbTtasPg0Mjns+ETcq7++hib+7D8Gdx99HFgY8VibBOZeX7omG0fSds8w+ASQSKKkzRr8Tzpm0HPD9eHToN9Pm/u9joGPZ4dY5o3bk821WCN0ikR9BQQdPoDdfxBrZ+BU2PrkFz7g0MZqfh+LErMSr+KwDL1n7/zj4sfO8k6eRul2pYn7mj2I5yOg5hLf0j1l5zBjB3rBWl8762NmeQApB3DefY0Ai6dvVh1YWnYeWW3b6hmyao2oC2ZjNYuyxcG9DprVl0D5TmPgAR8x+cmJZlCOtUVn6uLAUHEg8L/zKgmiTcXZucBLXBmjj7goScysZsf96op0DRnirZO0ikRxxJOga0iHdBuXcBoFj+YSiVtyp7Qh3pZL8++8EvohlDpQd1TVJuASzLpwCsqC578pflG4TR1FW+iZPGjwsdQhtrI3c3JmGkxcnhIDzfNROnchhndB3XCWKzT4WwuzalFIb6oJqdn48gat19vzoznh99XPbUoDXcQ8aPu10wzTSEm8f9qGSbztQyzaDQ3KZtB3D+6L8WTUPbm6731Bdyn+vr7Wfi1uXzjMx/Oido63eZAAAgAElEQVR8OaqvlrV8ip/J0O1PgkDReWvqVA7qjK7z0g+s+VcQ+0cSh/bkNi/ZWb0rt+zGpm0HcOyEQey5Bl1Ui51tCoxlFb80oVehSQS0pwbVzlRLeQ1CeIU/AEyio1iW2l7U/rUTsuK8b2AKnilkRi/4/eMlpqEZdAjrM3cAwyiew8bdlCVMUqCqp++y1PZiNdE3aSqw91ho7bVs5VP8VnzSFZ6wIsqCaONBqoRWO1u5zLDmX2Hi1J5sZ9+ty+fh+HAeA7nhopaviv831fxU+9kRQGu79+PLW3YXVxb9+cnyAwW1p5pqZ7aTt2gCcI9STV7xPhGKxd98J2RJF6dB0YRvDF2Fm7buw9e69uGGcZ1FwW8jKzDnPpdJWK2fE95eGS5Lbcf6zB3FaqLT8BZGHvxS8rTX7ET9dtXkIEbLp41XO1u5zLDmXwXi1p6ixJ7r9pOZfFqbM2O16ZssbbJfTMET+Xm4ip4uFXZhk3v8tDOPk9c2AQhrsjj1AmDPj6W+gEHRhPtGP4i/TP9cqv1Pp7eldXSkYwTwxtb/ifeIQ+gXk7FxpMPS6POj+MkvD2Jdk9w01JayziHzxcg0+pVbdqPn1cMliV5+Zh37eIsf/ILHNzFu9DgGH12D5kprrzr7+cgJ+WdOHLUmeV3xjXJp436O5Rr3B7Dwr2F0jT9kBDEvqZx7QsAT6TKDDuEqehr3jX4Q56V2Y3rqbRzPTkPzxWVaHqtMAC0zrbhyAJh19phzkNKAGMVg9hSrm9eJxbhs3C/Rinc9h061zsCOla7MV9WPfG4HPvDjk6RiaVQIZUMYapGco4Au0csZfaRrquN09L80/pB0ITQh94b0/MYEFXx+Dt3hY/LP5YfMzHrl0MZ1JshqVy+NARIG8efVYOHChaKnp6faw0gsqlA+J+66QKpev6qIINl7K7fsxr81XY8ZKa9QGxEpfGX48+jOL9UWJANgJjxU+6xthdK5vHZAd9tKjy37YbtNTLL90k1A08lA7h28gSn4xtBVHht+mgiX0L+VTJKA1RAme8V3lQJizuqHlTpuW2sWOwolGWTPP5tJ48oFbbh/Z19x+3bFs+rNT8GMdS8pzuSD6t6ddQ3wwmPyZ1o00bmwJ+y1LeHG4j6ObKxRtHPV5/2up4oQ0U4hxEK//Vjzr1H8TD3ZTBo3X6qP5/ZzGsrMU5u2HcD0nNycMY7yYw7N4aVq57JpWJ9qnziKh5m2GVRlDBdqBU3DW9jgcuKOCeEPFRvCT6e38Tomo3/BDVjkOEeQMt5OU48un8T5vdg40uGZgAZFEzaOdOChsG0cVY7QnjtRnJTdz9TPfp6dJK+/ZILKvBiHdq4yQdaBP4A1/xpltqbRi5HNGlakjipXYIei6FfXrj4s6vog2mT9bQv05qdg6dC3QQBeWX+JdwcTrUm3j2o5Xo76QcpVRilvYCo+cPxbJcLUL89Cpr27S2870T0XG9nKYSza5+1S30QB31WamyBauv1MVc/T3ufUC4Cdd1kOXD+yk4Cmk/y1+XJq56z5M5XGFigq0kQeAaESQmFiwdvnt+HZgzdg0q//l7ctYYE2OoRfj1+BVjoG3Cr5cZpoTbp94moOboJhKOk0HJJPdBpUZbxPakpjcGhUmejlfp7OMt8pIk8pEXdHMjeBS3cXfChG2M9R12PhyEHLQZ+ZYFVg1ZHJAhdv0MfmV6JRS7Ur18YAC/8awsTO7/7h60w7OqehjkXLrgVmTwQe+LxUCBABk3DUeiFbapuYbfz2CRKvHQWDxjAAPKGKJg1eVJPs4NAobl0+Tzphy47rrBZrUkNKRqDkL1PBD5Q+LwB49Ea5ecckS9te9c3tsIS881jZScAZlyujvKRjCoLM9n/ptznah6kMQZuOqD5ja3qR0vXtL7mJYHSH4ploTUnRrNyrjOxE4PgRrwA88a4lIAr7mzT/0U2+qnDgIGG9QQiUYd4yUzox5wVKM9hlz2skwCRjk50E3PjK2Ou9ncCDX7B8Lza5w0DPDw0OFqKCp8x3sPVz1rh0q5CEw0leNUSYBt86006khDNn3X4TnEttk0SuStSFd6BNrHKWKL7xFWCCxOadHy6pXmpiUtOV6FCNJ4iG7qw66iwtEaR0uBRFgts/jX7Eqvipel5+35fsJM9xi2YeJ0+sKxX8gQhRwVM17tzhmi73wJp/DRG0wbfuM7amFyrhLEwlTfdSW2e2cS+xr9hcVu0qSH9mAEDuHfmBHBOciUlNFbEDILCpzk17ege+Mc5bWmJSpgnzLlkRuMd0Cc6WlINvlDiR12fS+OZlCgVCZ2t3Cnk/U0oUm32YCp6689VwuQcW/jVE0Abfus9EqsQYtJJmEHNNFZJnTPszFzHwWZjed9nku2T9k4FMdW6ymTS+2nQfmkclpSUyW9A8/xZ1CK6pDXtuB5rndpQ4n32jzFT3jdKlqwS/56x1wisqywLhzYZ+Tv8aCu90wmafGiKMmaYslRi1X3aylu/ZSQhlromjIUxAAkc9ScwebsES5b4HNdV98uxZJa+vXNCGySNvSY/RrMrsDVnBUtdMxoPqvl3+/eDF/9JN3u2pDLDwr8dMhVG+h37jdlKp5vQxw5p/jRHGTBN7JUZdU4yoMc5VSJ4JHPVkGGrant6B9vHrgAm9wPgZQHoNAH/hE9VUt2T9k+gXk6WlJZSCqhIVLOMK0ZVFDgVxvgbN+tVFKtVYeKcTFv5McOKKxJH9CKNm74ZI5w9lGnP5LLp29WFTobz19NYsbjv9BSzad3Mo81VUU13/QA4bU/LM3mbVMwoz6Qa51z/7ylgSF6UtDf1j/2B0PVLChvqGNSva56vxYm5OYjH7ENFFRHSAiF4kotWS98cT0ZbC+78kotlxnJepEnFE4qjMDKdeYC3fnaQyZhOL7JhbV1gZqZpG8FFNY7LGOdN3bgxtvoo6numtWXTnl3r67W7MXKd+RkGb6AQxE/3sK1YYph0eK0at1z/7itH1xEpUs2K5m9NXkMjlHYgoDeD/ATgfQC+AZwF8QgjxvGOf6wDMFUJ8noiuBnC5EGK57riNUt4hSKvFuhqHKj0+OwkYOloaypduAi77nrfgmlsDK7b4U1CmEhCyMhkvj79G0bUtQPG5kKiKvvkW2gtSMsOvXINTI75lkjwxjNLAzSHq+UTRvuMoCphwKlneYTGAF4UQLxdOfC+AywA879jnMgBrC3//FMB3iYhEUgsLVYjAIYaa40QR3HGNIxAqc4Is+3N0qNT2rFq6B0k2i3H5LnPQqso5m5qvVM/U5FmrQkhN+hMY3xOdOchtSlFlBAfJFLaJGg0WR1HAclAFc1Icwr8NgPNu9gL4M9U+QogRIjoCYDKgaoTaGAQOMZQQh+COYxyBCdp+0SlsVEt30+PEHE4qc9BuHOnAhqYfltY/MvSLqJ5pz6uHS8o16551KCd/EDu63/NzTrSqWkCU9m7zQ/XsH/i8ZeLzE5xJyRx3UqXeAHHY/GWLW7dGb7IPiGgFEfUQUc9bb8lD1eqJOJps+7Xzq9Q4AqMK+8tOku/v1MwiJfnMiD2cVJap+3j6Q3ju/X8HtMyEAOENTMXfHPsrLHlkirQtoxPVM/3JLw9Gftax4Rf+CFhCbG2rVbBNxoJPBz+vrp2jSYhqhTPHjahCeDMQj/DvBeBMm5sBoF+1DxGNA9ACwLO+F0JsFkIsFEIsnDp1agxDSzaqUMIgdVbiENxxjCMwqh/hxRt8Y+jNl+guncM+TszhpCoH7aJl16Lrw9tw+ui9OPv4t/BgfikW/P5xLOr6IMTaVqUTWvXsVEXbyjpJA2P9kp1jLnl+OoRVqTOVRvF5UBpY+Jlw0T4mz95PcNpO2ys2W6+3rtAGBJSdKvUGiMPs8yyAU4loDoA+AFcDuMa1TzeATwH4dwAfB/Bkku39lXJ+xpF9axKj7nc957xvKu555jVlCeGyoTMz6OyfRpU2yfpxy46jcgzrIlv8YvoNCrHZzdSL4ZeK5b3qmaYl5Zrt/cuGn0nC9p/4PY/8aDx5IOetsYS1X48FP8GZpDaMVfJDRBb+BRv+FwFsA5AGcKcQYj8RrQPQI4ToBvBDAP9ERC/C0vivjnreclFJ56d9vLXd+4vdmyZkgi3G/CYQv+vp2tWH+3f2lfyUCMCVCxQ247gdU5r+uEpKnJOqiJMZ6uP49WZ1jsfdDD6gkHBq5TeM6yxtcA9IE6lUz/SWOfux5NXbcQoOoV9MwcaRDjye/lDJJB274mKS/OV2Fpezjv7cDquiph9+grMSSW2mVMkPEUucvxDiESHEfxFC/LEQ4u8L29YUBD+EEMeFEFcJIf6zEGKxHRmUROKwoQflxEi++Pc7g8O4aes+X7uwjV9MuN/1qBqGP/Vbic8lZAkAJVGOV1y6/8DfTCT77FnXjDkcKW29Brzj6bkzkj3WqZVPV3U/cwlF2TP90aJX0fH6JrTRIaQImJE6hA1NP8SPFr1a0nPZnW8Q5LtkMjbldmf8u8oUFJcm62tqImvS1pGkNoxV8kNwhq+LSjs/44i00UV2+F1PoOuNW1uK43iyEMVTL7Beq6I/9nZa2rwz6WjPj4H9D0hMFyot9qBlJ/ZZ+dx2+guYvnMjTsEh5JFCCnnvTpQq6QUASJ7prV6zShYnsOil7wC4FkDE75JqBRbGJCE1yxkIZFN8zX7Cep6zztYntSUp5LNSzYkccGE3F5V2fpZ7svG7nkDX66ctyRyDOuLSvpxa53lrrB++O8v37mVjY3vg8/JJJ2gDcb+Vyt5OLNp3c1FbH0d5+VQiRv1XPAb3KvR3SbcCMyhi58FeWZU43AsCOQ6nqq0pqyLDAP/VWZjrAoJ/xxMMC38XugYbcdO1qw8pkqaBxjbZ+F1PoOvVlQAIY8IJWlLABGm5aQG88q9jYwucXCR/RgD0QkYyFuWR/ISVwb2yvzPuJi6fOvlXxX2kTWIevVG/AgtjknjhMXhWTc5rjCpE53ZYTdx16JSIMNcVt9mzyrDZx0Wo7MgQ2PZZWfRGnJON3/UEul6dYyqMCaccjq7INltXPfhM1tJiX3hM7VyOawWj29/gXq268DRsf+B2rKPSJi5fE98H9p6BrtElHuf/9gdux2Wpw/JJyR5PGJOE7p7EFWnjd3/9lIig15UkJ3EMsPCXEGcJZFX0haoXa5ooer19F37XY3y9uhIAW1fIP+OnfamOF5agmcMe3Nrqcev/lc+p69notHJV8xLZ6kMnrAzuVfv8Nlzw2P1ozpVGFI0bPQ48sQ6bTnzb8537Mu5Vr0airMB0NnWVEH30xmDPXvesZUpE1Ei1JDmJY4CFfxnRhVmq7LB5IapS2M0YlbYU1oEWt6PLNA4cUAvhEsRYY/CgKxXV/mddUxo+6nccG4PWl8251+XvH+lF/3FJPogqAskev+ZcWiGqu1cqRSF32Kr0aZr8pXL8ymr7x7HaSJqTOCJs8y8juuiLqmTVlpOwDrS4mdth1YrX2emBsQ5SfvvZ7LzL307stmMD8v0/9g/xhvbt7QQe/IJ+xdMyQ/rd6hdT5PtnJ8nHY2r31t0rnbDsudPchi47xxU/AG58xTv2OEooJOU7HhORSzqXi3oo6Txn9cOq4rG4dfm84GV3k06SGl3IkrVeeMw7Nl1pYjdrj+jPF6QkcpxsmKOPVCqMw23zB4CPN/0C6zN3WKYh1/6BSjkHyd7d26lP1IojE9hNXKWck/QdV1DJks6MAl3phSCO1qTU/PelCrHKSkzHYlQqAv4VKCM4A+3nu/D3j+OmpvvwhzgECiJYdILfUVu/vbDJ+V1aeuF1GJc+K3opZ2e4r9+x5nbIWyIWj+WYXOIStnGZbJL0HY8IC/8y4ld6wcTRWpVa+9WmktqVy5E6msoglR/yGoP8KlCGbIM4+OgaLBt8A/9NnIQ/yBxHE0YKn4up1oxLg5Z/52Io5ewM9zWxq1+8Qa/9r22xkt+EIykuyj1JYinnKsM2/zIStR0fUJ1yE1Wl3LHUqgqVK5/Ds+/fgKFRV2qSAH43+Wx/J2TINojNudeRIoHJqaNoopHSfUxt0qpkJ10SVFDs+3bkIJTVUlWrH7uV5toWy0TlymZW4hT8zuOFKXWcxFLOVYY1/zITNWy0KrX2q0k5Y6l9NNOZv96ErKvwGhEg3n7J/9hBNUtpMpoE1crBuTrKToSlxzmEZSpjaddx4PFnCBTzIZwtG1VRPE5be+6w5ZwGrM+GCcsNG1pZRyabOGDNP+HUXVSQH0HMJyZZos59VGUdCprke4S8gdB7hEHDuaCapakAk60c3Kuj3GEgPa6g6RfO3X57fIJOlTVtO2btss5kKE7stpwmDWFk1GhoZdJgzT/hxFHzv6YwdczJtPitn7MciXaMt3sfVUx/QRC/SVMxDd4J4E2agmnO86r8EXG2QQTUKweZMB4dssod3PiK2fmDYOLkfej6YGUzjvR6E9co5X+MBrfTxwlr/gknDr9BTWEaS60ym+QOj/kITE0rlALWtuLk1AkMiVJ9KCeacPD9q6wXcfojJNd5QqRxWJyMvLDaPnpWDiV2dwlBzCFBauv4+TNM77Pss86ifJd/X78SyE5qeDt9nLDmH4FKhWDGWW4i8ZiWfNAJOtuUYyoMC9rmyfnf4wQsAdyKY+gXk/GL2dehY9m1lnB84PNezdTpgPQbs3vVcNY16P3Vg5hOb6NfTMbGkQ5055cCsCzqr8y9pPSzfiGpJuaQvZ3eMEu/KBo/f0ZQG3y6Sa69u5v02BnYTr+CfQ0Jj7WvBTjJKyTuEEygDpK0agnf5CwKXeenNz8FS4e+DcBaae346CF/wZvJ6hO8FElga8W1uOvoYs/h2lqz2LH63LENftdrklDmN4Hokqt0AjdIopys9EIQqplMVyOYJnmx8A/JkvVPShO4PD9apjyYCLLz1pi1/HMhhBWf0i+mYNNIB7419SG9cFMWanMIU4WAHMyeggVHb5MrEekd/q0R7fOYaL8mQtr0WE5MViWpTDxO6DgyjOscU+HPNv+QNFwIZtLQNfSwTRJzO8wjUBwQodgqcX3TD/UCM93k60j2/O2gOfeG3KeT3lHqX1DhjLjxw8Q8E8aPURLppCA/7B+fb+KHqLPKmtWEbf4h0ZVuYCqEHV2jM0nIEoUCkMUJ9ZuUspYJKpw2eE0Uk9Snc6uBEzVo5IupGSxMXoUz0klVR8cn29koO7jOKmtWE9b8Q1LJjl+MD86IEbcWrNNGs5OAhZ8pxuYHM4ASMKHV0mhVDB0b016DVoT002QpPSakTbX0ID10o2jSYTq0mVbdrLPKmtWENf+QVKrjFxOBvZ2WAFYxkrOafBdKN1AQxyUEkHtHv4sddvraM1ZF0eGcOoLFjUrDzU6yxh2mLv0Lj+nfd58fMIuskVVQDdqvwNScM7fDup8777LuI6Wt/gh+184RQh7Y4cvUJyZOSKDUUSj9jKuto/NzgOFkIWkNGSYyJ5MFxmXl1TDd1yETdMqyxi7s8QH+kTWqcc5YDPzH9jEBveDT+vpIpo7cMNE+DRYhxA5fprEJUztHVqJh4V+rzQzG5Qk0jcxtTBvBqFYb7mxbWSKazuxil6x2lqUwMcWo9nnl6TFHuBi1VgI681SU5D6/Ym9xNHKpQyIJfyKaRESPE9ELhf8nKvYbJaLdhX/dUc7JMEaErZ3j9h+UdN1Cqa0dKBXQfjX/VeNTCWzA68sIk21rj/e8NVbIpQwxWhol5R6jauzK+2ww4TkxrY0UJtqHI4SkRNX8VwN4QghxKoAnCq9l5IQQ8wr/lkU8J8P4YxT9QZag9StvMLdjTDO1tVmZgJaWJ1C0ibTr39/6p1Yugqlm6qch6wTd3A5g/B8oL9NzTt1E87OvALdMgpEZyTkGHTrHvcmYVIT5TAMQVfhfBuDuwt93A8VmQQxTXWRCMt3kyAtw2OFNYttNTAdu7TU7Ccg0y483eNi/7+6Rg96Ydz8N2U/Q+TmpnQJaFR307utWU3tlETbNhBeVMNE+HCEkJWq0zx8KIV4HACHE60T0HsV+E4ioB8AIgPVCiC7ZTkS0AsAKAJg1a1bEoTENjaxOjF35EvA6Tf1i23UatczBCugdzsOaKKQSXGYgO55eNU6/Ojx+sf5Zh+VWFR2UH5FvB6zJKEy0jymmtZ+ifqYB8I32IaKfA2MVbR18FcDdQohWx77vCCE8dn8imi6E6CeiPwLwJIDzhBDaDhkc7cPEgmnUDwBtM29VNIo79BLQR+VEwbSEgS6s0e9+OMswmEYHObGb3HNoZdWIrYG7EOIjmpP8johOKWj9pwB4U3GM/sL/LxPRvwCYD8CgPRLDOAgjUIKUG7bt8LJzqDRqQG4OClri2ARTB6VuZWBvVzVQt8swzO2wVgFBJjCnw5u7ZiWeqDb/bgCfKvz9KQAPuncgoolENL7w9xQASwA8H/G8NU3Xrj4sWf8k5qx+GEvWP4muXX3VHlLyCVtL3zRpK5O1zBWqc8js+ZSKV7vPZIErfqDOSo7DZm5Pbrpx26asE+8GO7Zfk3smUUQV/usBnE9ELwA4v/AaRLSQiO4o7PMnAHqIaA+Ap2DZ/BtW+NuloPsGchAA+gZyuGnrPp4A/Agbq60Kv6SU12lqZ+GqzmFHo1yxGRg6qs8ezk7yzwFIZUpbL9qJVbLj+tnMTVtaFic3DS0zrGvWla4ogawyGX5N7plEEcnhK4R4G8B5ku09AD5b+PsXAM6Mcp56YtO2AyXlewEgNzyKTdsOcGkIHWFjtVURKSLvtZ+rGpC7z/HEOst5rCKTHWue7jQhTfqj0qzX9/9lqcBU2eP9auCbFkUzNYGdt0bTjN1FdlJ5WkcyZYczfCsMl4IOSdhYbaUJRbLd9Bx+E44deumMWz9vDdD7K33Wq0o4N52kF/w+jemNx+08l4mJyTnJMTUHC/8Koyr5zKWgfQgbqx3kc6b7ahOKZsoFtYnZKujqxq9xuvtzJgJ9aND6X5snocnAZWoGFv4VhktBh8DZjF1Wg0aHadmAIPuet8YShG5SmeAlmp3bg65u/Mw4WVfUtUktImdjdfe9uOx7lolHl4HL1Axc0rnCcCnogLjt2bIaNH4ECTt072s7UmXhpc5wST+7vCps0img/RK03Pg5bk+8OxapZF8bMJb45q42mspYzua1rWPXWq3WiJwnUHa4pDOTbKrZszXOUsAb5iiEv8thair0fvYVq8SCH6ZN2bMTrQgmpyO7WmWPG6wEc9xwA3emPlBmmWqyceMizoknzuvY21mIxjH87V7xg/DN3SvZGL04GSlWNNyk3Qiu58/UB9WsyBhnKeA4r+OJdQhUdmHr56yVhy4hrtplj01yEBq8BHPcsPBnkk2UiowmiU864hTYcVaWDCME7ZaSqntQ7bLHJjkIDV6COW5Y+DPJJki0jpOw5SCcxCWwo0QryQgrBHUZ0dUue+w3oXEJ5thh4c8kH5MmH25M4ur9VgZhJx4neztL6/aLUStM9NQLrLGEWZVIQzYLJRZUSW02KiEb5Fqjrqhk+OVOsLM3djjUk6lP/GzYpiURolanfPRGbymI0aHSSB3VuVXo6tP7lWzWVS41uVbT+xYUWZirOxSViRWO9mHqE7/olUpFt6xtMd83rjo5ezvlJZszWeCsa7yNVmwh2zLTP56+nPetJNrHJfg51NMYjvZhGpsovW6rRe5wPCaUuR3WJFIsD+1TuTRIO8ty3jfbvNcyEx6N36SCKxMIFv5MfRK1121QVHbwYs9gQ+IUcDJfiZ+Q9hOylYgKSuLEXIew8GfqF52jOM7oFl1k0cUbrLIJpugEXByOVhMhrRtDJaKCqh122iCw8GcakzgieWx0kUVzO6yeuKYrAJWAiyN0FQhW3E1GnPctyBg51DN2ONqHaVzi6jPrZ6aY2+HfOhHQCzi/CcYUv+JuJkK23P15ddFMTGyw8GeYqLTMUETAODRorb2a/AVcWezgNFZVNPdOsoQsN4AvOyz8GSYqp14A9NwJrQatnCAMQyRNJhgT3HH6ucOFxvGbWdg2GGzzZ5go7O204uZLQhPJiqeP08Eclx3cJPOZaQhY+DOMmyBRNdKCZMKKp3cS1VEal6OVwyiZAmz2YRgnQcsXKIWpxEQT1Y4dhx08LvMRU/Ow5s8wToKaRZRCk+LJ1o0bDqNkCkQS/kR0FRHtJ6I8ESlrSRDRRUR0gIheJKLVUc7JMGUlqFnkvDWwwiXdiGTa0SsRp8/UBFHNPs8BuALAP6p2IKI0gO8BOB9AL4BniahbCPF8xHMzTPwENYvM7bA6ZclIqh2dwygZRNT8hRC/EUIc8NltMYAXhRAvCyGGANwL4LIo52WYshHGLKKqoc92dCbBVMLm3wbAqUr1FrYxTPIIYxaphB29HA1UmIbG1+xDRD8HME3y1leFEA8anENhEJWeawWAFQAwa9Ysg0MzTBkIahYpdzmCcjVQYRoaX+EvhPhIxHP0AnCui2cA6FecazOAzYDVzCXieRmmcpTTjv7ojfHU9WEYB5Uw+zwL4FQimkNETQCuBtBdgfMyjJ5aMKXs7VQXhEuqQ5mpCaKGel5ORL0APgDgYSLaVtg+nYgeAQAhxAiALwLYBuA3ADqFEPujDZthIhJXieRyE6axCsMYECnUUwjxAIAHJNv7AXzU8foRAI9EORfDxEpcJZLLjV9jFYYJCWf4Mo1JrdS4UWn32Uljk1QtmK+YxMHCn2lMaqVVoCqM9OIN1t+1Yr5iEgcLf6YxqZUaN355B1yimQkJV/VkGpNaahWoCyMNY77a21kb182UFRb+TONSDzVugtYi4oQxpgCbfRimlglqvmIzEVOAhT/D1DJBaxHVSpQTU3bY7MMwtU4Q8xV38mIKsPBnmHrG7dw99QKr4bzT9JPEKCem7LDZh2HqFVkOwJ4fA2ddw528GNb8GaZuUTl3X3gMWPlcdcbEJAbW/BmmXmHnLqOBhT/D1Cu1UsKCqQos/BmmXsbUcZoAAAMLSURBVKmVEhZMVWDhzzD1Sph+xEzDwA5fhqln6qGEBVMWWPNnGIZpQFj4MwzDNCAs/BmGYRoQFv4MwzANCAt/hmGYBoSFP8MwTAPCwp9hGKYBISFEtccghYjeAvBqtcdRZaYAOFTtQSQEvhcWfB/G4Hth4b4P7xVCTPX7UGKFPwMQUY8QYmG1x5EE+F5Y8H0Yg++FRdj7wGYfhmGYBoSFP8MwTAPCwj/ZbK72ABIE3wsLvg9j8L2wCHUf2ObPMAzTgLDmzzAM04Cw8E84RHQVEe0nojwRNVxkAxFdREQHiOhFIlpd7fFUCyK6k4jeJKKGbr5LRDOJ6Cki+k3hd/E31R5TtSCiCUT0KyLaU7gXtwT5PAv/5PMcgCsAPF3tgVQaIkoD+B6AiwGcDuATRHR6dUdVNe4CcFG1B5EARgD8rRDiTwCcDeALDfydOAHgXCHEWQDmAbiIiM42/TAL/4QjhPiNEOJAtcdRJRYDeFEI8bIQYgjAvQAuq/KYqoIQ4mkAh6s9jmojhHhdCPHrwt/vAvgNgLbqjqo6CIujhZeZwj9jJy4LfybJtAE46Hjdiwb9oTNeiGg2gPkAflndkVQPIkoT0W4AbwJ4XAhhfC+4jWMCIKKfA5gmeeurQogHKz2eBEGSbRyexoCITgZwP4AvCyF+X+3xVAshxCiAeUTUCuABIvpTIYSRX4iFfwIQQnyk2mNIKL0AZjpezwDQX6WxMAmBiDKwBP89Qoit1R5PEhBCDBDRv8DyCxkJfzb7MEnmWQCnEtEcImoCcDWA7iqPiakiREQAfgjgN0KIf6j2eKoJEU0taPwgoiyAjwD4rennWfgnHCK6nIh6AXwAwMNEtK3aY6oUQogRAF8EsA2WY69TCLG/uqOqDkT0EwD/DuA0Iuolos9Ue0xVYgmAvwBwLhHtLvz7aLUHVSVOAfAUEe2FpSg9LoT4memHOcOXYRimAWHNn2EYpgFh4c8wDNOAsPBnGIZpQFj4MwzDNCAs/BmGYRoQFv4MwzANCAt/hmGYBoSFP8MwTAPy/wEwani1EnF5GAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x28a963becf8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from sklearn import datasets\n",
    "\n",
    "X, y = datasets.make_moons(n_samples=500, noise=0.3, random_state=42)\n",
    "\n",
    "plt.scatter(X[y==0, 0], X[y==0, 1])\n",
    "plt.scatter(X[y==1, 0], X[y==1, 1])\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=666)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 使用Bagging"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 决策树模型, 非参数模型, 子模型差异性大\n",
    "from sklearn.tree import DecisionTreeClassifier\n",
    "from sklearn.ensemble import BaggingClassifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "bagging_clf = BaggingClassifier(DecisionTreeClassifier(),\n",
    "                               n_estimators=500, # 集成多少个子模型\n",
    "                               max_samples=100,  # 每个子模型看多少个数据\n",
    "                               bootstrap=True)  # 放回采用"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.872"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bagging_clf.fit(X_train,y_train)\n",
    "bagging_clf.score(X_test, y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Wall time: 8.66 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "bagging_clf2 = BaggingClassifier(DecisionTreeClassifier(),\n",
    "                               n_estimators=5000, # 集成多少个子模型\n",
    "                               max_samples=100,  # 每个子模型看多少个数据\n",
    "                               bootstrap=True)  # 放回采用\n",
    "bagging_clf2.fit(X_train,y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.872"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bagging_clf2.score(X_test, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## OOB Out-of-Bag\n",
    "放回取样导致一部分样本很有可能没有取到, 平均大概有37%样本没有取到\n",
    "\n",
    "不使用测试数据集, 而使用这部分没有取到的样本做测试/验证\n",
    "\n",
    "oob_score_就能看这部分的得分"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "BaggingClassifier(base_estimator=DecisionTreeClassifier(class_weight=None, criterion='gini', max_depth=None,\n",
       "            max_features=None, max_leaf_nodes=None,\n",
       "            min_impurity_decrease=0.0, min_impurity_split=None,\n",
       "            min_samples_leaf=1, min_samples_split=2,\n",
       "            min_weight_fraction_leaf=0.0, presort=False, random_state=None,\n",
       "            splitter='best'),\n",
       "         bootstrap=True, bootstrap_features=False, max_features=1.0,\n",
       "         max_samples=100, n_estimators=500, n_jobs=1, oob_score=True,\n",
       "         random_state=None, verbose=0, warm_start=False)"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.tree import DecisionTreeClassifier\n",
    "from sklearn.ensemble import BaggingClassifier\n",
    "\n",
    "bagging_clf = BaggingClassifier(DecisionTreeClassifier(),\n",
    "                               n_estimators=500, \n",
    "                               max_samples=100,  \n",
    "                               bootstrap=True,\n",
    "                               oob_score=True)  # 这里开启OOB\n",
    "\n",
    "bagging_clf.fit(X_train,y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9306666666666666"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bagging_clf.oob_score_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Bagging极易并行化处理(n_jobs=-1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Wall time: 978 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "bagging_clf = BaggingClassifier(DecisionTreeClassifier(),\n",
    "                               n_estimators=500, \n",
    "                               max_samples=100,  \n",
    "                               bootstrap=True,\n",
    "                               oob_score=True)  # 这里开启OOB\n",
    "\n",
    "bagging_clf.fit(X_train,y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Wall time: 2.9 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "bagging_clf = BaggingClassifier(DecisionTreeClassifier(),\n",
    "                               n_estimators=500, \n",
    "                               max_samples=100,  \n",
    "                               bootstrap=True,\n",
    "                               oob_score=True,\n",
    "                               n_jobs=-1)\n",
    "\n",
    "bagging_clf.fit(X_train,y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Bagging的更多探讨 \n",
    "(如果都是基于Decision Tree, 那么就是随机森林啊)\n",
    "* 针对特征进行随机采样 Random Subspaces\n",
    "* 既针对样本, 又针对特征进行随机采样 Random Patches"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### bootstrap_features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "ename": "ValueError",
     "evalue": "max_samples must be in (0, n_samples]",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mValueError\u001b[0m                                Traceback (most recent call last)",
      "\u001b[1;32m<ipython-input-21-d045c1d60368>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m()\u001b[0m\n\u001b[0;32m      8\u001b[0m                                 \u001b[1;31m# 对特征采样的数目 和 放回采样\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      9\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m---> 10\u001b[1;33m \u001b[0mrandom_subspaces_clf\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mfit\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mX_train\u001b[0m\u001b[1;33m,\u001b[0m\u001b[0my_train\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m     11\u001b[0m \u001b[0mrandom_subspaces_clf\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0moob_score_\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\Anaconda3\\lib\\site-packages\\sklearn\\ensemble\\bagging.py\u001b[0m in \u001b[0;36mfit\u001b[1;34m(self, X, y, sample_weight)\u001b[0m\n\u001b[0;32m    245\u001b[0m             \u001b[0mReturns\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    246\u001b[0m         \"\"\"\n\u001b[1;32m--> 247\u001b[1;33m         \u001b[1;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_fit\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mX\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0my\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mmax_samples\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0msample_weight\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0msample_weight\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    248\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    249\u001b[0m     \u001b[1;32mdef\u001b[0m \u001b[0m_fit\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mself\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mX\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0my\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mmax_samples\u001b[0m\u001b[1;33m=\u001b[0m\u001b[1;32mNone\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mmax_depth\u001b[0m\u001b[1;33m=\u001b[0m\u001b[1;32mNone\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0msample_weight\u001b[0m\u001b[1;33m=\u001b[0m\u001b[1;32mNone\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\Anaconda3\\lib\\site-packages\\sklearn\\ensemble\\bagging.py\u001b[0m in \u001b[0;36m_fit\u001b[1;34m(self, X, y, max_samples, max_depth, sample_weight)\u001b[0m\n\u001b[0;32m    304\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    305\u001b[0m         \u001b[1;32mif\u001b[0m \u001b[1;32mnot\u001b[0m \u001b[1;33m(\u001b[0m\u001b[1;36m0\u001b[0m \u001b[1;33m<\u001b[0m \u001b[0mmax_samples\u001b[0m \u001b[1;33m<=\u001b[0m \u001b[0mX\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mshape\u001b[0m\u001b[1;33m[\u001b[0m\u001b[1;36m0\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 306\u001b[1;33m             \u001b[1;32mraise\u001b[0m \u001b[0mValueError\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34m\"max_samples must be in (0, n_samples]\"\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    307\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    308\u001b[0m         \u001b[1;31m# Store validated integer row sampling value\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;31mValueError\u001b[0m: max_samples must be in (0, n_samples]"
     ]
    }
   ],
   "source": [
    "# 报错啊, max_samples 超范围啦?\n",
    "random_subspaces_clf = BaggingClassifier(DecisionTreeClassifier(),\n",
    "                               n_estimators=500, \n",
    "                               max_samples=500,  # 这里也可以改改\n",
    "                               bootstrap=True,\n",
    "                               oob_score=True,\n",
    "                               n_jobs=-1,\n",
    "                               max_features=1, bootstrap_features=True) \n",
    "                                # 对特征采样的数目 和 放回采样\n",
    "\n",
    "random_subspaces_clf.fit(X_train,y_train)\n",
    "random_subspaces_clf.oob_score_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.872"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "random_patches_clf = BaggingClassifier(DecisionTreeClassifier(),\n",
    "                               n_estimators=500, \n",
    "                               max_samples=100,\n",
    "                               bootstrap=True,\n",
    "                               oob_score=True,\n",
    "                               n_jobs=-1,\n",
    "                               max_features=1, bootstrap_features=True) \n",
    "                                # 对特征采样的数目 和 放回采样\n",
    "\n",
    "random_patches_clf.fit(X_train,y_train)\n",
    "random_patches_clf.oob_score_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 随机森林\n",
    "#### sklearn封装的随机森林类是在随机特征子集上寻找最优划分特征, 差异性好"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "RandomForestClassifier(bootstrap=True, class_weight=None, criterion='gini',\n",
       "            max_depth=None, max_features='auto', max_leaf_nodes=None,\n",
       "            min_impurity_decrease=0.0, min_impurity_split=None,\n",
       "            min_samples_leaf=1, min_samples_split=2,\n",
       "            min_weight_fraction_leaf=0.0, n_estimators=500, n_jobs=-1,\n",
       "            oob_score=True, random_state=666, verbose=0, warm_start=False)"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.ensemble import RandomForestClassifier\n",
    "\n",
    "rf_clf = RandomForestClassifier(n_estimators=500, random_state=666, #500棵树\n",
    "                               oob_score=True, n_jobs=-1) # 有放回的bagging都有oob_score\n",
    "rf_clf.fit(X, y)\n",
    "# 有些参数可以调, 随机森林是集合了bagging和决策树的参数, 比如max_leaf可以限制每棵树最多有多少叶子节点"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.896"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rf_clf.oob_score_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Extra-Tree\n",
    "### 特点: 决策树在节点划分上, 使用随机的特征和随机的阈值(差异性更大)\n",
    "#### 优点: \n",
    "* (1)提供额外的随机性, 抑制了过拟合; \n",
    "* (2) 更快的训练速度\n",
    "\n",
    "#### 缺点: 增大了bias"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.ensemble import ExtraTreesClassifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "ExtraTreesClassifier(bootstrap=True, class_weight=None, criterion='gini',\n",
       "           max_depth=None, max_features='auto', max_leaf_nodes=None,\n",
       "           min_impurity_decrease=0.0, min_impurity_split=None,\n",
       "           min_samples_leaf=1, min_samples_split=2,\n",
       "           min_weight_fraction_leaf=0.0, n_estimators=500, n_jobs=1,\n",
       "           oob_score=True, random_state=666, verbose=0, warm_start=False)"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ext_clf = ExtraTreesClassifier(n_estimators=500, bootstrap=True,\n",
    "                               oob_score=True, random_state=666)\n",
    "ext_clf.fit(X, y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.892"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ext_clf.oob_score_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 集成学习解决回归问题"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.ensemble import BaggingRegressor\n",
    "from sklearn.ensemble import RandomForestRegressor\n",
    "from sklearn.ensemble import ExtraTreesRegressor"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}