From 382edcbc28aa17b77720974be9ea9560d93180e4 Mon Sep 17 00:00:00 2001 From: "zhangruiskyline@gmail.com" Date: Tue, 7 Aug 2018 21:38:03 -0700 Subject: [PATCH] ML system --- doc/system.md | 102 +++++++++++++++++++++++++++++++++++- img/comp_graph_backward.png | Bin 0 -> 21948 bytes 2 files changed, 100 insertions(+), 2 deletions(-) create mode 100644 img/comp_graph_backward.png diff --git a/doc/system.md b/doc/system.md index 7707b6f..c635aad 100644 --- a/doc/system.md +++ b/doc/system.md @@ -110,6 +110,97 @@ ![DL_sys](https://github.com/zhangruiskyline/DeepLearning_Intro/blob/master/img/DL_sys.png) +# Deep Learning Programming Style + +## Symbolic vs. Imperative Programs + +* Whether to build networks with bigger (more abstract) or more atomic operations. +* Whether to build networks with bigger (more abstract) or more atomic operations. + +If you are a Python or C++ programmer, then you’re already familiar with imperative programs. Imperative-style programs perform computation as you run them. Most code you write in Python is imperative, as is the following NumPy snippet. +``` +import numpy as np +a = np.ones(10) +b = np.ones(10) * 2 +c = b * a +d = c + 1 +``` + + +When the program executes c = b * a, it runs the actual numerical computation. + +Symbolic programs are a bit different. With symbolic-style programs, we first define a (potentially complex) function abstractly. When defining the function, no actual numerical computation takes place. We define the abstract function in terms of placeholder values. Then we can compile the function, and evaluate it given real inputs. In the following example, we rewrite the imperative program from above as a symbolic-style program: +``` +A = Variable('A') +B = Variable('B') +C = B * A +D = C + Constant(1) +# compiles the function +f = compile(D) +d = f(A=np.ones(10), B=np.ones(10)*2) +``` + +As you can see, in the symbolic version, when C = B * A is executed, no computation occurs. Instead, this operation generates a computation graph (also called a symbolic graph) that represents the computation. The following figure shows a computation graph to compute D. + +Most symbolic-style programs contain, either explicitly or implicitly, a compile step. This converts the computation graph into a function that we can later call. In the above example, numerical computation only occurs in the last line of code. The defining characteristic of symbolic programs is their clear separation between building the computation graph and executing it. For neural networks, we typically define the entire model as a single compute graph. + +Among other popular deep learning libraries, Torch, Chainer, and Minerva embrace the imperative style. Examples of symbolic-style deep learning libraries include Theano, CGT, and TensorFlow. We might also view libraries like CXXNet and Caffe, which rely on configuration files, as symbolic-style libraries. In this interpretation, we’d consider the content of the configuration file as defining the computation graph. + +Now that you understand the difference between these two programming models, let’s compare the advantages of each. + +### Imperative Programs Tend to be More Flexible + +### Symbolic Programs Tend to be More Efficient + +``` +import numpy as np +a = np.ones(10) +b = np.ones(10) * 2 +c = b * a +d = c + 1 +... +``` + +Assume that each cell in the array occupies 8 bytes of memory. How much memory do you need to execute this program in the Python console? + +As an imperative program we need to allocate memory at each line. That leaves us allocating 4 arrays of size 10. So we’ll need 4 * 10 * 8 = 320 bytes. On the other hand, if we built a computation graph, and knew in advance that we only needed d, we could reuse the memory originally allocated for intermediate values. For example, by performing computations in-place, we might recycle the bits allocated for b to store c. And we might recycle the bits allocated for c to store d. In the end we could cut our memory requirement in half, requiring just 2 * 10 * 8 = 160 bytes. + +Symbolic programs are more restricted. When we call compile on D, we tell the system that only the value of d is needed. The intermediate values of the computation, in this case c, is then invisible to us. + +We benefit because the symbolic programs can then safely reuse the memory for in-place computation. But on the other hand, if we later decide that we need to access c, we’re out of luck. So imperative programs are better prepared to encounter all possible demands. If we ran the imperative version of the code in a Python console, we could inspect any of the intermediate variables in the future. + +Symbolic programs can also perform another kind of optimization, called operation folding. Returning to our toy example, the multiplication and addition operations can be folded into one operation, as shown in the following graph. If the computation runs on a GPU processor, one GPU kernel will be executed, instead of two. In fact, this is one way we hand-craft operations in optimized libraries, such as CXXNet and Caffe. Operation folding improves computation efficiency + +## Case Study: Backprop and AutoDiff + +``` +A = Variable('A') +B = Variable('B') +C = B * A +D = C + Constant(1) +# get gradient node. +gA, gB = D.grad(wrt=[A, B]) +# compiles the gradient function. +f = compile([gA, gB]) +grad_a, grad_b = f(A=np.ones(10), B=np.ones(10)*2) +``` + +The grad function of **D** generates a backward computation graph, and returns a gradient node, **gA**, **gB**, which correspond to the red nodes in the following figure. + +![comp_graph_backward](https://github.com/zhangruiskyline/DeepLearning_Intro/blob/master/img/comp_graph_backward.png) + +The imperative program actually does the same thing as the symbolic program. It implicitly saves a backward computation graph in the grad closure. When you invoked d.grad, you start from d(D), backtrack through the graph to compute the gradient, and collect the results. + +The gradient calculations in both symbolic and imperative programming follow the same pattern. What’s the difference then? Recall the be prepared to encounter all possible demands requirement of imperative programs. If you are creating an array library that supports automatic differentiation, you have to keep the grad closure along with the computation. This means that none of the history variables can be garbage-collected because they are referenced by variable d by way of function closure. + +What if you want to compute only the value of d, and don’t want the gradient value? In symbolic programming, you declare this with f=compiled([D]). This also declares the boundary of computation, telling the system that you want to compute only the forward pass. As a result, the system can free the memory of previous results, and share the memory between inputs and outputs. + +Imagine running a deep neural network with n layers. If you are running only the forward pass, not the backward(gradient) pass, you need to allocate only two copies of temporal space to store the values of the intermediate layers, instead of n copies of them. However, because imperative programs need to be prepared to encounter all possible demands of getting the gradient, they have to store the intermediate values, which requires n copies of temporal space. + +As you can see, the level of optimization depends on the restrictions on what you can do. Symbolic programs ask you to clearly specify these restrictions when you compile the graph. One the other hand, imperative programs must be prepared for a wider range of demands. Symbolic programs have a natural advantage because they know more about what you do and don’t want. + +https://mxnet.incubator.apache.org/architecture/program_model.html + # GPU in DeepLearning ## GPU architecture @@ -405,9 +496,9 @@ Why do we need automatic differentiation that extends the graph instead of backp ### DAG based scheduler -''' +``` engine.push(lambda op, deps=[]) -''' +``` * Explicit push operation and its dependencies * Can reuse the computation graph structure @@ -444,6 +535,13 @@ Upon building this queue, the engine sees that the first two green blocks at the ## Model parallelization +## Summary + +* Automatic scheduling makes parallelization easier +* Mutation aware interface to handle resource contention +* Queue based scheduling algorithm + + # Distributed Machine Learning http://dlsys.cs.washington.edu/schedule diff --git a/img/comp_graph_backward.png b/img/comp_graph_backward.png new file mode 100644 index 0000000000000000000000000000000000000000..5662cd9d7e0b190611fc519d07ca9038aab250ae GIT binary patch literal 21948 zcmbSzWmFYR7p@?Rv~-KoDcvpIAl=>F-3Zd%-QC^YDcv9v(%p5(@4Y|n@9RQ^vygMn z%$~iU8iQq}MG#EG{5rNB_H;Qs%g8T|DTF31OvWiRTe^-h} z$MUsAc^QPPU24;i4nILxwsh7oCAz%j=FV zu=RWIuW#dE@v}~fIKAU-^VBZIe9rWA@Fhju_WWa$3lW+uND6Hm>gF? zN#Xy#FJ$YB0yp}ZIZCKi^Pn;RV8 zNgiBmY%wXR^o$Ia9{bqnXx(%hWJzi1hWdI%Ma8-Kc|-&R>}q?Pqs8i*8z-X8_d$sG zH3bB?xDpZ)t)7n!$S+)FFs~#eB;(`bD7oYziMjt?@>Z;%urQ%FgU6qN@UyeC#l^)S zBm(g;81a;}G>dQ<85ty4SPt)(He@X}W@hF|m7(F`J4s^i))%KONsT z_+gLG84WuLiHOKZNr}NrN=iogj7Nrq=SSo?`* z3T&1C#QY{8;E!x*Y+(^Ll#qZy)W-KwN>1hrcv~DC%O6zupy1%(4Byq%)SO8>pFnp} zo12?sH`)Gqk2=BvD=#lkoe1@z2ul9@_eBhBu+%gZ6xC!3X#MVT{|e(AdwF@;&d%=Q z;^M1jOF%-f)v^VP#aJ^ODlZ%?ETOG}f`YJcAo62eUteEd9x>5b99ZTX&kR*VD@IDnEb!y_QzA@$jEN=^#ULT&Vt6 zmmU0SYB-eG1O#A{N8f}579mI8z)(#8| z7#SI1MN!QvEw8Ku8&kTH2k9Qha&&o-PN)6$<14~%jP(r)^L{>0&-Ok6505tHoie&$PIeJFNZ;sa3>JK z@-r0`)z`0xmu%iI9(HyNi1df{jHHW-f`U+x0qJ-g4x)HJhr@q)L1!4Z&r@#=-AGGH zvOQw(h!}thaC37L5D>7orsW*;z|%@n7({jPO;QqmP$H0wj7*h& z*LqIyR=0Ib)z{xm&E4L(U4IT@PcxgGXK-_KvxbXqFnK4v(End2(t^k*#vtrvWMmu) zKm7U?e(s^b7S_D3x4!&==?LKvF43`FT)Y=xoAp8?N#o7r@qwb+d6-JfWkt@RZ&y~N37pyU~n)s zB+x{Sin2L)CVQlN`b_jts=B_uzHe-~n4PwWY$E8Sv9WP)dm9QGx)w3CwT{eOT~6-H zisA8DL};jllvKPy*OrPdOIvd@_MB00Ufn~*m^c9=tcn-&qW5Nfy5SF8*qQ~4QRzx| zipOVE-j{`-SG_PN7}W?op@D)mk_G4?>olc4K|w)L&Y%P&{uF@=1p@kAce8O_vCo&L zrlwzb8PO4V*{@BO>Bkm}ipYnzYtTY?5+I@2BUMlf>|CfLF>nfDW~Ql`Ds|}i;87Wr zQlcfl!sVZf<;A@X+p?(nns}Y<6aQm(XNNmE5&Bm$V{Y9k!~D5pi%m75nVN?L z1H&&KR+afJ-slMOvERY$vnxH>g~+om4tsqTGx=wOqwm{eA6A^3>RWC3Wj%2Y&&L># zTuUnQx%NpG4h^l!*c}@@5`QA-?kchGPNh0mey>eI@03+h8Cq0TCk^qoKV5HM(4wSX zdh%zi^l)!FIqWQ!z*>!6lsiu!q1o7osV{gP6C+x~iy9na&#_{%8rJNasvuIg+X78{ z>WpX-^!(UX6GSnhrR72sM8@(}ECKzb;e7hAZDuAWI+>bAk5yd@3wTb0i5fI9;ou-< zhOV-*a4`s5L2hgvZdF)gyh6t4>23WNyKMhyK82X%TDh?hN0#td(S%W!X_r``Dmm)7 zYu!ctf75XbUDoP5&v%^flJZqs(RQZsc9&wxh3`t$>B6#Pon{KOH7ZlB`4RYOL>>DB z8L3;hp6)M?=r0*#W^HPD-dnXd@R)S+c9*eX!iE~Dnek8>9sX8f`rdkbbyan-`=x-# zhy?%Bx-l9|S|qc5SO1=yckI?+l(D$weK{?O3{&6hemA4{U^RU5xe z<2!=p2(AHD;bOwvGxp0|_@-R*YqW5EyUPO1fvwy$hl@A)T`DON^Z9ht_C{qc+33_W zXo9~)(Buu!*l0nk7Gf1L@SQVKO8lys17obci>jibub<1bko?w|o>O+(9sVCeXoG}R z8ZmsK;G^guA|%9ns7uRE%f}+&uJ5n;5Oo@pks&R#wafT&N2rL|I`QbNrmOqqIK4PG0ci z^z`&tK{_^Xq8kIv+xcN3?%*$>3_3l@CQ?R-jHRk`^1Bv@u>)?gjd$cb*~#$qhDu`z zIddv`_^F%qYONOOUmBylH45dP;>VX%6K`LF{jE5j z)*pXgf?>!)YU5`Y_(aqjM>lbxs;+L{q3NqD&%IOdnNNLBak9FazHO?FP)irlMAVirEsg7ORsbLNQ31CyPxfnNq99ss^cs^x}&^8BWaWDut1VqiPC6tV_b% z(=EWNKogG7yjcgT8F1ip5d;}i5-O)8Z zVuk5*`0Jf|tAz8-?JX`UHC`MOJ-zL;w77VPYr3W?9-NYlvla1o3_*Zk9UL5{W+EYC z3a>|^QGBL^_cXH5Cwv^7CPJ_LQ@GP5^TFbvn5=escNP`>eI`xyw;efcgzk70UyArM z7>Qeq`R<*Bs;X*OdPoQ?jd8+I&undBA+$h_9$n9f-A8QfV229=WMtjbCfAD&4%Skw ze;^&k#%jGyRaRE^v3ITW&h+=|;;Uttea^XhqF2n^(?f+{xjH8&XShR%bJd&cw5m>8 zSo)a8c*@at(=ZHNEG(q7%uLoK|04kX%t)WO_~=JRM;8`{Cnq;*jb)UTG0%lz`^{B+ z+zRsY7F)M7g54h;7aCgMN3@C5HXCx7x+B)F%i2O#7y6+?EN zLA5zl1_p+oajZlCk&8P7tKz+@tGd;|#g!E*&&*F=@8Z0BU+TQv-HeRhqawh-0XO{A}Q%trN_<*X6Ml9!|C&S|6T^?wTQREl|Mn;&MxBX!-B9G zSoOx_PS9R((a_Lva40JkIgGr+Yk5rTG*+v;YpIBdp&r8`Bas@s+c^NHv%XT$F)!r9sR{NWmF&)A@F$t$w+?8hiF@;zM3u5Ysb$jJ^C8s(z)vHK zE(gs}PL}+4x_oyxV$1$h>-ACX;vRZCm8M+W4>}ndl66xO@UoyB)~&Gm@ddeSYiiy@ zLk9&0(oMw0p;aR=^#F)OjDUb(;NQ@|@`9ZpdT)_}ZVv6LsED3C_<=kG*J8D)8gPvI z`s=O!Q0J4?F9qbhuXcg~{)Q9RhcEuTMeEJ-M)`o4Azl_4YiQujX#rZ{>B${K+>#82 z0x`c>F7*!p=AB3fbOMgU!omQa+5GsBC)edCJ3a76L!XV3Qv9Uu{X{ESrEk5C*Ci(4 zBtPjPL$=;up!AK6jUnK&&_$)Dj=pw2O`Nik?`m|aPx7c&d9~Big3V#I$ zZ6suolSY2xZgD)s$anMh_I7uF{M{P_s8ncZ=<#l|(eZNq($dmt+a==o+EZKG@4XCr z*|BCXPwi(jL#v8v`2YJ7*xA`9l%V|m{rd*{`ml5jVN~W~zjantf|(-wiRF7Uk_xf% zkS8YTiGwd0V2)&DxDX^~E>CyoYpu?enypU1dw$3lO5?NJcO*!fsNpA2LP_onZYS62>~zu!ZH^P{J)fBlY6!dOr5+j>71IC1FcKP*3|Uwon> zp~J?-CE|^Xi^KXo$>V-^_GKwIC&y2B*?k*zq9Z9OX=q3^u$-m~K>O#1&J73|ImXVXO_AB%C zmi?q|GLqM=b3fbpFSzLF#|-cFg!HwI6?4ffP~|L_Tt<21Awd+>)PhX-%Z@NGFe(Me zNeI&m^Yb}*cSQoqMO)*b>3e&4JdMOtwYRsE$z*`l(Q$NS_DYV9j?T=~*3#;*-ss2` zi6r!U`}QqpYtEqOLS5Z>hy*%$5ETL~s41a_Rf+hhc2q;}! zXkVY;o(xM}eSNdnAAS*$!PvpIoppf(P^Bo->MZmmsJW%lVeyulOI9YAJ{QHYYU(cG5mDL&hQ3(u0NdQIqS*Ic- zp))+Ro~&OAIvRHKw{k4i8!f1Y+>e74%gf74uU%~okPQX#eu=pD6*e?rlnZ{9lN3Z< zVqt)^nL^?e3hiw&&AbHL?<*<81$wfIFQ0`oN=ym+znmVFIso5O#c?UTY4)d#nlhY?|H;0W5&z+r}=H}*IBA&aQ z;aD5i;)rWGPv6w+R~B|L_D^spqDD6%!ZoY@S z%+=FoqwxZ%RIt2ef)pXzHegkgld*}adbm99Y*aI*Z?-vTY5(^Y;dyzVPi~GdMK^rH{pri#XPGoMU7js}&7HlyQKD@ZGFYm}yT6tZIT-*Is5e-g z7p`mcBYhUIE9rQ7IBaz~UN$lYTnUsD9svRC_onS+1ypu%&Bp2DVc8P(J(ca#E zbYx^3IdfuS0<5(Lc&?MRRyW!|8fBo8s;a6Q8yEURzoGYEPPvUuOcY#;QrV783=()h z{SG2{xhmuxA06%G-y5&~>h>`bpR=LuR6^-m;UD}qR;e`_5AS^P#z(WS*qD1bmDT+> zPwMLG8X9iSe)k?9A3r}ogZ{1c_@Oyb;CVcw?^k$LjtfNE?A;DorpLu(Ej>H?63~Pc z6t*Aw4VpHOR~o_eqNJouQU?l!xHv4Vgd{*>Kp zFgZ$8Sby>X1tT#78JWnucucjdQXqPj%z;o-X~u;g>-~}SenpJs5A{a$?asQ=yRMh# zwY4=?>vg7S3b?)&edqY2Ri!9p1r^yBJ5_mk)bj_~+8mXo8dJ|#bNlrj0DxC7H-I=3 zP7!ao1v*8GGJ0iBR@NqF=iLw86#KJzb!&WHuhk?S7QsJ7PGGXEEH4*%01*nXqfa~@ zD%#qUZ(u_|Z)GLnMHd;nxpAOQcl6)8xx1UrjL^Lu3-JZ*`a1*6+dN_^S=otC`nUz!Pe%&{bfyH`cRk6)~ ztpu}Vb#FLgtYYm2k03FNUJU<(DZO29yuof$_JA+>})T(Tp)W)x;0)0wEw&0-%0j# z92{n~OJOiZy$X{}O>Du@h|=%J;7-OMs)-N)Jopm2_$iky?uH`6q{va_fi0P)U?}1n z-*`S-DUf2z*s1ja{~be=h(LZHtc^lJKgLK*9MG(&prBJEGhbX8>1c#(z=$aR{X2@t zNbl0H)UMp*Xd*49B^6gKuXj#&wL#+x*2=5eRw*Y7`f1%-i{ogLN;0k!_4IvXWq!+g zwfaHR2aZ&(w^Dr<`N@7@6^9qt-4?fhp$DKG&yd< z$Hvyp9*dF#Xnjn+MWLDi!#HI2Vj76*Q&T!@yB1YSWj=?unm_l(OMD94YJLNJcoQwt zasDN;opI!co1MXu(Dd~X@&6X*tF*OKS}e)-C)t~FCKMBvp|AZ%JJY;XKOY|4P@=Rj z#3A?b;-R$ssIIBMe)W3QiKgJQdP&jVAQ==$4&B0EnQYrSk#D&$r+Hjo=dMt*2toP4 z<#tt6SBF;K=RT3nt@(H$;mWRhgZWan)MnZ{N+R^=eHG}XwfYv1&4!ONUen(|)R{D9 zuxj;~Ea?D71(HFx9b=5#VQQR56f=uRrK~rne6s-92|jYe;lTq<&AK~6#nsrm)=o>1 z>%A(or~-sTt7BwFSfuOMB&R|G)o0TZwCQZNJ#^!fm*;~7f6_0Fl%4$i(NcP`T!Cba zAqt6zVYRTbGFmXq?=UZQNS7M00lA#^3BuQ_hsac7L{FxY7 zm0+%o%gWHBi9f5VsihG24q_cnL|+qE%*IZ8c)YUW;hC&&42IJJ$cR-dYvhlA_yU&X z(eZepao38WKsk38`dqYfFzPYmpzr77Y*a-(eUL4blcX2`GdjiIn@> zw9mxEdBDu^v)P{}DL|jB^6S|mB&F`bqQ|wHB;fE%PfCwN)Dilt^X(gCPH&gStvMtv z;Yz<+b}DuHP)Z8U{OSSI5Jo624h}<4CiJNRFP87l?s~iXQ7@o2NW4#HGcvsWY+#mi zvavOJzrLh%Is;^lwZ9uk*H+4E_tu*^hL=8MAyRb|}H9;W50N}Z{E{rJ4w zT@|OX2C#Xyehl{y4oW!dYikpngF$>TRSlyr#63Lr^Bu)F+{rg^v|1NFaJ}Ov&JVs< z2w@y!p`;9fYLe5?NOSOK_x}d0l;7kZfl{@A@B0^yYE0Od8fX7Y291=obOO`o^wxs! zoc==5I$!BEIRJgr3#x?#Fv$E!8^5%)^zig_p)j-kqw<04&C$~Iu`7`^F`G$L=tj_7 z@DeWT-`>ay_LUjUn8%z)PN%EzO3P8L+(hKJuqpdHt=bi~gv7bkj zPrS}UpDJG0Pc=PGn&rK(M1*+n_(3x|tMsu7Y83t68Nx~0)c6L^@jSt=~ zxaCJcz*vbnqEe%*nYnqikV68x4iEJx7Cmd@F%|^e1zzJ)uiMcgy^=~?$`WBE3KD!R zbRMde65y0c^XV&M=$(V{Zr=cX)tY$k-165;KJn~Nda-Ns(n5tMBdOBv?l_T@dd+-O zt-ia~a{~?QpIoGM)&d1cpA*JH=WT1eyHMhwF`XAuM33X%+@1UTdj-htlobHHcPQ^R3i2FAhJ8M9J5(pK+B?aL?c-*AaUgcsT#)raZMVWvDXZf-60 z=9dHExF#5{Tk7+rr6{PVsIkC2#K#|}LnKSwl>%zf80RYn?7%R5HQ%`!5bDRU+WBDN z;L1^`S4a+qeH2qvfB!Qe>7XUQYinCW#l125bR7T&>glp-TxRhLc18axT&$RGpnm(eky7H%PmMO$=6rG z16D9oQ@3&oBtmZRjfG;HdsPev-g z>Zbt=y4q;1?d8QoBZ@AIDSM!(q@<**%-9v-S$j`*BwmN+?Gr zJMoZpAMy{iZwa3GuP4$6+NiT4=hwS|^35di34tG~9CNk&hIDpe0ad?!jq%Hu?)&%S z0Qm#$z%?N43h_1)HjOmT7E4GHczVbxpIu#D&(F_=H!rCOFffeTH#4H6yF;XSJ$>(c zgOISz*T~$uHR)}Yxz~(6JXX=B6scoOyx3Q@{vcEdNT+iM)jl3A)e6Vuczh}mzSZ_Q zOYOy0$hlz&1+-ZQh&YGy<+&w#ce=7s{(2MF|75avhYPrSCGT7V-=_;r&`G-zie!9u z{)>p_Zf<2|WFa8%HbHDEyK}1hU^NJvfi(zGj(Yw#viof}L-H5M5J@Bi1Xb<1eH#Hv z{pcfrB3TfUBSJz5^Bt)cwe-fw>f`y0?&st`Lewu=WO#RkJxRHTvXCDD7_fh~y-8{J z{BR9KwU!o+<52~TD#Jg^UbnpRT9z>s**VC#!VhH;}9EWeT-bi5j zR`b{|6vI(4;H%RijjE`zsfmgAn%2#~dah zvSYRO`F%03hRz&|%iFalN!MQq)Vwc}?Bk+p<{ggQtW%qR!GvCIbDjC%&I-gt)mA4< zBO}uM^FRb#tdqk-b8G8Ljn?_}h{@tjofSA*jVcW)XB|B~$c-L+Wa>bn+Q7g-S^8hh zzF}*?ht2`cb2vF1TJwk1Ewu5MoIKTzB~N`*Yp1wly&vT(%q%Q{F&wO1DhZcP^yt?N z3_1bvT}Fm;@^tTJ86Al;96%qHhQU!uqp8gZ$_N-YUI7Hh=JmysXaqCMCVM{dhAKZ0 zxwMRn@%7<#F2U$@;`xOulVanMtbrzZ%~%>Y_8xyx(#opBD}LdTe=Xt@w;5Dh29ZdF z#Y%(J*QSNJIb?tC)6&XHwa%A`#E$LFl=+Rj#7>R0g!$)_R))-X(Xp|p7O#2X$Qf_N zH7VgjG@Oc&ch;&J6zUb4gk2De_yH(C-&a*shvDPO&VYO`eEA`@+2g(jxt?J&1}_C9jc{}KP@eR5ZJ+0Qdp`0l4TNEqUS?+QFz&>Q zc~>9*NDDtq^}0ayK1TMqy7EsJRyo%*IW{&nPQ{FCPXl7PJm;n#lf{_hF4w`)(b&XT znS@mP-KxFHuK&ifCcU)B)i;Cfj=juAL*rXcPu(FY@^3qH=M`(E!08sa&Nl8^ssN@y z!{{|=%?eCQngQjJ1&j5n28)#e0>#9ag9}=~OToxz zk2jgcM!iFR=$72b$$dQFp zR_ZnA+K$>aSkk>vXV&qR#9`rk@lnBZ_zWePZZz2iDH4L6m*+<`ko)N8Ifepqer8F= z`WQHx#p1)(8J%=%7K0zCsp63SJ+WU+MP1$O2Nx2cDACZ;?ny>_wcA@dySCtk0x(j^NP-jA|sb3obyjO)*z zKf-aRsGmN4!o#zGLZdY@Gb<@B4mOH}K8XwiHX+C+NK1p9hk;KPz230wXNCJ0Xhy)5 z7isF2!fbSGZjMxG+eG~EgksCGviLe*D#_RVmC+)UAg<-Ca~Ja!rQ(Mhu;jcT=8O#t z@{ncQJ{>5}mbbNeI5^DyRD*bUcrY_F^E(3S%+=SMB{<`!P5)#t>OgWMjSXZsgcWqo zHoN_gjw~u$6rk5jwU*o_czE@_f9xkMbE&rl;5}BY1a%MBY@) zpRO%i=E4rtDRLn2*=%-=c(#28=bwj%hm>^S!E=vfi-2HEun&mWJGMqdlsas7+e5|j zg<)N0j9gq}s+-)0AVU_sfF1!R_`%7^4Gcto{PSaJ66+CxN39p7)Hym^B*^{%A^2YV zW+eiw2O$v=^f6r(wM;g*%h?;bCb#PnU28cXS{ z7k^h(ZMLJGun&5?Jp~%F9nXW?ZSz(=^_I}W=!Mmh{$ND;1DRa{TxADwfRB@t)9&sr z3p2B8q)&|!&x7LF55f(cdAO64Dx>ifFreM`q%gsO10pG#?*fnMZ*PJz;D`q5X*Wy`{TP-1ZOi2>l>6p%s zH4_Sj6;IS6BBj{@&dDJj`}DPP;s61I~tx4c(=n4)LcXmVZ7H zB4eEM8=~}!BO?U`Gz8`5h6x49QNQh(`J7ZBX9DR`5ZG6v2;*g3^fjbZSNV}U{l>HM zU9W2D>Y#clmboah{*q4b=W6$BK&8D+Q`^>$1>T;18Ut=l#=3Q;&E5pQcrTFELNRG^ z|GY!+y z(gewc{!Fz3s1cYxC;vWAF_>p7?n2iM3fa)0OBKXwz6EK?Br9>nOkj4CRjyPKOJH_+~F<&ovn&>s!nAfS8W19z30+MsLnoc0*)DU%$c_5YV@!obC*P8g9=L{YjP!9m0_E%MQ~h}CsQs$5s{wLWvI1`uqmj{F7A+b8yeFz!U2V1x>bVDf+-u&jg8Ii+ zP7W|-&Cx*=Qo$Bi;8t0dJ24Vkf@VO&A2)5CtExj6nYm zN*jIyBBoJE5Tr-@`_ZDhO4pL0Sy|7N6&3M$@36d~5GGfwQWr;`PQtG%W5F4LEi5b? z-5>ae9EOQ~7q%|JWJO7abOS{H`fy=pS4)FCq}gh*$~RX_S-H^q?-_d)pTKk58|`YD zO?blIT(}I^kci9K1>U6MroaSvGZUzUcyJKD&+5MiGxZ4MK(_mwI}iV1wcf0(3&+D< zx1#F2JSf$^`;VN>a=54nNJz$5*#>bR^h88NR8+7?Ll_wN@-&V}GpY^T$0AjcCp`qY z~VqJ*v&5|?20qIJW5Uv?8IG1B!lR_d8YYh)u4YF z&$Qm>6c7>n=O8FiD{L7X86{;?3r)Wn{<*)uuUm$^(|;H52D!7Q#!3F+mB8c77ke_B z!{hD@w0UR1z{V$lspNeToB7dt-Y?EvGATNm|M+ob<3))O9ooS7)6@A&$;jFY+~p;U z^_H^5ic{8a+Hy{N#WR*`sEryKv)fwZ$=&4*y#Zeqy+ov4u|HGRJYW;Eii?MrXKDq( z4_E%rSdAY_2=PAKmakB6nDu;pTzFaQY<;zUr5U*YyYZA-T~+nxb)Of#Onu>gUR}6l z$7f`M>;bbK)mvVM&-4BJ_cr3&F*tFgLjd++^SGRSc02%17nf0xFHr8PK!XLnl#wx( zsg8@qdi}_^aMv>t&{kHh*U<4)2Dq8n2G7${&e$IDj%#Y(_YK~+%{{!2otFz9wEAny zp4PMLs`;z|?%8gB)=Olvt+^R#JZTK0b}PZfa9%fHE8i_tYSU6uf_dxTAnxVW(YcOS zMoomqyd8T{s7!mw$auSGp-Co9?H+1qU|@jGySj6Dm+hvue|ky_V3Gaq*!FgwE<`QA zd~trhVQ(M;@9X_!I0ZV{*XN6^-S9)WPR#3RRm!@y`4Ha_^T{3zg0jO#jle z0n}3knKSlBsClQ!3~m5;5!2(~X!^utawE8>4xI_c6g1?)3iq`J4buSQ4($&B8h~(< z^F$<;kQKP}O6%(r^$H(Bf<6pF(Svt)O>*~>(EFDc7j-_V=uXlT517uLjD7q}dk>bc z6Q|Z=8i)7k`}GGf)*bgoQc~g7IUR2gKq?J%^gJ?L0vbxn7U$E9Fvpwh}Y3^U+!2Ey2J%`7^!3nd4%WG=7d%oRZ zE%Cf{IV^9yIaxE8tW+VV9NI{D@@BgO!bNxHtF6>EQusdikA;%Ge2GMp#i!?u*Y@fh zo_?TswYId_+1pp|0m~5+14Btsv3bLFxK6Ss8?aUyPTi$S4+^bO;NlOlgu4H#;fk1F zU$0o(4%sqxNp#EL_2yw>LR$;_`BM}~-nHJs8teE~ABqmFHaaY4pHAo*7-}3FZpSH) z#-{sppqm|@sa|dp?Q2HF=EBJbAUMC!4q*@&I|EKFPJE|( z8i(Tnu;ZBvl&LM2(%PR}C)tPow-V3%LieKsR~bEUb2Gecs2~*J?@Sndv<(6{ps%JH zbq%{e-Pxj{lF-!1Yifqu58-372`VYggHuv&4G+Uq)>>4hbJ$?{V_tLHJ84Dz`}dgG z*gsJu8UXCSEhs4~)0mRN2hRV;^_+|*BcSrS6oRV_CMG7o=H`}ajlrD{-sdZM0f=3F z50A8qUyPNrH|%aLt+fjUClB+rh+gaq)(flxb}VJ*PZL)x*H^fwXu#K*r1!iWXQhAZ z0ip}HV6y>6yQQjZn_ge{y1y_Cc*|zVd$;KyOD1Er(hzR{Q4C=l(+3G2J`QN_qe^#X z>$lpPj|38(m+m)--#qf!u92whzayDB8}7e zq`YyRP~!83y=5gAT4e2kkdx54ewz?CQ2Wm&0s;bdJaH+*EhxR5A9yG*8ZDMCPFhcK zl{5q@11vN)Yc=cJb=f@b4S z_wM=I4Z>-_y3dud8Rmx5RO(5{c50HLQ)gJHE(CF}R1%fcp^YTf;;j2Gf zzMo2EwMG$miooXt1wcWmuFnvCwVfC@Na!>&Th^{QC z_y}IFxus=!QPI%E#DKI9M4SWI#M6$)3RijL9tBz1EN$q93y@=R(gcdk{l2=>ZtCD` zL7jn!(yrT5GCc4Duj`em);;&q=h_9}`lA&u=w_RQ^UVfN+ff@VS8lqY39xXv2-4Hj z0ZSiP_2M(uS<(sw$g6r%m=v*PO+C?F%3hR8$gy09xea<^nUbywckU zU-Tou*qAfmUPnr5YOx;B48zbQ3Q9{uu|pOhwIrhQ=`3oOu#DoP1no8l{q2a&7L|CbE=uUwA&p^2I(685uY!tWbTj7UK{-Mdk^R@ORtJ;YmFO6 z37MIvUI}cT4_8|nQ6AtR=fEGTt4nQY4J0rYy&{TCx0;)Z@3-CEB)T?8?5X{O# z3{%iJFHw>R51K>P#b_qP3<&3r%YalJ`x!VX0Q-QuKH_e+H)3n|7o)VTGo#ErJRLxl z0O=Sk`S&Q}t& z!k&MzG(ht1Rkt99iS%0-L7`;(o|wVm{& zm#T|eA-1fweV|i?53gqB<`R*SK@aP{sgA{E9p1jny#CI~o}APi)&PtTMaRGX0TmTr zQ_M*19sO3DZ2e0@9QnSgM%|Iv>t`4xk`47c6X|$c5TFl8m2W^AQi<07g+mP4I&}Hr z77-pUr`44$5X2T+%n3T}cCGMiXsNphi=q)fKmTVJthoWuJHfPG+UDz+(Xq|SDBVfF z*m*~3L}YP_V0T&eMO9KVtasF5Ey&|vLk2#EDZn^jwFD@v)>=s1d_B^*-EXcJv^UyZ zFWpbujU_+2JU#94R+g^SdTBlsoOd1yw~Z;NM=92AJG^N z9L8h_J5E1Q571n}=*xqkL>y=t8UxH< za%E;lKj2s=fNAb0YxGw|XcLtfcz;PTF@6p)(oiES_?fg=gUzgrF1ugy=5Hg-BA7y7 zgzIwdIKX-2lOr??L0?Xspi!8#Ki}&2@P5t1z0p=vQ&U!MY^tQ8s*_Rb=geP0Nux!y zd5{%Jk+sAnI+BeigaCjw%{jF7#Povp^@0|#NDzY*>X5VHW7YP$H#D24Qm(LhsIq^$ zPUZgUa<&QKo>f+Mv^$;(|9ad?HfrIjI>7m5-a;i&E2{P-Uh@aR_3LUnximwP zeZt*brI(N&kh83pYWndX`woiWOti&han$T`i3vqX6J2y{4UG24+$GsmljeV4q@;dM z0p9I&POef;cLE4?=qx%87vcdUn!Z z7oImqA<@a^_WVBeIza58y(>!OQW7Guz4|NHAic4D6gm{!P}_m;hdSc*Jk_gMC?4Z-KfgR=^igbF_b5f$G_7jLBZqT zHzuK`mi#(T`@tdM$Fv$_&y7=hQDLnK3W!I6i+F^Qi!zduAeYKZolou?h*?~iDDW>@ z2js-?GZa1!ia-oG$giA&uU9}Hk0Tlv-8$BHEYsZ*HErz&N5pJ#ujAfT{u;R;5?E7Niokui{Q z{DsV$YTk5gTCo3Tp`N4do7FpEG|5iJ#GojRec<|Gz;siVo-0{#Pz*&~k^$$l+yW+T zaj|J|<2N^dxAj(M04zNL=7(@0navcHmd8DkFXyrqP?B$?!(Pdw^_!eh?mK0XQL6Vykj9k^5QU!y!M`8L8^ZMjKVS!iaPxxMc zrPesIzD$+8v?2qNk>;P|=@F$S=jG>%K?VgyrJ0Y!zkLKD z#hV3nP5?VVOJHvftJ;due%itW9^guquK%GmCUEZn7K)Y-0he5Zg1{Zwp|J~Qq9CfH zxY7&akNSs)hoYn{IDzYS6Zs+1QfqQBR+ekW0nIYyFZ^QeqZD z)lQ4%2gA7DH1{w2u1!xfn3sDFM&^ro6SI zEfL`f0ylQHzgx)~WdbwS7v-HnSzcIpxTA~ZqXw#kTZ z<*<(h9}aO66Ob&3-v=e8%SFcylx7&8@tV9%8vNnJHt6RsgqY3zO~1>+*M^!+g~}jv zmtzV8GS~sNChfXkS0XzvFTlbS3PT!PJ8e!^%Jjb+Dim8B5ZX3SoD`Z~S^})3>a%Yz z^YOa{>LJZbsu4h?q%*jZQ&UT|i*6b7NCfu95C-3%=UtAQDV<4+1=@bk@(EhmNn?Jw zQc2~Jt1|&YrRgxZ4LiA_7{(+kI_;6;DfCRXraU`tl6h%pZZO+CVMJNeQjh9@{F%J$ z{bgak@mIf+#N9N<;Mu;}tG>4#hc-M6G-~j|DyrBqF)?}0X#&FLX}JTwSy9AD|5O|M zF;1$gUUUvTg$JfItgK~ClW}0v*Zy5tXXoRoYly&rs(;Hl-W<`jwsWNK9ix+tiK@Vj zIoZ|4=j3!lc%u1-{QE5}2NgA>5(>YmiHVAdO!*e-)I*+TsCSxC7y*=6!4$1cCHDM6N2VP**Y(N0ilRk->Y8rnL*`+as@a*d5 zRGOQ@+`ReK2*bD9fLuu_V>8hFQ(V&rE#Aw%k0IEGWr($b+M|ppQ%=tJ?H)6Imwb*l z11<-fBErl&MpZZ;O8Q(GbxY_)>)^1|AogXA-szc{QNBgI4-rB_LSnRM>E;N*3hGJw zF=9qXb$NnlG%>AtRmcFNZUA)@LUEhG_L(tJSGwy54_($lGcgdQL6F#1mgwjc9$jSd z9&kKTQc{55RH4zV2g1^rpL?pQwb?JWVGMI*&}fKen~ui%xjTk!2e*r|!_O=X7OjJ| z*jEH_R@-dz0bcl%d~InmGTj}!*zRH=Dk=&hzRi#`u!x9G9fV=|awaOR73HT-m~@mz zD9Fg&-RKksN-ET^W7-~aCtIHpM&nacT3TB0=6#0%_HyM$tii&?rP4>1M`oJ`*{(1w zqq*tn9%6V@n6d6`!~T+(85Kgt;E<3QQPMr?&+W-Ds@P(^T*<*IY|&-7=fo_It5782 z5fMhDdC{@&%#9Im&|Og|hSPLo#f+d<1?KFFMPH?ijSG>T-bT*F0hcPviP!6ituPZg zpIoaKT=D>zKG7;OLs{nFJ=kJb{;hxMZzznPpZhjim_n!4G{C zeDFJk98(JNJvkm82Yx>0>O*`uP>}V)eA2<4y)RDFkt%wxJ|%IC6ciNOtbU?oDLq9mKEECFT6*{3< zWMqvTJRSS)U|EzS}}hdO+b^DGuLThB9K(?vCzku-`<1TJu?0cMcH~Vi9kbw4AM( zLTPJ>@oUTF&KH`t8UB+!p*efkESk~tg5Mnn&!K;5(OsRKSYAV0ee%gO1dlF(#wxsM z3BU=tjIR^d4^omEDBT~@0$^i>n(cPP<>Y>qU}lFz`cbH9BCH5}%(wsbvqr!pe1>fG zBlRJbPIsJ8*P`g>0}A?YAZ@$!L+}9{N}9<4CWLT3oY*!jLi?W*bmy^S6LcGfC})&x zoMV#xIzB6KCn9j2f}lOP^kS57!SzpFq@oh=yOuu+LF%Ru=+G?k{Qgy5X(9yecJzaB zBahrR1jsSCoo(_HWs$@A)_xjA>qY%zu(+H|rjc^YEk2dDGFU6#A&)*Js&{LW-tX0$>>g(T^;>xjp$d6Wb`(eI0%8)}K~`2GR?d~+Vhl$^5fcJnHX%DjU}D+Am>8qweh;+S^pcw`s<*GAJBnggA)C z>6aOt>3pAWpe$~0*b56!JUb9HGc(iP-d42X=zxU6@KQnP`qB`0*O* z_V$1Ln30(H3P%1=GaVy{OTBnXB zIe-zK7UQy}RGO3X6~snl9-w5!)cAlcYXL90AEelSC`J@736LarqbH33)X& zzQhInT+~l7sPN8P(CD}(_;%^ioa56~_1lZk>aT(*PqRjtR&jB0fw;Rkopud|HD3Ty zDAq5mmk|-U0nI&Ty+ao!I?F=BSgA%O$4Y=LqRFkFGT;TYj^f!UEj{y=;W6r7VCfU# zytsv#nFeWiO@luq%65k$Uw^cm=3Wn2Bi{#65Xvyf5%OJKAi8}Gr;&>~AQ)5vuhSI9 z3eLsdwfJ3!mVfWVdJSAweD5hwTP|p{L`bk(ho74UEI&G7mAq_ zy1J}Ythu5JjH`WryBZm#jvvTQPX|cKDdr(thDX*VIOX8@3%FPgs?ccj=W0Eqtq)s(Oc!Z;E3&wQ->MBn|Wt&pG~x)EG0-~gBv__=~hsfv8ry`#W= zHZmU}{7Z0f9j{)3c~|bejS{Ogk6$oo;W~0eCQ|BE0VKO0rIP}SN3Tgz(B+TZG+hO{abwQkF zwC?R4dCF85w}}IshhcM2pbRw;4>u8U_c&Y`m`dd`#h|IRgbG_rD>|{s1>mxCl9GWT zscC6l!XHmwxKQ&K*BdO+2a_Attq0U{rL`}6o6l0dzWqJ%w5&|7$iOd&k=IR=fS!z+ z+%CRL3Mt7s4Be?0*jAHBz4ked&d$krEJ1Pa^RpynRauJaXC#?PbS5G`CT0e41^FFL z@Z&s#`VKd4u)ZD`8L@|U;q}j{>1np=?VxZOdy;{X5zJ8PI)dZYKx$lx5Q+NzXzo0F zd8zOrRX=d60_|x#ectEiRF_2kf{y&?+S(eN5E5q9mf9t2LO1KTD{0Fs1IueAz7h&E z4fWZ7(8yM6&_37TcP=h2UV{}q@bzIV5Ki83a)8CI%_ge12>}TA9csk)(O0VR)dgz| zu*trF1j|M}R*34{9~2Gfq4WQ1iOuI}TH;Pn_SPq`Nt&By_t*i7ATM;;k0)*oL`C@# z1O`r!`4{z|C414UnuKJ|>BbNyV`frfTAF>pm6i;6DS(#jySuZ!ahJDV zjrR3y@b!G=ndNnwr-6zg0PYB$g+D28Zg5H{+SuBX(@46nL5u?E9WY5HzqP&yy}vPV zW(9v1j0%Pmz*d&PNkp67(&FBvPtZPg?kFdxp{c1a#IaE*c9}g~$B*j>uAY~Xv3{)o z_Y)}4<{SrHMuIoXC^$G6F#n<_PiSEBvprh%nbWY~Z3zW_VB1avdx3x@6&?z`(i)Pt zi;D~7b*WpjycB(VO^e1Be}X`(YA`}|9-Jn%w1i5*0gyt5tq4*n)L^{4yy}DWWaq+s zfE+=`KM~p%0CWzR*s<8wFJJNy^M~dA#o^)jaExx#`Vk1g{Exs5OCS)E*UL#!1W+ur zoQIZ|mqWwD=UDZU8)(=x^5)|gFWB8+ra{5L>@`2%O8IwHp6F`)r+p{$)RQM~L>v?X zD_Ll9&r2-MyJ zRd-aJ@&cqaAsYJYEl@xUcr)mQD>?weJwI2QLz=d(uJzM)y4}(orB1j9n>_+s28Yal z`)gK8(Tg`;hA6kAc^V;_SA8$0@BI*tee~pjOloS36iEk%V_G_x_>BXBIQgBTzhXeo z@h6QZkxaIlqfAU#geQi8B@jv>!G`QH^UM1We=RA^se^ec>(Pe%cL?hwQ97Q_5HUjJ zk{||i2Wz(~->Ht*&%5+~XejKQw5iO^2;9|yE*=~1w8Ci)Byx%`L+!P+x+VE#sVH_x z{|^a{T~g1Qr@$S5S7oKLRL=O zcua%7aj0r#*1V3bqY1eY_BTDt-y1fm3A@F}-ya7)zcaTP2GuG81CVsr5|?2`+3rIm z-bQ5OhnrRKC`+FBrt#-AOR|D!0|18X0yXfWiAE0(jda7SGf-R4*cdOpuFdOFN!p)P z&c?5G*Rhvb_I3m$49GNjWtE?jYOC&TM$u9ltgtMCe`0ym)zG{OvG6C}0}Y+s-P(Y6 zvi#VW#R=&r35TD5@rQaW5u#?|tZ&`A1xiKb{R{5N9`5c_C{-gPE^7@@O<|6!B`;n) zxpZBrq-{?JzWb%3mf#Zo>1%Eg;j-rl2l@;*3a zXio@KDF8+@p_P;@CKf@dI#UKanG1nXwt-(PJyiC6Jzg;hNZM$l3`s*E+B_OATVKD< z&6jTMtf9fO-cn(xC!MO!swMnO(#WFBrcOl}x)Q!vgfVk>?i*V~aIL8Ufqwtr>u{O@ z(T2?9ZiR+9g6mHi^%|V1=~%FV;ga0BNf`a1EFsOMxoavB8z=DyZ~4|eHe@oJByoiw z_oku(oB(|>`Bnq{{U;r#KCKuTF3!zq>gcrLB5@q>jPY2^`9QA$F?&9k=dwpc`zKCq zFl>vK9}fH*=Sj!rt})g|E-4~j^qIa?@JmVSF{<;~|NW0fEbYFI$}&i^Vo6rr@au*vEOZPKSNCXO Lb=1mKEQ9|8r-o-2 literal 0 HcmV?d00001