From 9743cd13836e979a9e5362fa1212150133e00412 Mon Sep 17 00:00:00 2001 From: Annie Talvasto <46491425+AnnieTalvasto@users.noreply.github.com> Date: Wed, 28 Oct 2026 00:32:26 +0000 Subject: [PATCH 1/5] docs: add Platform Engineer to Inference Engineer field guide Add "From Platform Engineer to Inference Engineer: An Open Field Guide to Inference Engineering", a guide for platform engineers moving into inference work, plus its inference stack diagram. Signed-off-by: Annie Talvasto <46491425+AnnieTalvasto@users.noreply.github.com> --- docs/content/guides/inference-stack.png | Bin 0 -> 101740 bytes docs/content/guides/inference_guide.md | 708 ++++++++++++++++++++++++ 2 files changed, 708 insertions(+) create mode 100644 docs/content/guides/inference-stack.png create mode 100644 docs/content/guides/inference_guide.md diff --git a/docs/content/guides/inference-stack.png b/docs/content/guides/inference-stack.png new file mode 100644 index 0000000000000000000000000000000000000000..bff9dd29d470b762cf8ca4214c4f55a496c497e4 GIT binary patch literal 101740 zcmeFZXIPVKw=SB1f`W(@RHR80>Ae?`UPT0?i}V^HbV9M9C?Zk}hN{w}D=m~z1f&-! zg4BR?>75Wr_7j}nT;JMzt?Qh9?Q^d4<4k@`Q{M7qjAx8-k9*wjd~sV>jfUz96$ApI z(NI^u1A!b@gh0sCPmqC6=B-cGLm+g)Hr7h_NdG!}yFCCOGl73?ZG<0ML(W3ZghC)W z5XnRj@}m#HQTP5jO6|{4ZvXixZ%=n=(vi~E!jjhFl2Sr^_CgYZe4-De9`IRvG(GE@ut@UvA284Tsx#dO5+|;ot;F`Clg}AD!?3=K1d@ zoG0Cz-^ty<)6Uk(&fSaOM}&0FB^zP5W5z=@dQqitykOrOT@wl^AMm-W4LCo|$<{{N z9_Hz0?d9a;X6;}n@X+1C*~uO5W$kWbr=je}>*w}R+R4dQ+TLDR%vM57oKHkpLYPlf zSi+jm+D^)rPsm0@NZj7m+D=eN#MRTz6xwL1j!tkEdnZ>r7Hco?#*W3)&coXd?!^MLXR&7S2AhY)+TE7L&Dx*E(b~t3 z#m3Xl+RM(C1?I`(2D5dtcLJ|TryX4a+}sPe49@QchuM4i0{`7~0PjE8x&wp$b!$&+ zSJ#_=e5K)T;|hMB?KbdO%U>Q={O>(XG9|yQwU_nZ4pxu-q?{|q5Mcl44D4#CV?X#J zl;q$H8*5hwd$^;uu$Z{bLu*?d(iwmM{pFL@8xF#qMn*{rr+JlG8?ImZ5NGuPUoa~o z6oxbyG><|YE!zFa(ctI1JJ|!PINSMwCBsQO^wCG~-rCda605zdox6jV<7K*+5D1z4 zU%!Lv=ywR)xc+;D9pwR%Q=Q!Y>{Bw@|9QDia3=?MYcFq4J3pO&+i6EzOW=PiLHyZ` ze_!su3`!e8F;Q^|NeMn1AsY~s_78;lq@+Y2@YxDI5EQf#mk<{emGb<{yUJz1oz$fh>)tCF@M%H8{Vy3k(d5l zm;3+fx_@u7zb^Y&@~sZ}@!Q)Dx1MwQL_Pqi&i~4teK~}q%Y6}_%&WUzN8Ypo+uY96 z+6(4s<>YP;OFP;-u$%viV-{5qU~mT?kW#?bu@mAK*Zy+t#k{l zl9%)a`FW1NzhJrbC&F27ff@lK+sWD$&TQcyxjQdC4t+!VZ( z1aHNKq>kPSD*yMl;uhrkb1RmAyVaje^{+b`fVcqJSkOf9JO$;^^2wlo{SI;{CAkvh z*t-BKm-gG2j$im8TCJ2JWv0CFb8N5(r@2J`9i2(X0~PVL^*tPIlT{)2x<11%9TcSS zl1|hxWj3f*lfGI49h-Jf&`9t!SO+656iPgHjGT-z^vE`h9LG)zo}!@Qp@1HPKAE>9 zgMxS9u=AwD$OIvR%%pd0$1VxdgY^2M9`K@ zPzcU=Aq5)H+iv)}OLbB%TxyWRxiI+nOz9ylOpww1?^lq;&t&CM-v*B|67 zuyUC{p{|y|azouf`9k!ya82s)-U}*c-;BFBK6=Xe^a{(FW9QDDQ&;i~S?x$f>`jYs zh_JuWp2bP-K5UuWtngi_hOc(aiJ=b_Xpii6H;~xP_s4=iXrWA`SE1JxLn%owSg6iS zKuIsc$&LZ~75sgK{By|vO}i*tg%bCsqNWhRhl^HK-a_|(ezvaMyT--UoL|7?UbQTq z-0<}YWtF%7JZ>&^@SD^3_m9lz8De5%JB`r(oA&7amlgXmc(y26&*$Cc9#f;z{gMg? zYfpwg59XOn8O7%qWW6r)-|A5tlfFwk)f_drJI|$ndtvrmv;H*%@o zHZj<^Y~hXltshiS!f&>^X5CQY*lKWA;oetD=5p^(9w*ksD4>KJ_thYw&W|C?b|==1 z>#;y9UocTUu9w#R!`!MtJhhKq-z_Lz(bML%^29K1v?MElZ#o5NI*|?vqZdMT>J21Q* z>vIee?rg|B!I&AyIx5YarI(tRh#zuD^`8HJv;IfjBXR=+gR!%`GQKNIgN0aX0SrrI z;LcKoZR_Zu!r8f!@FJIE}oBvExiwfBO-FiLODBn}gi>W*5{CJh>HM$#5RU+uBH`;i-7YCL{D>h$U7I_&FQN2Ni zNH3HGMmM2z8pZf;5)SdJV^xE0`Z=FJui;l~2am^BBuK&SZ{B<~(&D)?DrEti6@A00 zli=9;viIT&?)V;XCoQx5!SPi6G}UBj@00@klDg69YaPLh>hu1``8of{OxLwTxFL>(D5d3X>RSt z36f)Y!)rbryTsR%tS^u4n1|1H^oT=!g-V0)K5l7MWxNwlPo5txbs79@eJd*?5BK6c zSIt(h)`q1zIj*h;TgYfu=Gt?z?Lg9yo15E|L!^*na>?Tv@l=w0(6H0_$C!DURZJg< zCZl{)8b)c+3B2bJEW=dY&e?l$C?fk!i8wJz|XlkSseQn>5 zk4jita_!R|Z$ys=uBUJG@LKU}`rMR0sN6e3tZ zByngs00Q#U`}xyr8~HXxA1h}QT+*=ENqkkGI6!u!4#;=Goa0_It>GHzAMr2Npq;krZIRckTyG$9|zVhG5si%A~LAqO7* zAMQitfAYxh{IrRAxs24L!%aSC_AzVz=KZ^0Ln*V*uP;+4gs_>KjNBa$EDXsMJIy3J zUxHqs+HIrQJIkheu$84ynmFb+6^RWY1`VBz4|klUS>Q-Sia2zYJy? z)gG21eHVJaU7$#WE?kqTb$J)$-0@E6+)3LER8x_Y|5tKMp4rgp(gTTFTx7`06O&OP z2WhBGo8c0t;n@wJo%_^G0iC(`p}|8%c4%yH;LyUi!%5l@!R+qWk7L@FJPZpht4oY5 zl>$agFS$moVJVk`4nT@QtW?Zt+n8t#tNL84GH=Ci{4ll}ilzEA3U*n?GkQeD!R{7_ zMwICg^YRUUCj&cizCbgYkk~h(^C9_0yDMW2r`BSWN+XrI`JAvd)t7r%U*s}y zQx;iQoj~w<#M7t8s!SAPj7LhGrd~13;T=7Nr)R3)ZdmiAQut_@h*%-kRo+1@jY>Z^AkmZF@k7h=Yh@!<;t9Mf@_ycxVmQy=sswNy1@Tk6x7hEZVR#$&-)Fd~I zG8^t4QVQ(VED%vsg+(CWDIQnx2#sG|up(^4XMX8im{%`6BobDMqpBI{X6N?OU2!lX}A_!SoOBiVJ+jtt=eC! zA;iEeEL*jImX<>-M`_^QUgGAe7PpB$gM@ntCqg3~N&p2wU}XB*!cFO{3#y`9*y{0Bk;6&@3!03@=5i zG4&`7iFK6cynjD2n+O{)EjD;M&XyE(d;A0+Y#dWjan7)HWDXZx_Q|5E%)*^P#wQOI zS1U;Wo!Ukmvw>_M>JyNu^IaU^>(KeNw?4J;(VdMME-Af|b8L^4bQ7};vMd9aOYP$l z-78y5Og`U#w0ZtRCr)Ea^Vr^Z5ti&9C0ksnHwe%lAUGtJn&Q`uH1lZoqy%^jY9_Xb zlgIpU68C?-Jw+TagH>e6cNLzT-IWX>?hn>3(W2V-d`^diY+YMYcsef8emGv3#|VYo zzNvq8;)ene*G&p^kP3GOV~(lgZ;vFxYMgR@O-(g}d75}Xdc_FzW9)UR7^w7=WslD4QX|e}v4MkjSn%}1~2vEy&*;@uWlY>?v zZke-hg9?{D6EUB(vFvAtCOQ)&<{VTji;W+42Hqq)Ag7gmAGVP?+3>ubf^#)`>7D7% z>OfyNap(T)e9?oR1s;3AC?%S+ZAsLOJ@Ss@WX1Vy!Gm9H)J!ZE2MgmVNH#$df6>>z z(P(9ekBeh>QKSYYo;?#E`?Mm|@>21nN_m!*PpENGLmMY6KCN1#rOY;jkl4>k0Li?K zf(7>6b-ea)8K&cJ{uF8{zhWD`%)PrjVvHUSotwJ8sqxW8Z|8xD!r}flgShKEamj$K zPfz{b3;^yJuMX5g#_H+I?B$zR^lA%!gfMq8Idq4LzPbDk0GV`+*p8UC=eIs%Z-pTi z97Qzx0X_g=A`|mTicwWO7=I9MDW$&5-F5G>~Fwx-OqfXz3W%M2W;Kshz(y#3g zV1n0t=LPLhw3WWJ^bYjp*MS-MtfnkJQ?fl?Kq4}1tG4vu8H4-x2zw6HyQZC?cKcFw zY`gr#l0Zk|&llVVneRLo&Y@xjLpBq!?>55i$5XcgOCmAfIK2QAHmB}1fy5w6oELgC zOg9?O>p8xVBcCn*G}n%vO>%hQzhUoGQLU%p`nf8mO@Wj)I^)#2emXG&>wSP_UtH+2 z{0b5k)!OCDmrXZ{Qx7}FivxC+vS)-}B5m#M6B?eR2JN7nxPK287S8m#492MM-sGa# zOT9>}B_8VKYTw^Ptd_sXP84@5O0YUdAPcbWP)!dEP$5G|CN@4Ir_`P-?v{7K?!gxa z@^`G)!pSf%x5h1YxLQH+W+t`|cdoEh=+}^GS7?vAqIOmT&K~!>7!V~=DuG25!v)K3 zKA|w>_}a~nMnxUR->>Sxm~`*V!P8+GhisXL__Rx)pk-5u)PWj*Xb+ts|0*9Y7eYYv z)c`=4IqI5f6>?aLY}rSnA8yJ$DBM)9O=cTru2$ctjSl{I=O-7w1StcRbDWya@dM5| zzVR2UEo z%^zn2P}i^`0<#_ynnN0dwtb3ngaqWKQ*sv^2c@bvWTux6Ia;m)2iuH83GT0Pah zGFIjCc5AMrdf`EWZa(S}fR8f9L0XPIIg&iK!}z;(e#V@2&x?-| z$s;m^;Eqp9e0)~e?DZr@&)&X1ozR{t2mk2RCp%`}Kk26xb!xrSGG+gwmnv`J!-H0N zO2&V`qWJ7Wie)rp_1Hvc;!{ji=69mcB}N$^(NbfnI(+vl_It&_N&S4&u>-FHC+eVl zd7}_mcU|CW^;~~fl9WO2Q#hS(1p%P3g>awGQCswZDF-CkHq57rUB7gZ)oyd0e2-zTYgJwg4_RapGAIhUFGQ2Ne@|QdN|c9^)z+rP7@c`wLlmNYBMKs3*Zgh1IFHU_Y-0dZ%1+Rp*;!wf#F}tz ztYG$k=L(Oox?8$|^gh!C0&?Gua!*j=B>XplnYh;s{M?bMfPLjDMvfusJ^Hma*TOiZq_#%fGz#+PdKxN*t=EBVj>xexj3yLfh*QzdLBUu;!%kIq-^9hllI} zHgSIDjd6+I2JQ6@2)1rjY7I(`06x+i=XUGE_``9oimOtdjaLh5 z7cvt&^e);{ByF<4t9Gb0G{j=B{>OwA7lu7>1P=Ft_(~QGhvMqk8ll{ z6!;Tk!9!5_4xRc@CxM9r*O9W$Bkv$)tGsLGh^XxRdtd6aO9d`m5p#~CGjb0_JFtk( zQJpmd-9-mwTx}>g7u_qnRxjmO&@nladwzn$0#B5+K1RJZiI545UW&40KYXsP06+Mv zdrM%RFe`EVzyZ)lwbYIILXW8?J1!sn^fap`bg9?XPQ9gbL3YlwOyVB_)EPK-gR4#r z1PQ#jK$53)xZ|uTur|kNl-$m}p)!sR05ng}GRfZg)rI`F(mFFU!@*K$VAp|ZxOyqhE?OZ&=F;5aJLmi2#xhzgfF1cjz^-IIm*?x_&1?O_y2EV; zyT8MiNkMaET0HW0xa4G5(w9(~z;AwQ7~L&&Hb#Jv_c_r89^*d#Nq(-8kj%&BerUmhUe!XOJTR9w?peQDf#C!8Gn~ zFLqux7OBIB#Ds&i_AVQ>*%S#Npn=N8#S;p5HWZerVea%V$P;9?!Tus7rRD;I{FwdReb7$=%llaS-MOB`JFlFqU1!XA;@j*`$QD`BGx^ z;gb-gc#sZ!BgKE#Km17sYkEe8#yhm$ryYflcDbVvtFNft1O6w2M0L*r(>^WaXWP?V zdI_SfuiWkHh{*b|u&~;_ZvrI)Ng7NKQt#|=Z-H(|w3ekYOqP!R^%;+PYAG8iDeL$X zA+Xa|v<9&?*KB?@z5ofitlN1-nC0d*85#UHdaHhgPC4qk>G=EeL8GbD)qs~-A%Sh1 z2UT|vFFCdvzeOW5bbY2~=HodgKx!%T8==kC(-CXb)%OShG4M^(S zw|1F3ptF}Q-RZtOd&;MnYah0zhm`kJs!?9rI1`v8M;`yhYHnr*WwQ#rQv$2A=O4*d z50$78ov_&I&>1UK3YFjzYNaH#5;`^_v=>jVon8JfHv6*K>N^?r%UHx_r2K~RMv6hk zHAcB!*y83~_afV4_BUEev8TTZfPzxBT4N?{+vQebm~m_mAUSIV`D)`rOWph+zJil$ zvqGjG12mruRaKu!@r;cAY8lPR2i(i7hkFx9qTZ+J`OW8XbZWGWlC&vuK1=WX_3=w~ zA)IxTm=L{ici%b)b594Q&Fg$rVl2u*2bFk2lXf+gc#uV~q8oVSa+<7N{+DdGD=3_% z937{g{>+MFoQREyNq2E~zwEtSI+%c;w4x}krNrEjY=ho_!um>E{DaG9+`>10Sf<4P(d&^}y2;LMwTC++ z4DRJObAN%xt@c;-?sye^PM0tKb-4d$)SGb(a%3!1HS)@kuV!)8_I_^!y}*D%#-hDd zreo?^XKX7rRf_N6!*eF#WE(bRw9I`F!?RDmFk7W}xaHbC-qEBB0rdB>$MWS*R<$*) zYXH`X4BB9QM%-pYZNq~!aj^S<3K60$iFK>1kTTfw{SY#j>|26TmLc9f^Pq`rg;XgB z(DAFHNLL%q>plJ2I+7A5Z(QHc>@S5%))ETnaq{y-x}WTI?+nD7UNgG1&G3m+D$Jz) zxH1(JPA9nK3aZTlK%gR7s#;4zt+pVvW|benn`Hzxbbf6u^vxQR5AAh60|iL%dqAkq zsNL~ByLKe1r$Ne=@mH)eijS*nR~xqJ130b**K>TPC4$S(X_?)JM=ILKq00_2fiue< zJEvM2b^JjRm4cdGl}G%%vTzN5^--Z_s=)PC7y^Oe4%z-;tbjugq0Vx{XzO&s3FxT9X;bXYB|0t%10dGi%PFByB|4*U6Tfd zF=QC}NhMq{ZCSV)lfUwKIq_glVLz*4v-srNgZ-_A%<8Q#2@){=YG%B=V%Jv$USo>X zPIxhAG`LBh@u*=_4=~rg^RhKV_~wh6-Xc)`XZ^y{h4C9UA+vA1%@;nr;z~vIj;6~U z`G5Bs3U|Mw@Z4{tnfn))&bQS$03Q3~&kvX8X6GB`4)7FWdh<*~@Rd6FYa?gyaa84x zTCXOY)U5qJZdy;fwgVlJ#D?_+TrvqPV72G{dDL$)Pv}gXr3K@U5m>ETml?96m)Ihz zfSW!q68-Acskx>eXw~4^%8gb0SJ`srJ=r>2Qz>XLpQ(y*&WnQi@S={M^A(4kF6B21vzN_0bBrLlAbE89!ZGR8)j%58&zSS&+mQ+7xBC zbU1Q`9IQ8Qo2(d1*98vH)_Bw`zMGCP9IPYwwh3R{k(_oZ9Q| z(3_N}o1b|!Om;pnG#_Csq%o<^1*tvf)e0}Dh|!R0E(!BWED;CZy+?{}@|aRT)XGKj*zgZ z75#RHfKQ`Z*LVP!ujZQPC>{CDLj`0!ugL>lp@*YheLAz+lx$l}b2$FyH)Djyl>%m%?hJUnRJ* zNS>hIf%6tk#1W~FheExCC-kijuWQ?e&q5#= zR3#u_A7FjgCaMZ*EpczlMe;#L~rpo2o`|3a*$ zO;?9TosU<_E~0g5#Jy%9H7@^1)T2-0>xtW?}PeW<+juSE)T>M@Cj}j^ z>9Gl4TMDbDedb4fs&#F7xKw0r_r}84=QY8N5w_3<6wSq+6#l|(Fr+)&ttE;@ay-*8 z^3o;l@cYJ|5e~xpkAbE$X1ArdAg=}kDP;DgVGu9T6FPx3Q$oc2w7t0muYZfV3CI)A z#Qm$XexF?i3wBP82IH4EoS2u^QLH|GJ-=St$1qo5Pc<~|_zI7$p&@GG^-H;4WEVl0y_GR!)8MRTM)@0`L z;kb+VV0$6Cz9&s>tWuHvp5}%`;e1H3&~q;d9h=KSjqA+a@eMjtB_*Qh{hSELSw^tC z#HaQZJjU+ge5u^dndoS}xDK7_d&^?>tuT0-XR~}YxA2;a{Chy3&G*+2b?MX+NeRV# z*(ly{J?J|UphM2LF)_c4DCq%gVe6F}+f0_aR?Mow4JeiTT*1~$G2Y%tB`X61#wBbm zHcuK(j5uSvFnt~dn8=+&ULGZJauIS;^rLSH3@-F#dFM4YHr}60xqI8%da}TxYMR5g z`D94bjrfj+-#>D_H4YV0dH9nK*NgD9Z}BK{NO{`>o;qWRnD1MITV}9oF)^_p*>4Mf zN{st8ASg%lW}rDz)pTP-r+Y#lbrQuZNA?xoGeB;@nfhxRL<~GuC^7%$8hZgbxgl->2zG>+VnH#r*!~hRmu+-UL^V0v*S1@4zH2@_0 z?}-qIPz1#4ttT7~vZ&?ZOPWdFTD--)nFINc=Ua^1m73^l;-op^XroJ89Jjj|BF6+A;7~1 z?+iJR^RcPD9<2^Uk6cg`vtMCkoZX~`zP~oF08z|its^H1F3uc3u}1nAq6{UIJ$j)Z z3OQ14C|+Rs|Is1u_bK+0mhu{$@W8F*{(WtY-Vw(6NzHYC{%#vQ7t5u0y;tk1)tfL@ zWuR+&mG63sFJ2WA`p%e<9AQ7(8p&X^7xG0&@Xudk*Jp?Bvd53f(5XO0VEu!qj-HNg z>1RzV0+w%4{BW1HRqgbuLbS)-J3&SR+!TNQo?Vejl78#4bZ_{>lfW|e{L+1;i1hEv zf>6iZ$JI^2%I1kru6`4^&!i9hcBH|^#>u~v<8l-rhmF2ybWK*zN+FDv91J-%J zu~CjnLlk+l_aU_~Y?mQNl&;CU`1r*EpzW#K+*uvTxJ($WkVT({oDs>9teZ^ltIl92Hk5GAm)jZPGNniRdk6yznQtl8ZjKJ9} zXK2{4G6tPVN(1l_Jg2P z>4b`7-x^-NI#WTf#Dr#^N* zdV2aQw!2|tRldzQfZ~ZB;Ka^EiA7n3%?^?Od?sm9DCR)11q$sUXtq}*jTxaeIoiq6 zh9P|#dV0RV3!j>fZu3C2t{Eqcu(tjnGyFp(wD5-TX61z6-wi7C|8f>E)qA^5g52k-!ypTX3&PfsBT zU?7sFWCkcDN}I0Vy0gmw!{`Gx4p4wmSHx}z2y{Dvk>e{c0%_~t>MpD$RJx6nMP3oR zBJDLlRP3;fWF^A-mb$tmeD+n}{k^qG z;Zl(FTZkgU!p}}!?)_}t0A>IoxS4ny4yS7qotVgzCh9&~4oXmO6SYhov*g43pigF$ z_Ik5Gdqs49;_DOlh&wz+d2e>r2eR^f7K4wID-iZHZwuWN9Z2P0hAc!WI{wsAa$c2Uj26#fus(7$6c^vaiLDtr``WJ4pmihS8@V00!>u z)$2Wj?g-#n*ig~8l2ddOK=njev?2~Ot&;=`67ET3O{u)EnEXf6(%R zs3YcD9k#h207+W=sgX#u+lUWnw1?e(59C(`Y&#!@ghLdZdeccYb|)MNzm~z!H`(6A z#N^!SvVEBXP=E~cPB_PlRoHi433RqE&C1M_UY*THk5>ZL%FX_DwG5KqB8R58x7T^F zV7Z^LgRUL>V$+;KPY!wi637Tykl!AJ#=puky!GrS4=`VT_k}%=->j@dR79f)@RS>=B(Vdn+}4}R>tO09`EVk%?9`X>p63)6 z+$j|R=KVRa$8|)SlceAs_k2N@1`FO~IIMl;LbQDkCA4y-_J{ImS#$D_2d>KXZ2*mr z#F$rXc8m&O_<~u9r0GQ|`@PWkv!!-PFv4o(BEQI4yu^tYGep8}ZK?UdG{6bhc(5g} zipen5!!tkZ7DPMRlSVFE8smd@$GC>%yLP9(@u5c_s3Lx6XO);#Kx=?=O+Ug(G}Q0Z zjb{mifTGDcG6di>1-N_Uh46rt{q7kZ6LZJh+_kr~|2Ovv;KLky(_hdfh)e=r#*8oe zRCctaf`Zk#Y&uOxS>KhUvVQh#)*a9{^lt#{(4TJxnl;Doie~$idjLnCJn*RbsQ{nV zMxE!D_4xIa=?=1=aY=Fw-YrKzhleVKXudfcxcxih{3__ioid^FT@nodL%)sb=;#1G z*P$k9Li+B61ch&^U~fG??O$`g9gK?#!#uu{AG~3`Rq_?Y5)~Cy`ZS>Z`p#(#W)3|* zPF=DVLd2&s;S_@Qw@7o`u-VWGlVy_Hs8ir97;x_`=!M(}^Bwoks(ET$IXs0>!*l5* zfT4PhJbp<8J|8?-U@_cF9ZZAC3*kEoqyk_&4ls0IPE`U)>fFCl3uRxCot9G>0_S|_@yW-*YO;l_z91bs?PK%Q7jai>+PP8ZGF1iFfb;6Gt z$DV6_9mS{HyA#B;v^tzu2Mbfjdd!!iP!`(!jLhwO3zdoQz$1Xua zbE7~;{lZH}j!B;L@tt5;u>>VUm!GhvP^-vrLi$i6x1ofx4sNXGkIYqHcm{wpKj9G%>*Rk)-EzxcD1f z|2aHnpEO~etPi7>!wOK(TI>Pws6*iy_}eO+N<*Fx*YCtl>Ln}7$H(`8mh7EL^-ASp zg1Fla*hlymIS2%Z4#98Do2A^N&uebs#-~ij2$q_h0CfXWG4`QJ%s z+{tMx+a{&S`~*Sk$Mo0hfQBYyAm+Y%FrX5x`%d8HabqdcKrBpFug7n$Va0fu-D#L??H>^TfQy{1L5B@g zRvPdt#84=C$s?0xlB%dZ?|3@^3ndfW1IV%7;?tiOO@;vZDw}56{e`x#?JbXYUNci* zhj~S{P4j6!%qvK(_oC6^W|z!rNuqxZSF4L*9V9h1wQvwe{B2xYRCoyxJ?FYWzn&9T zW^C!7@##~`<@E%Y0`6UnrMH(RB7rW5-?U`jint#yz51y%{^8{eHb2x<0ihL~VM=gw26k8N@bgnI%#+DGlgdJ56X3>DP^`}S-T16)%C6Q(s z?c{%kSow#!Uz;KsN^gT!@*cH!koLrQK-fhZkMmPY>{@IB{_Kt2 z1-`hYN}y&&rCt`XmtCtPH@$w-*lf5%j~ob8U4RloRkaCJ@y@h~ko~^YwtE4xvKzUU03RJ6(^K%q2l02*rtEGo$S&Cz^mpqJhX5gi}TwXh2YRek4J7jJH0O5#fS zLVV`Ge@#|4u`g{Oj(u9_E2OTjUOxGZ!Rs-Qbpbc70Td=x9ro^!Syr01(i{BLs9}jU!4s_Ci)9BN(tNEN#|tNL2D6+-nbg@Fp&t z8pGX55+vO#oH-CsPBAet?wIwaA~uWu0t*@ZkgIBNFafBSXGD<|0AQu!;P{9WJ>V9lL?nS=s@nC`~&+DF91ph5C!G%wi(C7IsnkzoQX@yYRYDm z0-@Z=2RI)M!;bsQmM60}g9(IQ85j=ll#W8lF_K>MZyN7cb5vCVQ7MqAEQ!Ca4%`J* zuXvV)NSd^OFglq8;A>xuIyq*pO0n)0KX^2t4Cn$|c&bBX?}c`^wk~}C@I;CrKOoeZD zO1J=R&4cNd=-Cq?NidsvU@=SZe{E6xan$4Fet{A;(zCL(c+I~&P?es%d-PY>U53_doSkU5 z>*`SOTYGQtHZCsRaC>pkHwW~1|Ni~T?X`&~v^5}A1u*;d0I47RG7(5M#(KcRE&fO8 z{McTbTBv}rkB?8C*=R24PI`j(K*uNe*KU0a0}90eS$`nUc4ySQNJq!X$qCfDc#Mc&t1g!4>q zEf^G)VdqE8h;iv$AwJ8!6|OvYY39Z%P8qMzFiLqo)y4tE`9dnTW6zXuRl-Mm$VlIQ zsEDKJf>nr^+(hb++wLnv@<9PL>#D0H-lfao+she)EbX%AD3RKsQn&*cggM2=mH0QV zS-*I(n70mjdQ=JWo>A__a_Q~N6LP%5lRvm6B1i7r(JpSIT5FHzTOM27CNYT@KR5b? z0)@|eS9YtQ?F=)0xib@mMW3iidg5Na9xyX4_mr7Alx!PF+mcfn!4_t_giFr4M)24scJrDsc79d(L zPH?19d{GPPrJ#PiL5r|O2rAM>1Eo7Bp0zj6J#x{i*w zZan}ld*``?l`t=(Q-ZfZ1Q8GFJrk#TWv+P5bE#=429Am`}yFtj$TR6?a1e&wu6{HtTpDzV2&y@=V_I~5~2+CD&iulD-t zI@UZH!aNLA2%r>BA(2$;H&m4ab#!%i0GND|Y`Ah9ixm*4Ysa)YceJ;|dD>!pMJ6Jy z)C>_OM=LxtZCtf9qK6Xq7WUBi-B@BvxqCz}q}B%SlVz^=Y`BP{6)-dxZyov6hWr3* zX)Q|q02CCDJ!cQV^H9~`V$2UYo;m@mvXWbPpY@va@*LyciTG+cQNP!L~*A$LHAd)B@eI4 zAmu>|XL5(JS}R@`)Y4)G+Z1^>iTUA-;+-zH=cgGF)y*!$?>*A?W3-dPK?ASQmI?@a zv9$S*t;36iVV(>&@k?^QZDbf*N2L8X<^rCDiD)NDeiapA|IESu%54L|=GIqXfCtU= zFikz`J#oXGKX6ZCE2%wkMqwRhk8o23k8&A}i1U@2m%TYoJ!_TVWirFx1?oX~F z!IBj6#u1%WOJZ&1h+{t#KdcPX^PKvKut=$%2;lw-{Oi z*T*_w%M`kyR+S~}iwO~Dy_%yfJnQ#7>^obUYHid|>?SL@erH$2iaHa$uWRkjG6dZ* z*!@YPw7MEiu?Lc>Pe^{Et`3F2UKF!}z4LaOwm7?q)NJ1^`L+YS@HL&eUq)9}@msNn zV@^}6hd0SHJXBRxAN;D^W2wCbFn*dmPQA^pNtR6Zw$LB2E?$Rr8dH)EpN(n%UNSgB z_*Qn#n|)247+g03bL#zx@sD-lJRGCgUQ~a|kSe(H!`KozOg~bJlw~pOwObuq^j^-? zsx=`9G)y))#apKJXX-Fy=)}iMzh)G1Zkf1wZf&~zTLe9dBHor@Tm41A)9VyU~+G*u@Ev+L_Vo+41Bckot zOsTE7?Pu71%avovGmPS1pZSNuT&F`>sknQb#!gD>AY*n@x0mv?>0j?Je3ZI-DNu|2;m%LB_u~<3-Dcv*!uM2|MYXM_V?{y&1|;hXFNdGj68nxKisb zn|MdoShv1PmAdo%fUL3gGTF?uovc5;@$npNtNS6Rq~99LWxM@&tNGLv0Z6+zr%=WQrHwWFo%cu*d7iE+f1@R+`~vK#sXD7-&)={q*b!2{ej5@G+W;pAGrnHX9?@y&IH zRhet~Ayc|BAVl>BB6V1QEb> z%46@9xQyhuCp+le?!VgmAxMTRB3SZ4d7M^0A=`LQX0BxMG1?XPn2a95FZGI@UC&LB&LRb60R8R8{b!fj}|6w8HYpTP4!`DOBsuvlvy?9>d#XG@Z zbHu{Vnc1qT@2#3Q%Bbybxe*^QaO>#_NKwlCWVjMw_Vz#^MA~7GtH6vWEaua8_d_xa zZyF)nmZs6nH-~o4upa9I#xCVU~sWRz@qHSX|GxneBk3(}Krm7n5a5<0)b}gY2EbW=bjK z!B`*e10Q6_v){Zwv}PA+^Hak6N{HR;-3^fAg{u7}N@C{uVLq=n>-IdNmryeI&n@gPJew=MZT=wvXfdmmDE z-6^VV9#`bOAC=mDOz9`*Y3BLOyZ&A)Y)5|}REhVsaY6}r8ptJbvbF4rRRF8%yVcjX zxDxN}KM7-9EsrZ)vp9WbjV6+|BEOAEpNoJB@J@vr6MbnYAhq@_v&+aXQJ;QscYlKNK_BZX!ML zxCC(%+Vz3%t2x-jpsDthq}8{i2SDXNGam;;Bz(zA@x+><${(H@umxAd^0<<{H8$K; zSSGguA|fKVwADYE1(1t~C|?2dHiOR2jHRz9*LJEq`&Y&|q6Wq1zG_~NvEv;{S0%IZ`gg~gc*lJs(eS(CEY(H62qn}LRM~wRQ36dcQ4U80%o16dl*cb+#5Xd` zrLFO*^#R<8fYZ1-UrO^UA9!0;W}EEoKc9!qpG?q7CO>VXYK3_$og_s|Hz9|$TMPJU z^?cBBnQl#-%JNc#fqN+JCJCV&u69YD};GA_tMOYSip6&YiI-ar^SEm zw5KOeu17OJ+D+9xAw5Kw;cHn!8N`XHh)4mwO98-NAiA$GkakLG^W`<@Y~U`kU$fb? zPhYayQcepu-Ku5W{aY^S8r7Ocjihx}CI%xb_kHfzo~2yiJ8?~s7&r~4VyLvOldr29UFrMqL%4SOto zp7;Cqcd-A1e;;g5aLqN>yyv`Q++&RE8rRG|d?KfK&;D)Y15}qkYtwXcM1I+;19o>O zp@~1;qQ-+^gvZ5rW2)+b4qZZ(80+1O*ZPsXsKfQYCmZk)bvm-FYeNAkmIAH2jfXU1 zE_x-{W-h`l<3wxGixly)H*V?Lg}@9_19=31SN-r1I>s^K=9$(+>HYouPKue!$G&~5 z1H2B=XfnY~yTJ-4>778mww0oU+9#{>DR2ru9>n^XnBygLKlP!LX0eTcE~ekVvHT5k!j%^FtjQB~nY+5x@P*j4SNnkcuo zB#zah=EL926Mgn}gQq^@Nez8h;|bYLSTHe?e7iMBHxswXG6q0%SXqX|yP?@rr{;&0 zufzgAm2AW@n~KHb8E$BhcB1d)oUm1&x3W<3%$+%`(nnh@YEg14=I8A#xoss6#PQP& ziGOYT5tem_;Tf+@Gs|xMb4u1PD4XvOHn?o^^YPXu-5F+9vd57D(gp6JPowXc>Mt_`qNKjIC|De&2yxHd_LI=f(sp5J^bW}Vo z7NRyx?4LwHrJB=ds)KI%;o)o$j9+((tu*tI-Y`MmNBq>GOnbz}K&H52ui>&ziRq|R z%H7yyJ2TC$Nb4g`wQA{XmJsR9`(0%8ngv1H%I5>l|i_=aQ^@Xd3gt_SnkTPIL_Me zu5CJ}jY8-fq{Rc(mg6Xwui_$hq{=?ez9)rZPkTxp_1{V>L!Tq;wHlo!?Xy3<*;!aH zs+;KU9?i%?ud)(~YAM&Qev?jX68Q!uWBb`Y6BYa}tlQa7#KQl!QlMeJ7WQo8Qpm&} zqu1X7S91!HI2sNrX~}YNWNqJMn05*M2sZsu7tHy4w@krN3cEd?*YJ63kc6jLwE7lT zJI`(^r}=Z0a&0i=On!fyL?29tF8k}I@d6R%xYsu;=msjwCPyV%V{|OcH`JFMS8pa6 z+<*XyJkMKi*{u5uxRe z8r5y#a~Cu!=D&$28aycGx*gQ`!$Eh#nktCHdH;UZg6ieSBl}55&G^0fFw&`C03kvp z&+CoRp)w_B?3n%U(y1OlB`!`v+OG7VGkGONKxtJoF&G~HN!yxpPU0^I&h?QG-_D?` z`;IKk=fu*fOHRERES#gf5^Kms#Amw_TDiF}VfZMXuwjiJ$A@hZic&PUiyg7~Z5WQ^ zb)+AKU2mUA`4?@C1M+;kiRV=mO4wtat@B4&TWM3yHZl+xM z$XtD7jIg_KyPM+-zy=t=UpQq$hap+!gWul>BV(=2#@yENNt!E&xXt-pVU(N42b)Hd z%MW4#9?@C9bbrv=$!pL`>%95cMw8!CuSQcrYV-1)hdLLu|FMTaH<9Y?f!tSEj%N2 zSq}bnPCClFy|Y8NAh}?}B(`yIx*{it_VR{6jcUUzR2B6;av!3_Y1^g#yt5zQz0NGYlcvD+Wkmnm^??}nf&6b$ z9b}{4r3w9Q_W$x>B&TarHpt>zRr=^(#ux+CqgRLEa&sFY`EIyrttjq#7x>$Fh1TN1 zKm$WT5d>vvzE?YluaOry%bToe3+G`1V1+1 z^PHJ^3G6r&6oZ@mPV&dHlF@4aa?&PtJ`Lrl(@u&256<2cPLWsL9(=mYb(8N$6R^~N z`efiCCURmmU5t9@%Sqt-A9HgtJv)m91#IJRk8jduP}~=urMQvA@VuBp)8$yHK=dZ_ zzXZ%1ly(~ut^CpJc45$V)(D0J&CYH>g}a1KMao<`>wiuodu*W*7GAu0dSq=riG@P2 zlVS0W^U9TN2%?@UegSXPlEkO5u&UQrr^GzZodgXHt$~C*zo4KqwynZ)+H;f$ z2s_HeFpW!FW#jl*F1f+e0GAQ&o(B$E{sDF5@!(G_%zZ%9IG(~EB)AT&IRf~j1!n=0 zCG!8~$toW&32+{~i5K9XbV#%ZEh^2$f|tLx)0OUyHnc}hu8oux09s!tDK*0P5=`4q z+d<^heBeetjmpb+@oYw_01mbNu$%@cXEFBDS&;`@;p+*T#m<`}3l*D&QY{g>vP`@t zGCRxOTUE|gZpXU}e|k^gpDS#RrMu3G?X3Pofd#FCCNz=cVhoV0oEqYyhan)Yd43h#~C(z z&QusC0SSOQ22v7|JrWyV9>ldtkZUBu=o3cM^IXO9*m}H#q z63E-x?bo4Up6VjuClmsthmLG(@)o;1q4A$AGtW^M0^g^c*%R|nA>M^{9yl>utvCoE zOqmZB<~=>=NfZEII;T*pMnEC+kz70#`)j>%p036d*|yD*iZ=iCZ0(8^?a{%J zZ!)Xl6pyn6L&8+Ht#q|q${_iD+;s{m$|U(aCEBG9TX!NoWu+GM+m{p;JLiAn{evJ- zu$%SXgB)VV5hO)Y-W$H|{}IQ32gV2)bRA}fqOe|53$~7*;OC$8*a~Nk8ptJ$k9Hn3 zzJ9;f*BxC4?R5BIVv4jpvE*GK9JVip~FKNiP!f$Ql4C!!c&b zXpV_CQ2F9r2s77S9dI>oINqaUC4KSk!maZSLy!R_VqX&*`xT&ryb}i@bpW>&)e zZ)GUR@y0`!VehTk~n{r`6eZ#%4-H?&Noll;igfLXyiE~U9I~= zvy7iT^KC4$`rAoe^IaA-kqX5s?@XuY3IlFz3Aqdp)k?pM&>JgN@PbIm?g)S_{j6nO zw--{dNPm|qxPI*B`F(nNy1>f*k@YO{iDcza{Rh-j7|<1%Y0%-@gee`VM2*rE*!#yu z^`#HDjlhlX`zC&%1`j4{C~`4J>;$H?8*tyL3)P|F^zY~ z&9@Zi%>O1Sep>{^RI=SjA%ooubMqYi*?rdHA{d!WAdhmAQD-i!zyNFL$&9xQ3lhj?VZZZNn3R{7ZKB5_&#pX?O6$c%Q{Cv z!~ga8?vVP}a>E?>Z^@7ML{rG@6={x6QSU`^#{ur2u4RNxV;@w;QW$Fmv82IzZ`HAwx@lXHiy%TiWm8jgLbXc@3#65oIa3Z|L zL_D>Jbqd6lHDO|QvEZhe=@6p&V;#(R0Q(3pCrXku?(E zk)~Ls4nN{E3H((T8Kpg=I!MJJ7VKcDgb{b#Zh0d0*piK2WCAc^!14p2&G3oKx;{R5E7dUMz$&!!0?vv=+2gP5%WM@$x9^{hQxgGd zAOJS9@dERzp^Pjn0#uA^yz52GD>6(jHv<50j-73-M=FnIOrIEU0|Q75TM8;MW<{uvp;N@ZZ0u{3%HI_ov8{|6FpShX#{2FM_u;5$WZsU4!da zCL|>M{F!|Iah&Sv&1bgaz)L~NxdT@V#d0NZTaOZjjdg$N}kgqI~0lu%njl9CMoc6dX#VVkkaJ#NjRPMFcpY zC>co3ROpNUe8fk7R8Uqfv|E4q_xI)LgO_y5dmJ5&*x2M?L#KtJj%7(UsLIiL|DtVy zgM12qZE{_=4e6iJ`Ot`81;%>b+FJYR>5h2>779wd3HcOwadzv^tjy_(+fZpxbN|n1Pb+&Jf z8o((O4-L-${_R_>8PpI-4$#+R>e}6zP18iW4{YFdDjmG380L`Dii(Y$ zFNUBXZiR5leYi-aCgG3IYg6~v={D-r&pC)yt8p4QrF;a5(@K+TQvs9{xyxruJzmIS zj3>BE_wNsv8)Vyr3}$P)oS$+-K@fp7!~%;83$L97E%i!}ML7gJ&?eM0t6K59SS9=$ z3WS6Z&Ept1?aP>h1CHVXWp0Du;cKTTrOK8K|t=yuKdQwRkLDi45-V_G4uO*e}b!sntMz zQ#oApqHDs@!RYPy>M_2h>I{id>??Yvy*Wd=gTk0O+A{iK`1OE%-jxJTavqFqu5bQqQv3 zpD2LJK(dEjYmdAMdcS_zZF8_}?W{HwBM`>I#?w%my2V!-aN!*+d+QMm7PU8l@CVX^ z&Bb^t0UV3}oMLUAkBFCVCFx9+LY_hxv$7(!T-Ds-g+)!vZY<+b6BO%n;qj=KHToJ{ z0E_zP=i_7FI|y4_4ns<6??8fY92ntzwR-qAH+PGuN0J-`3WJ7f1HQu4hArJSOf2$=DmbfS4L62<8m{WcVo0>NC zI~;_s3n}g);Qm^X2jS}xJYolUN(1CU_eVKU@F0%3^? z4`d79aghhHsFTK@KnsELKL5vNv9UdJGc!l4U7eMcm7hO<4m^GOXJ3D~9BqR_t)S!f z&u4!+tk(y5e-iil>M*E2RRiM%N-v^w4`)CDR7OK1s?3k~vBd=CtUPG^G+smATP5Y+ z=djWDxB#3G_);bs&ad1K8*WXMSvQUKVu09Q%ADoaj3}Ew@^0tm|K4p14bEvWopfm0 z#c0jhC~V_9+uK)ga5&9Ib_LIyF60%E0$1(K+Q_rvk=u~H7$~38aOD(Om**?F=Tels zJUqvcVslyp>M>Zz(1z5cq@_IIZ~A}!Ou=LG8k7_kJE9{%8n{Vn@d)a2QuEF9^^MO*c_Dr_|bSg38KjSIzcRYErYDFqVe;|c3 zMC?0WkFqc`pDCd^A8e>hBdaM!hN@G>s9g3kbwT7IUFzG&1RhioX7oY@!+D?JSx68{ z3ex{haiASC+~EkrlxiQ6*?*m@7>Ra+&kLZLUw@w zA5lxfq?+~&y)B=11^lo7aaSp|P^54uE_fXDb;b9V+iS5Caf_0RiG%8cfQ!R`ZHg@Q z0j-$O44EK@qTGU1%SJwV#N-76|3Y3#$qOCxDrI2c#c9`ta^yQtPtRPJc_yrjlbQ?X z&RXQ6DHQ)_Ha+v`>gZ7U%aK#obFURp|uR=?gGySVS@ zBKP7rHxJ0v=et0f#7fV~@=lYE0{!B)jX7-;8|_We3#c!W%s-fP*To=2uCtp$f9K$< zE5Iy_`pWojazz2W7ghd6l5X3N9qsLW)87utFIwc>?x;(i6i>-b0pE#b`r^d>!h-*F zp(C0Qn7`Ysgm)5`Ud%ocaIf>kcz)K%S=|Gr=W3J(*N-k>fDIa8@LuhGW~~k+ydx5J zRBpRU>u1sx&u$#-{qxr^$!6duCW&0K11SD`F&%Pg@Ou8S2G5#^55ec}8B!x}^kcgF z?CZA#VwfvEeZ<$Vn-Bk>e)LFjS4X2#$jz-6vXU$v@#9$cVo&$*3}hXCH8n{e0+(;y=jskpPxG*HX09a!0urxvnnm5S z6}9{pDA=Og2R$jmZQ#r`Z4z+D~KMzqWSyD=PjulvaMu71rt~y$nl> z(Kxi0>Fj*4q?&|^ibYrJ1WLkKJr#|FH!Y5bJ`>Gw6PQv3r1i4n;1|pNy)oO7@@vHY`gL^%f~9{WPqe{svc6 zQSnv~NLnPYI6VX-sfm3H1km&o%KM}_ogV;%gY8kC>C!K2>j80Xu<98U3gBF2#g1a9 z|Jy40e_9>t?{B{Uc8YiM=qhe;PL8acyq4DezF$?nw-Mk%=U>THZRv~`WcA2pr;?pa zSE9%67rZbx?Q3gjG&MC2yct+V;upBi zQcMyQV1NR-7-(_O%f<1%8UEpZQsUl+S>E;@-IWr&=(pEkL#jaM8%zz0U=U6u4VfvB5=EFSex6+=r&*V6z3=w?M9 zqjabufnI)_pP!$RQ7ZX#(fxc8;>*mRuyZLYpfITXHQ)aZS$^%P>`9r4??H=N^>)}L zw99F79imU4er3>cH2C$2WaR)jc69TQ;3=Y|>PKwn=RbO0b&ctC!@j1EcQNw?BA07@ z?wD@}*-^^k6@B2;HP3MriGGG$IgB(B`1X(Xhhv&97R@P@H(96QSFEv%S81H`d=%Nna z?e@pLqiGXU)5OHY0nE8Ns0@K>R0EsY5(=L>FED)coe7=6);SFwL{BOIn4hwjW%}L^s+@Za4BWOW#E&^?$k|K> z?_Wa_L?hq+S+y2_S1mTR2fh&8xq(yVt+OTZOFGa0xH(n-x;c|U3h@!qy!JU{f9}i< z>mTRH#djXwKJdt6GBdZ@>Ehg#EeiR&b%K9fsl%&nFcp%Hsz4wW>Fj~iMgMn++(X<9 zhr4LOT=9~%`TMk}|Gyos|1Ur9C#GXr83ThFJZJ~j%TYQyy0jQzlYG6l?h749DGcE# z*>wE&_I6>R>H2Y+?&B_n%ns?*iDIbb80HO6ND$-D^M-8b^uc(=kyBJ{e}BJTNLJP} zHqSu$_kVW;Uy{QwUv@j9ACtF<^A^tvfsyt#H0k|{f%7uE+h;n zBDRDx%bA~VDeH@~HQTW+D0%JBqo=MuqLD!R)3oV$vHWh?!vbFWW1M||gC>8oOZ9>i zN7vkPKWdKO`~2?7cwS3$^ECLt->>|gD_5Bi2qs69T^*;rHM(k*L$sQsW zEiMbOLJ^HNbd6ZhLWp^4a9aR6ES^t1`!Q>_?Pz0TXM-Ee+(GwJbD-WB59MHBS+ID- zKen}6bGF_CUl{Z6sNU3?n$*pI?qty#9OLrXd{4^zU}+*bBcpX};n%x8>cs}%t5Mh1 z0|y^@sM6EW+{CPQ=s{%@_Cl3&*`_QSsVp2c8?8_~h~qEkcRL-Had2`X*RPgj<$>Tc zuLXSNz1ke@EDyBfRdB|y{$o!|YfoIh#&PUuKc$P~G4>`UCe}AHg49`@VFiWn@5=ZdPtCXT2@a6Az2AX7k=s zFDOm&^sErm>u$)+&K7+UWq_V*tu?f|a%%X#p{~dA)}b{PpFcaibYx!L8coV}bwNhN z+42Q!oU7D4wzjrHtUiz{9UDuX$Nd&{E*s+P!OG0sQe)kOYgXG~V9fg~?H=oph~gR{o5JB3R6SwrB@Zep;e#;Ej4n2o=Gd7Vf)QN$N5hN#|?RM zY!SY;%cibJy`sNWb$1yezMMk7$GFtM#H4jeAt?BEmVLk>M<$wsgu3x^rQ<^r{biU# zq!gS+)6~2I0y*}%RKa?OydtuWRoSI3N4?%@Z-c){FP5ap#Ky$NCcpfJ4*n>Y#Retu z^Inj38=s^!oGCBVigO&Danb2axw+2HSRR<{uRxV!vpNyk}R0+ zdDPggGhVsPaB_en=rs3(wlpg0Q(z=p3g<+pPf!9m^n;1_D4t^i0%xIy-k@O_Dx>8> zW{sQ$!x%-)rYvt;9Q#a=qIG7)q0rLBQn5b}CY*X2!yB=t#DNx{f`aOPU@nj#aAwm5 zbwA4QZ!;yfurC=L*7DE3O@Lt%dXYzPh@L0|@Q-{&Z9>a#T`<(ar zRTc})P1D9hw3jGdw|EUhX*YVRq~EOZ?VXE~h%vLpv`y%8)r!6}o}C-l;=3n^3e}Kz zGS45m`=9s++WoWE1Bnh{>wfBY>+`( zd|CpcS{6NZCe{0zWiKPwh`Y8JiEeInfz8wC#KHO+(44wHgHY(QYL>46wVx-xyzdkjPnx3#_4O!UOj7m z`9MB;*&FyBwPbbEudKaYwzp*a>9eR%f9G1<5MF5NRMI3PZGaI~z*W`&HYH=nyfmrk zfMt$?GUa_Mf8Bx+6Bm1QeiV3+M+ZA@5yS=u6NWxL z=gHP7B0RWFI9u}b4tkM3_a19WyYH$17J>Q6wZmG{0BXelm@8LV5REq+B(QKVO>pnNRqV?$G@dQJpWl0LPN?5>@xs zjg^9v%yyh2*HKmIgXuJf`f!?z+}wyojj*YyI<#SQ-Y^x~)zvgT6O&I;G;Gr8s*fIC z(of()oof0p-EcoOQrj$<;KI9mWAY%9YGVN*c%)CSnkpBapux4eV$u{qFch?lC-4k2q%1eI9zANp4jW)I3jUG7`Nn)0K`pGDS#-j(`&{6{CH_i#XQ!}ax&5k^ zs;VL9?lzYVqPC#p4aK}Kj^lw#F~xQa0f(yXF9^H8&M!qOYNe6NvZD^%^BO)_(CBm~ zQZJ(TbZL>)H$5{`gkuF0OEz9Mben0w#yZOL3g3Pv=bY5hVq<&wz_*Cx^{)lV#uOVr z?S1fsnuF;+D8(LKF0u}iw4 zG=-+i=TnO#z-F=HDlWnMkAHzl>JTqoQ~e^tQ7dlGe*O6QWN}(rw-v3fs%uo4w1=

Zl`2lWJt7T-W=rPpr1p`DN-yIK;R~X5vHWW!US- zDXZCOCrX2|Q_JSQNWBiOZF|eCU7qxo(78YDT61bb0%{Fx$1gbIE>&a|5@Y7*OqP-! zeiyk+Hah{{!?4NgrM8ZayqFR~M%DhF$+^+s#m)e`v_J?wyuTqd7&XfoC45nD5Tmp5 zWkpPDcWWnkp#GNSv^b3$IMf=AlWmTZy(h&n9+UD7RnTx(g8xTx@nEmvReUG)5r1yj z3)QndRhS}ZI@A~IuS11_i7|b%OpC3Qw6y#eD>Iuj@XGj9^4245+nmu~HD02;iE6MQ z{dcSyiXtL^FDWg`8&H1GGBmfsavCi^LVFn)v3R7Jr29(xI_uM!!opOgj#0M_8TV-c z3<1QiS)Hk9r^#!*vj55CxoyvVT%kc|$13DfaOMn&UlSJ>r|xzhNt)hth&rcDmSeML zKv-(DE!Q(@^AC77z2=LOW}cMM&Y=Be${F%wc%GdF1E<^PifY?NaU4Go&WkYK<{gOouwguihNx zAIebd_?FsYM~(2&Y>h!}ter(cE*O&u~uyb)(*b91EwOmDUBMSpn}v2pfy94=rS!ooh*#Pp@e zioGO*(wx>AA9Gs%3}jTQYN~eANhL0PQ=}OvscwhUbzB z$<)@43cmm5tE;N&b1P9?l$^q|&g)JTEy7Cc{5r?alG4%}49Pf!^X;!nz*r81Z|F=_ zEta&k@494b?45*tzw{{OTX{M8d<@ULx@m!jJ;r4chq4=kje^nl<(nlAC_P>7X!6)Q z#IGibU)3BLzqZT%yMr3x!74o(!Hr4cz)3e$_fb~j!{*on_h=1h_HU;}#Ktlw_W6mT zPsm&qve1lhe_{j38c2q0#FlnkaS7Sr6&_*u$Ay%bT3uaD-cyOcrK5q+r?R?9#yF|P zjF>n&L3jEA#rWPuruu0GI}nQ(qeCK3EK6WB-B=9slky3&xu5N7P}HjY-x5%-e0+kt5Lk`nu&+=KYM1~hq^n6#wY!{8kxvv zUk845C`HGx6M z=SMv^nv1Q=nNt_L?6KGRd|Sl*j!^vv**!#c|J7x6X+d!h5seW!^3K6@Jj z4O|F;>8j($HZ>8Fm3!D-l`ZYYy5Gf|;?YPQ-H7lANVKyi0{=Xx=Bfsw}X4F}THWef~~B?IG=M~mN3 zH&f<#dM$KG-SW2XVpsn4I$DzOo#Kj$ihAk9(FI;#E&lx1;Q;SrhKfBW?;}y8rrGnU z^^`Gt??R=&=_K#)+EF~YcrbK#J~<=|7=4de)Z@6LTrM`NF27gUgulF&A;+_Kw0M%$ z$QI)TrCLc+;B&Mj)&#mQ%F?gOppSHJIPEWuRH)fkQ3&?m)J$j1TWoukQ39 z%f}uQmSJI`8ess1K<{PSAcDwF@~-*gH|sBJOvLiO-PM}{jic@cOiSUmBc}rLqkxRS zz9gK}FZQYGeoZKCve#HLxfIh#t$b$F*&h%r`3~H!VvFglJ@!aVO{Jcza(tK{6~Ec@ zF>On3wUs`967eIhoYThvxr20y*Y4TRvauy3wGe$?9o&D(LGqeOHhy}TKq|ykMC0}ryoA?Mu z?2}Dsp-$Gg-_lx{NQ__m-?~2rQ~?+6Q2Hds9U52~Z|d+P2x=Vr8htj~@%rgK;yLa8 z=ebTB@mBMihq6QWupe;p4K`4IQ)yn*Ye@Z)6m$!}4#e{$rFde0*!l4-Wm$rv*y6BBdekw zV_}Yjm#K|!94}9hEQw9wKuu@iudK+;m0zHSW?l5LQY}A5&Wy(I88u6+)NTdMlChJJ zhrPABQ?#|bM~QZ*mUxPOfWN<|`^@vpcrMN@!J@IT_&w>jhd0bezqvobg<_9g%7yJ@QdXTO>w8NR7&Xj`Aw(;q z&H|$2x(?RZeCBfKcP_lW9r`bCV35q8SJNcL3jYB=?#`i4n1A)U(v{1IE|(W5*5-J} zS78cU<)Srjolm}%G`m0FX+X-N$P2TTxI=Zex-QW1p=8_9%)`@5a(N?m`>F98L0dVB z@4yhObR}Mfya*2I&sHMF>xY-Aw)R+GWot&6`Ioq_p!2&BySc6LMQY%><-hLngsjgC5zJ*&3+HlSEwxl^oVB*Y zwea=AG`YAnGE!!o`)~VF@Qx$Pt>(9ZFGqX*#8Waw%eX(iIzL}gXQuhi&9Am|dcZOD z=}l8_lAjh3}10Fsls+0oBw2ccdU_{y;gs zd|QwEjMtJbB0OB;L-{i5;?#CA-ZAFW#leo~kOcZE*&zNIRua^=EiY^bGDw0OQ`1gx zARp!qFPwu2AW2VkmgTH?^}mJVkb3L76vGYL&xaYsSM?d_=C1(0FU;17@#`19-r=R~ zCN}mfQ)fE0^pJY~{afAPCnpUh!shehVWH^|>f&WNAGDJh`W1lX^pTnm)c)O;Hh z8lF@NZ2@oJ+WIV93EYP&EEd+b#Rk`wpu05uXM@?%hEIv!LhcFD5~pT$1B6q(MPBn$ z!L=|=Gcq%Q{r~Ax+~THAX7{6mo7CAD|K??l6Ay_G_c1EXo2M>$U9KLdt)ua-#H!qs z@b&!{94}Phc~{#;`AW0o&0`=8u;NYFCpG-VkpU|oxp(u)lo3zJL?(9^HZ?QKCU5Wb zUym71F$>*iK|sw_Tx3ZKN0j4df^EfVS8?G>|3_4w8Xcqhd>S|jlV!GSG&@eMMVX6A zN}X09rzXwDn!X6EgUM!Dcq43m5nMZ5S!rd82-#g+4h=0BV#^sTj|_?qLB2?WrK4hQth2SX(rJC= zJ)5Mjtsm>XW{wNKqQ^)-3rQjnxrcS?g?Z4AwmaV@`Tk(122{k^t|1|OM#ffi{nYdt zVKOqccH-y(a}iK$^4Fwm%LEVb#9 z(iN&oO0R&pA+g${fN|au($r!UFw>2&aCOyO&HxI;RHL8G+ryce6GN@f09p!c^|wF* z{a90A;wY%BPTH{X%krS+1uMgHAz9CYkhY`RHx2o&;5U#Bhk(eWGQFk27awBHyvJ@G zu<^T=Zg|bj6?7b0oz*U$C^S^qyS@}-=j;I8S5_-cqLPs>uiJqe$76r|ne4dtwMX0D z`Z;8s*9?6g$3kdqw>r{-Yu5gesOc@&XtDW28m%8T%Y<*7&zyXabd+pyagxKzN(!!= zzK@uuCK(-E0DT(7C8f~LTJ*o)p+}5z5(P5F@Qrx`o3GTUeJL!GQ-Kf%7^U1-3QR6E&mfcx-$1LvzvKIk!> z99_9O-HgwqL>(W(lV4q_l$}Wrp|aH~s{it19emfxN=F(&LBXXjo`I2n67;UF{5Hgq zOCH?t;?DHdA;3rV3M@6)57dZcc8h3fm=_AG5qR(cwJD@_T+hq3`kz z&lUrME1L`6=fG{{c81ND|7)Cliu}hJ4qqIrmyU4p=L$_7LktPMsZAeSYtPoUwmeoE z0y1zo6Jx|mQAr;VqO(vlP7gA7edgx>QCRrXNCAU0EVp*@3g=K6Ee>LFsKd{qg3Hvq zkOxK|79H2XzCiIaENUJDgI}pJL5S=V$ua^G2R?WAz#qj%0XgA)5f$z=mLe@b`w$-- zVI@O!f#7|yPEK-lNBzM?RgU)k(~a@WCZqUu(SBsIJk&D#cY&d@ib}>mk%g7;YV`IpBb z-V7Am{CTw%y>Sjml_-)=g_NUz%~LyX>Q`Y`_((Y5xTqVa>of-iHLyN5Z}g?^@{5cO z3*&YhyCKE|{ARSU<+Mh9~RGAoPYbUDFKCsj(x5Bz-^cQ1L$!c|J3zsRyMNlwAusYuwLNn*l3JhjjArsG- zAMtK2an)XoVAbsluhq}l7y@SW1eD+@EvtN~ zsgT9%4p`V!u|c=zTdk|reU;l94!WtmIH=F zsJ`w=z-YcSgp%`6WGXX*ao)48rWA{m9^H?7mt}c; zf6=bt#f()EdCk4ND)jU1s35@~TL?z?{{-Z>nUK^uAfK2r`h!>|K{7OJTfKud= z1I}q>nf3ieLMQi@moZvtx%4|`T@zPJ%PQWAH{ay9ZB6p_K7`mz-T%2i#cKKsi|=@r z4VSruW`?=g$Kts3W6ZU72={(Z$T5qT8FjdjV$0*U^~|ys#bq8Q(nmbpxa%0g zHL6|G$8flwpYcm{Q%rWK(=&vp9AI{-GIPKBWxQHxqM?_TL1ke9*xo^~>Is9Tzz1DCaI$py6MC1qf~ar&|u8 zO$DV{$FpS^RFwQy!&bx?T}=*L0@in1jMx$d5wHEtAXAkq{FL~u=hH9ql}}LjbSfMa zOJc-?)oqa*0xfEML&kd40W=Udg-}SIg>2GcRT-h~NQpN@dBN_dwzL{XJ@-(I^FTJeG&9yzHtT%No3F|R1{?sLPnzbrA(=tYef>Q~N z(Vw?2#~Pyk63IvT)!^~8a$PdfT@XLDS4nWBf>C?wE!n|(-)P!OUn&vvV!aY7tAP7* zuV~oiA(V~#@uW%A2+-~fLrrtaNlPQ>Ym>03A9qDeUCPWW6`s8|lS}ign__Sgo3N=d zxx!K?%Y5P9WtLR1$-8RQw`j9OUw){{N=eeni{bpr=&@BT_Nd7(S0@nU8Cyp~fIOa_ zoj@-}JqKlg`6EfK5oHp4j{ta|u5EBSkiWIx@ZLMbe?_G##4`R=@J-4kwcgcJs;_x3>4uLSv7l$%)@&Y z78<$My9yLlaF+u#Q)m4{z=(rblvW0I?@H9s#H-$_a&-`yA43d-Yw0!6VMuemAWcs=dnizRJjRT%XgntvK zx&ErL9&h+vgghxR;Y>#Ar5{4Hyw-OmReuHgZPkL&eyXRU5|&l{t+pX_N|q}iqkLt> zefUPsVTLb=Y||{uP!=o zc2zC(QA3+PuV>LEcS=ph?5ypj$!r2-WMqVzKasR4e~|q%po*v#E=^R1pAn#au;c-4 zmo|#|QLklIN4}HkORXa=@tFl$&B=YI{1>fdwwxD}5u~hFh6g z?U{y8aR{wY@mFiDsfi`Z@iIlJI_TxxOx9+Mow}a$YuysWp8OHhnW@3J<2aI9J6ZXA zNra(S)^zEus_H_c-)gE%>=(irM`COiQ~a|DxpkHR*=n5yvUlXS4to68ooI|ubmXVr zCnVe{CEI60lX;6Ja1y?nq_xF`BPiQj5%f zoV3fqT>i;xV6_W1g5OyP9BBN6NgY01aXZ*KZlso)fXtLi)k`PWdS^1sM>Zc)Z$7u( zJK*j(0fvKwQjeYi>pg^BgC?2*m(}(C&B-CIFnrJJ7LdRYD9|XfDl-a5jh8)7$Eh3> znuvyXopkh~gJ>gO*hd4{HoB#W1!|vqmDjfYyfi^Cp))NiR!x&>zOexY^u^v`ULs>1 z!(vzeF@C)5SU0v)!!aKiW~d$LMXQ^WBw|jrE`5tVz44@I>44^$!%$9uQAd==^B`-= z4^Jf*3Mgt=f;i>N{boms**)xakIv1-($h*yRq|JMWC&`PpEy8J5lt~^XFrH#%nKCa zPoH{2nHJpT%*cpuhsryNOZ)te(+;bo0M&rL?mTbm$dB0P*Zb1gxd9*tW-U1D} z!L@3ts+~3hO=kB!Fec`Ee#;ho=WA+vWWKhrojFeH4J7{__I~i&99@hI ztT(@p=q+EZ`iMK(O`q5By=4qt(HLST=b8`KY&;^Qn-{U`e=m>T-iIEC_hzfdeINJw z#RpW%NwS5WSFf02YdN#Ji01n8>cD4F5t#V=?82vwgPK~W{O4*L&K1ZEZJ&*QAzP07 ztm^7UNE$5Qp8B$}qVIm{LB)S{EP;c9Rb!B(Eqy!dw7hrLLeJYl z?1!O4?%*0OpR2?9zEJx{{x~J71urWpfZ<=5#FLac{4U>py`_IHkr-Vv{y=ri2nU%4 zb~Vp;K457ZGMf952$IBiv$Sd0RP_xE$gz<_xJd}fpQ5N?)PSt(%o=*!$2{rZt^J;b zklk?qr}Nf*t~Ixf=VJus!N1hvdyh9{7O|nJm1*xLYbIe&e3{9S?#zfUJ6n zcK?^mo7|}Q^iN;*kPMh=H@CO5i~W*j+bc{D*?UrV1C7->EOL^f!m)3g91Y)W|94U@ z7R)_1>`L~Jvl$;Ws|W@AmHj=iH7bGkK7Ej%EQcA+q|a9nET=tXaQwt#KAO!rW#Bdf z2BQ9V5B*K_(nr~iA*hUB0K%DoOi-t$&Z{Z3vGJ)s{)!i2z<_bKKkk?8tlWYv5oJkIaHq(!!*pK|BiEg*3-+6ztUPT>ZB^BjWW00YIbKDeKRSJ@v&M0OQp9;&D$s!szjB|2g^Hl3 z%jC7p`{Y`E@a3F}BYmD-#aC0N?ylSz=K*q`XgR$@Jn5+Pl-J;w@t-%R*(F!Yt0&1i zYkf_J;yT`3Y7=gD-g&9~p`PF{!0ydKsAZ##>-A#*jC_6f`ZLuG7YB_L3n^GG4?x@# zDCAb%R0qzkSxjrCmYJYh)NLZ4j+lp4Kszh_q|8M)>rHOKNi6gk^Y`i~C$4yQ*T}uT zkE;#K6m2#GL<`$|u+C63j9822hZa{BPQR#y4Q|a3TZh6_(cnor@l$ttA z=S3z;bqI+M2d_EsHMVeiCnP3<9xw1O75Eybr%8e@CkL_rGhPl4oRn!*AxLxDnt2a! ztNi2B;+T(H0}(L*!0U7Om9gsXpTjPeOxD`b^HIOkFA~oKIP&p@3%`CIh8v0Md5DKT zlnK3W6zkYCzjS|5;l#_ki0cm79e*PmV@YJvZr&witbIhtHSVU|G?(IY@2@*4G8D8P zU}dTRBJNi_rD%|ob!2) z*Ln_!6yXfR&)%7cPGW8TQPxbAT(;&t%dYNG`LGF-sHd}>H|_FylcwlJQ~hmizqBbn zB@nMQU~~n1V&pkoY4Xo$EvM?-AxVmd*@@{@;;)||=~`x%9{3kzB`@nmzF7(;r7N=? z|CE{8U&%Z#^b3Gv*8=Y&A_Tpc`Smq;qf6ZuA039 zBOfUjuN^g|UijT|4qD1KG-6B@|9xx41=ON^_^6}vY>gQDl*%Eylf_ne){ovNeE4nn zsh~p(j%+4ysq8NYR$SI>{1GSaqnC_7WhZEH|IxeU@|YZ40*D#@04W{eOBUey^>psp z!nW_T3=!#lyQFzet<9lS#o%9N;!FD*g9elUCTjfQXpr>I!ZHcQx6V_Mz_CIDk%|?=wRlMoRi`M3fhYNac^zKFu()yD;jupv)-P%qm{e z^4P>aumY-=7q`NGr0&pk(b9U1Z#`T-=Oz{(inZ<~_ZX+k`%PIyl2!C7epkb2WZzF)q212LI9cX4D zv&{{C6jAwrIu2aK-N6sy0k z0+Jv6@Lu--%8#X%T;hK&`Ro#w6)oVU-PcVX#4*|ib-zop*eT<95oRocXE#E^teU_e zI8MNUnTq((BbV~(3|e_-wqJaj$lFWsNEt6(yjbeGI#lDH;OrHpGY15}-W%xur*!XLWI?uC zid{1WWurRszmd%YL<)jfdA2h|_U7|%a&$^Yy~ zxjPkR$qxIp*5>bPKo6AI$sZ%fS$VFYKx5>1wc%{lqHQB|wAAQ`o6YEWY`7=`?G^fe z{}tRqT;R8Q_pnt>PO)Jt+#dAJ`NhPlva{Lw`T2pAlljE*Q|~OS!o|MHG=%0qf^0kn zZvE%`u{Wjlsr(Y+;;ep7LI#>Jb&}1tG?*VlQR9D(UM`TT@cbD<0owRJFmH+iA>bx> zrecH#yE6|rH<A~6%Xaf3J?9up^M zp&G+$0KBL|Ck)OL;5t$;QBV{+&9{w>J&_m!T$z_u!@*1_Fy+7m^kS9mSXry+Eq^S3 z=zvsKR;I}Y&)jrW7ZVfnDpi_pgr2W>^N<8b>ozmHIX7VBgTv^j~b2!;k)MH81JqO zQffbbJo4@NBZ)1b+5k7qP=bG?-nZtqbR*D=%E48Rm5#+0he#ZqgFv}QsE$nd39KbK zV)#twlQpLk%sppl-k|bzG(Gf(*1QNCops*Bx0JGHO*Z~}r76Df<3|i#Np~K-ucA80 z&E=HPAv^(ct@ahSv@qIHraNv0Y20?WegOFgzUp}h^WB5*WM|XNXK5_6kIr_D&cD^y z8$R?R{$FHatcLx%O(oJ78{^8z^X-_1+L`ymrKGS@lM5?^Avs}EhS<4#u^yyQs zEKdG|t#|(1vPgM&)SF<4}u&$`^}{~Ru+v@?VQr|}fc1h%OJcOa!@9u%4jOHQRG?hiYe zay!)MB>gE_H_jRLqOY8`HZ>OP;s_^YV{od^1ha(?akyKX069e)M&qE}2%oAyva+(` z(vuk@_&>iKq~C8v6ciN`9Q|^?ex)f6l3lsHKye0UyLwLT-Q6)#*Fa{HS?4;lgx5~p zrmq^BN~`6e9*7SXB*hf@^?nuLecFyOA8L}NJGK&-`fvy{lEdoosW zG7?x32UUX=tU=S3h1pSZ)O+;C4_ZQO$RVFwi%WdCgU6?h)c&=JCIc7JEe3-_VkF)ZHAY$yeN*oRPU`KRAT}ptZ8)fg}~>j3H9{Wz`IjYpdj1 z)rrt&daRIyWVJ5QuOOv{evPrUvD@Sc~kAZ|8-pj$;!%lZBB3M zNXNsY(&vV%*#RTg`Q7uyb1jzl3DQ8BPsV1k4OFK+sBw38I9YuEF7AKC7-sPI<&m8s zdmSp&9w=aps>=1&$QvB&`=7fjp}a_B}J0c-QM2+ z<7o@!SFO=HM0*aBnlm$~VQ=jDwp|PP0qom)Fjs%<3+_&J9Mk^uRXBzZ)R3pPVlrdg zX}qXtYP0PWbz9pE7Ky)jjGqZ>15zT@@Ns}u7>@|MXtzoog|Qt zkMEJ$2zb=l+O7ks-hV69D3=5_!}K3FO`Q6?JZ0Zkej`Os(Nb-WGe6;a*XBv{+d&^5 zb!3Xp_d{n6o0{H_34d2XE~hN>UNx`ZmV5?b_xK%1GtkjI{gve5>88K^1>#L`zKUy^ z?)E<`wf3t>;6yNYST|itt`AiRz9kq4*oee%@8< zO1b0MCV0FlFPNRwFMvgc)+x(TV}8c8Ywv$N4&o4ByOl7fPtYr<)dsp24M{@JKe4Pl zp!CQLeCxhGrtRz7nS~hdO8jG8aCSpg~O%KG}AM&Ed(PUa=(XIwM|sk6;PU0ti@It(Mf!})Lq z5%~Ubv9U{k%!ttOz83$4rP5=XitDP_;YfyJ?X$nqCiEJQJs#ceo~l_qmbF}ur=dAe zZ#MGBlnGJcM5!IlgjSbYkY?~2Zdr0TR=MptcRzPyK( zCw#6#yZTB>QE@NOw=x@!#{nv|c7nm8o6*YBTTM{Y|4(=b1OO}`QpQP5bc%T6#nu65_*6X96h&SKl;#tBBFt(aqP z(@-7IJcEt8eR{Z#)H!T5D#gu2z`yzX`&k*hITy*`_uUTrmep{1m^pLsd{Z}N$mIB?ZE4%dn?4yBnjBR+|^JPyTW?x*fOKdiSH|0uaYkU66iBBWv7O6H`*`n@8P11`g=$kSTP8 zG*kUWHKMU=6dP*D)s+=^2{9i(#!`Isgqs*H=atpfP68w7Rj`1%#k4^%gN2X3G+3KVvPNl-npQ~V6BGu{`fyIGR>ahVVfJSX~b9w^0T76B%%#{yV7eVv;n`2EfCZ>|l zhv2>9hgW#ueY64WO-bQD>`j00X}gA0(Z!*!%fIu3*%&0a1bEitKL!6frQr^V7H}__ z1ob>jeck|sN;w_fGMXSeFSn55yI@6kX0w?C(=ZmC@vdYc?7m!k4^V7_>2T(yd*yzo zwW=s6Ak23)=#Y7dDp3v-;iHw2MBlb zuU`+61X`VHSKIj-KH|*eg2uy79T*3Gd80?d5j1i2=GK5xbSVo8rCscM!a!SL4f|>Y zCri1?3AcE}`{%eXOBT8LH^Py_UQ(ME5@V|Eg5~nbf7%QqzS2g3T*)kjwQ zu3~RPDvqJ5%KP#OfKP!be$|RPPk%GUHE|K>o~Z z@ZMh*#kJ(&rU=YalVrQ-nl*+XCfaCxlsGJPQ_PDFUS z@$YJ!=!O5-pIR)?Cs&lT=w@r>T5;=$jUv$aG7o$~hjLCtK!_jbA(ZplprG?Uip<48 zC&8ul)nC1X|GAMZ4FUm*a5WKnhzt$GzR2Ql=vhDEmBbOBftDOipU@1_%K|Nz&QH$X zH!(4|JZWEV#v%(2ih3QqD{%CLPq(DSI#!-fMC8}3aeuPdW92sLpn^AgJo(V{>Um_F z<1O5Gz^I!P6BEqo>kep&p9^`xwT$vW^5nw2rz)AhX<|C(P%v3J{h1DTVvvX(b?5F_ znhanTDFmzs2za7-NQU#WL|9oR1WS8GI;JN_Ugbe=zAwYWi=K)pP)RArOlXOvP`|jt zdTp}EH>kjYzS!2?!bm{l&bK6C&$VGasM4%ce^p)xLmeE5&N*a%Z+<%VSZD$FlHl0v zBIvJ_9Ch7PTl5?x#;}=O^VD|=m%>dr-tE>?NBSgBJ+W_s^|ZR zy26qmqy|GB#`1T&9Opd0H==@N(#ntNX!) zzvmdY;7JuD3ydKnwNtQ|MfE&!0~#+ieefjunw>q??b`b4|HuL?DP(5sK%w*Z6;_GM({Q+>2h}9E zTy)W>u5qg}h*|Ns&E6+75#iTtjOSE*#6Ps3Dj!^wTY^{%Gmw2roP8`{P@0SNk1Z1h z0fF*lFvq~%Q=0^@`}xmugg7CtrJfx(P374BpX zcZ_|#EEpgTToEs@c(-BTwmFF_6#d%HD%_t6jrxP?Aq@j@YiN9*e&A zKNhCAzm84@3r6|*2RrhCGFvtK6Kl^ZR96wphr|s+kV>fWlaS5wCk>3D@px7naCbug z|9w-dAi=gJ>(C%P)UbjVGat=YJfC2299Vjzp>P0_0YxWaf%_tiPuw`3Azv-`i<|4| zJh^SNlTU9E|SZQ*s29|^He#G!d(tUwVL*#!X#+)%bI48Ia z*i&qQG19x$OGB_$$K~EYqH0Om82#Pq$YV^`SCb*3ciW`_$tZQ%lrv^9jR$(f+GzwsI1`;$11#;P;1EC z`=4~Qo!Cc1qZj0Ju8io+id8!~i3_%pge>Ju5K4Wz;z#gN+fnPs&K2PnlF$a-C9p$l zWWVEPqsdaD1e4?4xaJ5Zu`tunq}L{(JNS}L_1wX?3$?RzE2wt(rbQ(v95y<=7Re_+ z?5fSG%R>p(P+!o!~*mHCj z)Z$J+w`fJ|0@+S3j*t-*OJxa=v}T^b9Krx%RPyd0TX^ZSG&f>y^2Z zN$dy5AP(BMyFkGEIhj=P9du0pb!4E!#@+LKoPUu8EkU zFfyUD^nJ?68cf4ShsDtr|!@VstqaTjqE z2t)23DcsmjbFi3Ec_0lSl&yKU@pcV#;&lQ0o%0arZnSsQG9gvQV0I*R);#hosk-{f z9P)J9(f;2M*046%*_LN7cDXdFitai$_fL)0y8aYw|KJO)YZoVp&7o_?%gN~$bd7)O zW7vm%KZBXgwHsHafi11DPvSo?JPbOzhu%-AVgYF91_f}-VH9g{=E!3J_lZ~{$3LZf zJ${e{e8F>Syj)NJmbzYQlNhEiGyJRZ#_AC;xg=rinY!R>_=3pi&&JnqV*(#9 zuQ9-8@acTPpq~(b-0sQyo-SSHnqeRyj)LxhYL( zGXT5)LgHN}+`p8*lydqy7>(>Y0OBW{&gJDynX4%s8UZ?DZ(tCqkOn_aA-Vn?0Wux}jyvufq#q;vT~_Ibcv2=GPC$Jte!S2?$0;?#ex5hVSEY>AS! z9)|Msi_zwO=P5ADKK-%8+Gm2ous!b9y(V+hOehZ(sIB%2==L`x7+&O-^MFH*k_(ST zriqM9?HPL$Ia(<3ark0Lvu>zSDLonId)afpX%FK*Bb<+)K6S(k$>NEmQvwVptecoZ zW9lLdLScC3Kr-%F4 zYZB)45g->ci@4pw0u`Y`Nph(t7yj)JLwzJ*>D_m7@7?fxqvz&h521g%c zjDEaD^RDBq4=FtMYyNnO-tH@*5;J)`u@Vgt@y#V>$+hFqnyY9HmpU)EA1~W;HCUk< zI*LjI00P{;8r@I%oz2s1AU)6ypDS61rY|%T#sED8QUqOb5*M-Tg_0Am7a?Q^6cxPc z0~wiyoB_fLg?hWHV=Pzj)@*Ys*ork3gd4H;pIl&~N|V&TPvA(1YX}fwnXdjEiG;Ud zw^*TZsrjc^H@RxH`a$w73dsVs>%(}SCkI{rc4ykvDSpt8e?2NQJKt~!rvc2J zS;Hq){f=DH#V;OEd3@jB&}WPwonx-fuZ{;A>?_o)W1eGPPDoG&J=DH~(U$B1EPT5fv)h4^@-Y&i6=q@|j`H?M=Fm)Zta;+w zAQ2e$!>G(bQfAIuK8Sq3j1RyF|CFJvLTf7#vU89;uLKSb4x$E8MTX5coa1yCL6Wg# zMKk(0fp{X_lRKs;DGBnQzkact3wHf_`~JP^>^^Yk!fpX_L0B7Uz!agXpx|Y0p0P=(OflaE))^7@E&4Uz}9AU7as~^xJl{q}A@gdIMa_cSycRtdGZYp?JxDA)(2( z`2NGJPGO?B)zvok-($XeVzh)1X5Ulr_V2YIaD2Z9>XGn0FxBI%EB@jW@wii%Q3|QS zcCs-na^H3pWTG1lY9y0|p5wJ$dA6(3YJOGbo)jc7e^Da94Yy9oHhk5h2pq}MVwD875E#o+Y^RCEC-CKQo7QHA+~U_U_HooOyY9 z#$D)4r{2lQ$<(rXhZH1`RDkKlveZ(3kt0l`ky+Bm((?%tzug)mP;m={@VwqB_KcR> zhR$Dj-{3J_>!Y{`uy9zbWbFUfhwy(kFbp`+GU?g>R6#D$Omfsdnri-CRTutbn^~N* zaWT?q8rmf(LlPEy9oa;+?LRd1t6h3WTy1S_A1z|r!1N=o%@5*_LS=+aXa{srcTk~S z78y^9GGfC zt5KXbQoKJ&Ui?)4*`|rKT+<8A^jn{4uOt?>&_HNAH^saK3MiBd?{9)|+9S2c+ zv?}Iwt)ok7W5c#6L*P5Z1L6id+ek)H&psTSjnTS%sf1UP-&}9xWQVJz_@Un6Yi%y9 z*pw5ZbjU2`T!Lg|$*yh{#O=qtc&YPM-)^WmQO|%Nhb;iQKfHMrpR0PSoO`dLK=vFQ z&}F=yD5>5z5qWl0hF{uaR5SFW{|9%Sv@rwHipGmdifR)?t;-HhMoTqDGADkDd$y!Z z{-`Lfln8Aq;llHv3DwsAiP3FR{5T?)2H#|6Q2$$p@P;>IWZqr=YU3j;Wyl`2_j zi;Jz%bf7X7x*S8f&+l~!mf?gwD_v&nW5)H=;8KabHNNV#p+_x&Xc zP@3rlM(`6r7>72r0!D!0NH++8BaTO9*AHrK$7xC*1XOs5vgWY5fVohSM6KhGqaxWf z=wTu1cYb1;-BVS4*cXF?A}3&U?HkK7Lp0(BlLdbWlBO?`-=! z&z_m@Bag%A!}RA`AuYdz5uvWgC^4r1eA11P8n`WJx1HH7zNdE)cw&a!4w zyzs`vXKZUj##c=yqtw_wyPEeqJP(x|qiN1^5+xYSTs^gPNT{&ILz_&*(zytbX)vjb zV7{x<;e#h9p3Z|}C*GGRn-rxp%`K?cN!`W6ru|<0k$q;4kf^(@EndL$ssvVm1|_n% zc0f09FHiW&2TG(=S{lApmoNi{QQve}Vubp=mcV2QBJVFxMIJ3KEtNv}prZ7;DW_zE zjbvH7MQy8UKkV@r1U6Hnf^{1p5HX7J=pFXgB!6$-#c{ zbz5fi3K5af)k=QHM$!0t%-OpuB6br!9iQLiScpUs!h+wxJ6Pd(XP2~9C`~eWT|&aQ zX&a$*@z-(0rScyC?;8epwv6sS&{7XDGWUw)s&)Rgmzimz<_DViuTc{d`eKoCiZFSt zqyy78r1}m{Fkx%iJN!t zX-_m5ZRU@Te#}ghH(JmsU!@yFYPY>ZYcC4(4PCOY5v9CfN38m zV0iIUjC+O|F-K6pm83W#qtE2oGfQ1{T{3JGc!H=Y-ya}zD1in|D|=PTlYWn| z6n7%X_i6ffA#mayaPhzuk#GGu9g}H947idyJP8zZZP^-IuvIe@4j)eO1@!#-bu80( zx^EJ4N`?JLZUO3Pn$k}J9E;rv2`T=}CKdL4iMO;2>4Ml912ZzZ-vXXt)Z8k%6N&t7 z68~PvziR6B3S* zk4w~pU;5w$1v68#=wLFfZ1hs=8dN>>cmZ#MmM^>gPu0%asDa@HZA_1RHD^-B5qnE5KtQ?5 zA}Jd2#HmlmM$;OSYeg$6B=pK@vf50rQMx@tAwp_FHu!?T^M32TdeB}U*@=#1a8UL6 z=HVDXfK$XW*OR91BMJQ($^EtIX%YX8%r~=Eln>ZAdv1TltTT4a8ut1 zQ%0lTOC6sQei|f>!Aum9)Mrh|CZ}_wVT4X zS(lKxM_=_-3EMDGmMoP-S{0(a+7wR+(h-A}GCNY4{tc4{;ywrDQSa7iv@2`eG;|6G zZaweD?QsF8jECw2Wx4Ymae|>Vw@L0EAz29WYyS&Zoao@yly^&;6M@|=C6IcDKa5W6 z*^o{|YG+GJBmel7tHs0=*%R0{(WCDvFD8{mvqA?@9+@8S>K@^RjLxp4F>FcPyD#mF z2{AS{dbQUCY**&87Zif#5n=%;mzq%E6ni}`R+c!K>mw|@lx zh?H^}qovy2SCC8eDUyxS;pPjHDjs3Dc`3_wVL&d7g+phpr~ZpLn>WNe0;)e9U(UTR z3kQ=iTz$Q|xHyFq6aTIFf(f54cJ1+19bJaO@|{ftI2u&eP6mR4E|m#3cFfzz*w{bj z>teVDl>!*G=PwRN#C6L|i06Jc@U7_R>*mWYg_m(wb8?RF+3LyDx8y&XuFe7Ub=e!T z&V)D(c_(_Esu0ZNCX-@Djla85+bT~zfQ(Sgn76AeRlxbFyBF;kh>l{Bx3$|oj5$~t zB-0rxYqJTZrG7tH@B7`!*lxr1<|WIw!f#HYb46S>TW&Lb%d;=Y@tGnkYNqLU)1$(6 zb!B$;K}vU0sLl`xt`@7T*HL6*qUn&O?$5sdU|hdHRv%P#Y-lWJ|3o#faIpa9A>{p3 zNiygTyV0`Fy`iSw05o8TPl%xeaTUL0{iX1ffq*KufSl~$lwEX->Oypu$+^f!_BH63b*49sL$MHN^dEWB2A*HeB zZ{d!O<6Lp`eBw=Ypzl)lF-ThyefvJH`J_>NMBKi|O zKd?tUVr7fQM8fSmvoj66`obqwSZsy&IW4J?e>c%~|FyoU)uH%hGjIQfZ_>|yB8YUV zLjSMbgMaiwMemk<{KwdPESk5LuI2LfO!ANspmK(j&Bk#()Vl53cuTAr-6Dl0-!a^w zPXYYdFOBVOJH!FTO<5%D(@uGV9o$!i6TduNRtvfxV2BqnC*1Nm0-p}pvp{T-)5z@T zxZmSCF8swxG4UXNFn^~smDciN=DVHCU+;c$3~2D$&C^X7QdN~&>8p}e!r08ed~3%z z1c2+#hLMK_aj!CVr~~+}i?GmF6y9%DO^?H!^TZEQzaYo6Q_;95V}=14(@oq?X{m23 zlU?gey{xdb2nH(+9X_J#q8=g=SXcH}$MxRn)vIn-SdZF(HI0&zl2)Ug0kNkckY5Ddel5a=LORC^A19>)>Wo$V zro(lqG5vw@Ld&I|M#4VCSE&ap%1@zSW)Od!=PK0ou_dNF}l1uK*ssW;~4*&!JXeWc+sr|bic@-aF2Mar`p9rtH&bcST}WiZbgU*R-1LE4Co{C+P% z#P*|irj@EtTs&3`j}QaKK~pF;`6WhuQTbXDT$}^eQ3_ixF@9>gwt&vlE1_pc zw}>)(&jjvI!ltcuz(vUZ2&OKX8eFr1%yj&lYqUk|yEktjhrA|CT*m9m#sp^TjE_^l zc#^W|j3i{g5+p_8$~))u%Gk-!QWCA!vduN8Sv4QOU%1D|fao6yYL$1sNR;MhX@<$) zA*}lkELL{wNs@7?kiPbVtN;ix1sxh6wNt=3Uz|KDqv2P;ES69t9h z`KjK8&L&!J)9BFu-b0W-I7F`vUaVQ0%Y1l0t{o~S?w4?-a`^FDwK|5-Oi=#uThy;n zXn)@RXL5Z+qcXjQ5$oDDYSrt*V5Q3c?n&U#`RKp|LEzQ7AYs1*yynnfI5B({vEQQC zJNk^bdwO0j%oMH`>Q@0_B=}R#=uj zZYRUk!G@G6apz^sGix{DPTC^Regz5-$VNS#EfT(&5oDG%$rMu z@{kDe>nD|@5dxQb^>7;7z$iX|+?}fHOMHc!)pfoDd>2KmC)*_P;#M#82iaJ8+E(P# z6q#=E%7%PORZ&v?gsn@)_cv5KQR((k?)LC9x>;5T$n!v1Roo^I=L$#s%45&7QrMOu zZ~C!mc%`tQVC$iBD09x)1D6ojKAk^fCG%>8n0|%k6L@u-2aL8^{w^^_EQ|kl+yf z`qgC_9gIgoL5w46ghXYybxXM3uUP78y<&xoY1+%DIgOuO@}P0!qzc0)O%#2asH>MO zAtv_b{Tn+^)BFw%_YM79SmQMnxQ7seJ%C2Hu|5&`IBeq6;COzH?(7`QOfu$>+CIE< z2x)99LW%r>TnusY81raw@V#hlez+i`fXMWBlVp`Tu3-vHxhN>X3n{OsviF44e*NODOV9th`rX7%IfvT z7o}BB#Sp&*nPO=(DzN=iqiAB)fhOgO{jX)4v9f7M^t5bXUnkwJ4$YyymBA8m$9w1k zYeYPb_iP7FCEeCIuhFee|Cife9Zy0{oOZ!3rp_e7 zJ%CFDiip@s)vYt%R~M7!=6_?bX)b=z)nh(`mMhrGU^seuxm`RA2()_d<`Lk2mx45Q ze`)AdQRR<6MkPAl_5S6(iE#JFcnxnbwD}3ipsLY)16r}C;8`6U)cxjN-F2^2K&q#}ezV4Lwi${JCwKQzoPe-5Z#0^JF!TZrK;3(>OMcd6?R~lQ z^iM}IkkO`s`N7!g(JtjwxjN2T4oz0A5AQ16U5<{9SzLIZJ9xORUvFK}Iw{zu{8yt# zDz5}BZn&cpW>IZM1U~+zFzo}L>=1LK_O7mntc-?NCchfocH0=3T$yozZzc2JHo4}@ zA@1G)kx=JV-V8>vp}ZkOnF1kj0vw%P?PeDqzW&;)$opgvc(BygIPh?fe_`{P(iCC$ z?@cIswbSxM0~%bGO~78dV@$8pd9fp%Zq2+qetII$Q`0#Elv(ocXyJwrWZ$^hzfk(P zpzXhoOX8fW$A%x|YXxt0nlQybzJD)ZJx35ie>JGJgYo8zjruxHgCCIw{1MR31)VraEoLfL}K+9bJo`+5FY4Iuh^5 zE@V_Ek(d29o79FKq4P1y+56;^3iAbtoOAzX_+9jWb+$=PF51=V`n@7KEQ(%*JT8-f zM?xY8Z_gTN|31;iEqqt);>?WTnZ{;ZUKBde=x9@5`|uyyRpy@O4BAO=e_W?iktr>g z(;ugg`|;m{c`m!Gt<=^%dR^FXFnb_6@3!=TVo@z;LTfWI?{Bn1KjZ{^S1*qJJmbjC zX5yso5&Vy0{nHrq_plMJQ8<+92$yWK52wE%=DsGA>~*fi^S7$XdwU7F`Fv=jeY+u7 zX)o-uqcW8M@xM;CaUvNI$^xs!| zmgI0awE1DCsk0BYOFKCJnTr?Cqo(9>UsrN?TYG-qrusy^?;U~dfmcB|P;LSkl_bY! zzKvR_F+0yozL?SI0t)w;umR&Q|zn z2yG9}bC`||guMS|Z*U<~N)LO;veI#}z_X&D&c%R4ZW%W0-wwsv#VbY@k~ zZjOdlLV3g+6@n(rph5t{GvRpc?uI_42uHC}JFl zIh$jiG>0)=$H%gx5H;JD8^dWhJU;&BW+i{x2OEG%{LDT&T zzu?uG%tQ=jRaJWA?{}1`K>-f=7!4NP8xfZ$nyz=Jh=|-h&koVX@LP;nAA$EC_VzYw z9<4>4h{wB;EW{IeT>Q;!MQR5Ik(ewMT9wvK3vU({7Rd7*&Ah#L#l3}&!FH z>&5WrJ!Rffvm7HeJV^StKgS~)=r616tA1G=jrS|CZl%CDgXu!hE5OWVkV{?O!#XC+ z)2|f`;G`-NfnLqe#w!j{gc$W|DTZGRoDki0(LULPoH!X_{4(f_2BQY~{<_g)CrV7a#`HbodFnpjzk$3H6_fJA_zm>IJ zQ2!foSLIC5na7#B`T*>gG?HP4&l;F}JIV&F7-cot5q7k-zc2ey*PHQ5GW%oPgO~WS z;3_^;8jPRk9@&lI#ZJj3p`gLF0t3(yIj@hVeLuIlz7Sc+j+9zd+*2D(9~&>X0(m+% zeO7($AS5U3SnAfbA>sUmTm|ft-XL*IRQr#T-ShPSIiV$;f*XZ|Q^HLw6PG_ z6+4#1 zRmPmo8u!hs7lQ(hl*a;|kWZ1W?aWTlMwxAfJ}vpg7qi7%|sFO>>npF-Q8dwF4C8VW9J1O-9jv zv{4CB{!pXR5bBIN*Hs$ZbT;F&df!^h(p1iI()Dq}`a~A2z$J6r$2hk1pKWzIHNx=@ zBs7L`5u3N24bJ)A{&z+4)U?l-R=XeB*%Wv^hdWXXqp)JaS!Jx&inUSk0<*GaWhT{&RIpT{|ODM-M^v3twoJ77zH{C7`gY5AtoA`;PG;|u^ z&@K&k4$6BPCH3{Uyn=cw-|7>uf&Sx1@)!*s8Z7zPz4NyqFO}XktP{8D=R$zczOLuF zPxFoyfg-mp7q#n#zRUgl9WMuPag*+R`=uer{?QgDSWfYP^SI&=ICUC->*07odW=DNj05D7{%Hf}zg)wne2uC7;qv5iyGBz{rgAFtfgbDWzd&su|QyKHcCFOI;O{X_e`mk7$fkWGmuy)urhD0VobK@2j7!w>+BSI z&N^tNxzGo@X~rcU!a1Gk{`%~zTX0Y_4M4xX$iy>MAc{!|&%?_c^Fns+>Tscmz4^vS z+U2C03#h0Z)#*RbQVuPtDuk0mzOg7mmhxG}C}lz4Wm=awfLS~jrHw`PSPVCjYAIgf z>SJ6FfB(Bt9M{*>b9EW8)e?@}Nh0FEzbW&EL4A!4mtAv{Y3dXwofkJ3h`;Es}3#IcQKyp z%#=v(|sBwP#F-&xD%tGd~V4V?h_??EOeCghTKYnEeL7?QlasvRC2*gcbjY--u?r`wL%1x95tqjZWBE8qQ8{XgGlI%82{I6Y;^oPF zek$c#@v^bSx=Ntp;+pWXZj=zAw)1y0FOJ=E1%hB+D{1_I_kAH)ijG6r3C7(-&Y`-##u_@yJRoA=Nb0MrzB!51_(JNHX&o;sM zn?43pN5OqsHT@@jcTlIt)b-IxQCr57-ySMq9?Z?L*Ar`L;uJ)H>Tx#j43_PPFprkoT+34;4Gh2WavpQrbdO>c>E42a ztBb84*e`6;T~YhWn~C8gtq~(S?FEo4f=A|QXtz3DJt@N=utxr7IXO_J?c8g17vfu{ zZk1Ejy$>(~kc9Xq8T(3d`HU6&kovT2U;t(W1ep)=G3g zn^Jr1;eiLB=dI4`>srXOYzIr?VmYRe+&gUXA*35qg%uywanbYrqH0~e2_|`nn!_ER z357xH!eJdUs#K{+alj@VWV~JIOj=(aePD>6%$^Qg>CejGjUSDqxZL{4g_W?#$aMMT zFIeIcp$!+C-$lQxhrv(O7k!OXuSV}g`;oLBc-yI{GRjGi61AN1WyP@1EY}rMynwoZ z`mZcd=)^F>i!KXej3akn>X9~ImvHb@d{SB$oP!@;X;>?Ys7qp4pC4?$=;N`lV`Y3- z_%laU2gY5mkDHL!_W%)Yad&Rfm;dIE~g}KI^FqVoqfrfFGl+)b78VK@&pJWd z8Hy*2?te9m?SKLX^i0N;l(em)51_2|W5 zFy^&3nZZ4M>rR~q{+rMU(ACwo4$6dHS94R-m_$rR#|;HlU6%EKiSYN*u9lXNF!AV^ zm>h+V58vgL>ylo>^T6}?FK%R%lc1#b=Q1mcWd9$kzB;bzW$Swr(%sz+f*=x-BHf5| zDcvdEiXb2*CEX<@-LNSsX^?J^hArK^gXf<6-1pye&IdTZnOU>ex2DO-zvzeH_is4R z#3q6Oi+F}&gAESN4FK!^Up-KO_aI&9b1wPt@J|=Zzyyi~20Atf?9a9!xjTh+-7^x= zcSotBa822i6htphsYz4vqs;!!kiWIDVP5Yn0?H+f1NZ?Km+;{eTq``B-{6llUnGo= z2p{-%O(HH{D_XDRKTh{pMOBY!^gf+pHqol~zaLOMJ*@MH7fpDmOMA&Cq{|8t?%?+_ zAbETMQ?=kHl4W+BcGSnpC7(%L?2w+8jN^ z2+eEoH~DF_)z{ddyN)u`%D<%i=UevuMS4zAx*{Qa&^+5kI07?@r~@Jt0^* z)8G{bNu&9ia>ZK3Q&idUG83)Y+ZH5Dt8&s~qP|ONVWP>V`(-h5A;G4Gs1zP{Q4twd z==;3lim8d2yNlE>Nn-w^{k4<;sXAaR2yJw<>g{!UtUsSDP$Rke>6lfXY3__FKZn_T z9{M}XRd3FHJF@GgOeD4_K^6lO^>fd%L+D$e(lwZ^KBpA&95_(*B?}LHspmSjA)FeM z;$5(G&vU-@^v!wun*4hn0+K`%D2|c-C*=jNvDS`Jxx|<^Z{Oh9h2i3u34|@|{vG~m z*YqGa#Pq{5>Cdx!-n>2iF%YN`yA%iz7!3ssSUehx#tjV>V;18qXIR4U7D=r#LI{gR zE%(s><|AIN`7MH5sYY=E#dUvi`CFL=$Y7cQ?%k!t!dlSUxrb-SZ~MQllL*?#011=Azf8$oo#_bPox0 z3HXcQ)*CXRuoL)Yi=TQ7@J2pG2R#{nocT@QTkDw8$;S8JtgEZE92+83tho$}`15hU zWUwD^t@miOS@d(X3(P0QoqAIfw(MQD4I4mcmh z&ZbQ={O&f;xmp=sec6L=j>#cuHvC&9g<4MsdS8I<@`QwI!9}p~Uufti2hL;bI>S%I zj}JF6i#gl>;*uZDcCxW_+Te{3y^D+B+{eHd1b2oXf}12F)s}xPfA698rwA->`=bdq z1{y;EGvD21oftTr;cw@9h8Toze?CR&Sqj*&De|T$k-weG)EVK)BgjSbJlP>mNsfTl z=$$nm?br|qSpD$Hd-G<3@;Zu+;j(|w%7y3^O|;OWbLs1lgtP?T-Oebnyb_<_h%kjF z>(NDY%h`@3_-&=YSA+u3w|O*%j_*AK%cR~04<1^uw0(bru&i_zX?*kVV%4I-7#^*~ zYs(P*i<3PpmA{JEU)VxKFlHE4Zv@*g^$K<_(pAi*(Jud@7pQCbk(6at|h|qK2`U{>0c0O?vEY9vK;){ zBntBX{*mRXOGy{+&>Ti!FL42z+P(tbXU~U$Gx5lgg~Jewc#8$%H=FwLXoUk$^KP#C4Nml>u$uSq}Oyx83?p8j?;+dMfYs=q3*Ud*u;bCBU zj(OeYewa?y^Pnco{rYwMq9QCR{1rVL09ErK7sq``1Q+1ku%f4<4(pJ}hCYHK+s!gk zW8LkAm$S&XIg)vT8W=Z5Q*?Q~kT|w_OujdjELUSJP)IVrnk9!TG$0vv=Kfg@G zn+L(cSSWthMMa|rw^T%-BsVk>YkAIfPk!Z%te{W(m&p_78k9OXSk-N!8o^la3UZW` z8q>azx9TUSSFVQ>5fWYR&xlIVbxUrE@ci<@ar3>grQ1K9-E|0SIC(s)@H--dV`M;`8=!I&edyF{G4yei4s!9ro> z0Lw}c)qIwSU32~qjck%-1!Neg9wMMRG1>Xnx3`*1*}Xw zl5cxcz6MSH0+?YU7*!vessk%ctBm!Qm?_yR8GW>OBU$PSw ze3;d1^g=k(t*~d`N`rsV5%<$0Fb0}idSLEOTNeW7vG(6D==$iQtBYhFKO<5&`$7Mg z(8;p)ip;MFD`a1do=pp^GJ;2V?P{{$a-AG`fr!~J_?Vwa&`u%jp!>~_#Nif>MrHfY z$By7psCEqI-Nz?O`I=0u>TGIxCL2gNtkH#iLg}CuRQTYl%7=!74c|w_eQ*W3N}x+o-R+`w{V_$fh$tP>r|oM!yuPcBha@;`L3-XgQ$TuS$I6^iwFZ$ z@3R5d-0 zv>kk)T@%dbj8E@J`ds`S=9=F2_%(g#!z9aXF}YVG;5d=27S*i(86xV*NqQ1~Lv3ze zuZEv5a(A)DWET+7EFr`fT4ChzCl6rgCqfA`8_xfHB)_1t9}uqRE92wT8+)pN_sN?o z)w`#_n)^Uq2nvCN$ z&XBTqCOkFU=>HCr&t@jcm$SxWWSkOx)*WG(ex!hS5Dd=FQF%VWK#2j{lUdGGYC>7w z{Z+hFoPk|RiwtT^rr~WO>o;(9Us3)Y@XPwj&LH=PZTG>{3nAniprbA`9DJSgK>Fdi zUZ1rU&|%@X7`3LBNsG||P@Otr&i#x=bqYKXueiR}Gy)2qOSzFk z9YY3afo|@Cs24UZ9ZsJ73$18jPT>SFFZ2$EihAEqs)L=7*2{|>Nac>b#%pYG0*LEApC@HWJ)=SOPmZ>Wpo|NdO@0OCfMO=Z8k zK>!{ZG$lOrmK(dNse9n1DS&R4UyZMMfFA!p#0*(k-(Rh13H<;4KI4CceJlU-UztA$ zw@S!3kuHwhjekVA>V2TJcwGH8jw8;Z=Pu`}@Qcs#men3!N^)t`H&wB3YnrL_Lj}v9 zBQumHWGd?Ruomjq0z~JX^s_K|c=#}rLNY?9WM*KX@4t?c-07rT0=@Q~_ne_`eK%Yy z6QNhu%-+$nzr^XJqlR9?aanRc-#eP+a~(WY7%Z#J#7s6YEKSU_oqQZbWEwx0;TJrE0(#k5%fXx#B$OHxpr}IINaURIy=A964IUAKx}+3ZIJU`>3$tU%B2{VC?phB8P{>N#0zvA3xqvjhb<3h^})1 zISN$Xtymo?&6Ps*#IxYEO0{Ay^0o-kl&=+bhbv#M68ezX88}vNT03L-`a^vp)8s}% z*lTsk(UO+MHQ7@YwYuEY8Q6gXg6Aw=%Se5T5^J`ZT;_f&cKAdPx-8yhkBK!d(vQni9Fd-U^Bm3Y@QojEi)UJ6(y>6$ z9q*Dg`dRJiMR`_~aTWFAKl4wgMh+a+*1@F12$)zL0MbLoZf~c!c(P6Y2M0 zzU>RleMgsiSY8i#qQ9;$WLj7YZ5S1 zb3)u}!Jlg0ia>KZaF52z{g;i^P!2}hE)Qh-DW>wQTr8H0zeMe$(dt5e+D=#8-v(2w z^61~r+zrI}Fe#=WFcly=v>a2py~c~Sn?RzpIqBN=Z4Q|z)ygnd6YO8NyUMq0;(vv7 zv8G0B=)d+@q$PG#IeKLXVMSg5hEQ|fE|an4j+@SFB07{D7E~D+PG>SMX)AB;7bS{d z2zo_E%ZrtsyMbRUf{1pDkJ*rww(7*l2N-DsWP0nC?q4^#+2wlPiE+I<-7AF=V$F`_ zVJb%-U9MMgX#(7gqL2YBdaZ)xdAK;=s}9EEU*Y|dYx))T#qcoAl~_N;uEZ%e=$!pN z5kfo-`4YU^kk_fv!RAx_$h@=W&5^!zU*s@_sGt1%7%{Qkxu#h=g%XZtPOngj%gNz9 z=8WXXm9Um*(Bhss&l6NQZ0K8l7a@D1TO)Vv+HoH_*o4)OI=vUd|DTcv7}cJZWVxms z(0-OSRIwlU#v4EcY0&0nYWGJT)pd6{UvS{13MYQnMr@tR%#tbht7rAO`O_0<&4jw_ zRQ_2=%Tegy#_OE=GlIzFgnEYmX~g8vkE?s%NBWPmqs10j{ciWM=>`F8g3`WX>9Ts_ zIW3tETCdduqMkea~8&Q;R0^?q9`_Uwsl@Y`z4*k1&owV$o9 z!G4}l-{plK(Bp|vH?mVQCM4{3q-Z)>q^H8@vpf=m!nGX}BqhZ9P5KQ68RQLliU7Kr z?dXsb?MNfl)Vr;9!59R#@my4&NSAg-s09Hzixg@!b8pF*Z5`nJWs%d22?Us_gfH!+fr~NtpiRCzFwkQS4KBkdBj8=U1WCY?+ zS2I_NI8*$~fuFF(lbGNA=IX7>ulIR$a5WaZ%;fu|Vn*)>qH9yUnXB#AP3(X5b3$`A z=I-QYEuG(Z?;Xt$8_~k~Np0KIzv;7{tr+9vZEZujJdKiWxw>4QH}c!inD|MbYLH>@ zc<}Gd{!ITF@iO!9$y2Mkj_jvwY%6pN8Fn+2^0bK2#@ji$1D+18x10yf!;A)5(La>* zXadjQ7{%CWcX`q&!od`@9H9C0F_U0a4f^CQGxP4g?t0m2Hn+*en0dZmeVL<;aa#ZK zoX=;G+zqWROI#qcwYRHyiG~5%T3rOWiRsn0g{SR8(GX{qa1>*6&YaVkhYRm-N6wb_ z{NY^u;RmY1FWY(W@5VpNS8+3FXd`*d`=nHRn{ne2o?=hG)FgM?BW^0ut|RP&pjKPu zf-NFWGya={z{-sZ#UJ$O8K7ph(*h8O@vFYD5EPjKQufJ!K(7?@IZ^#-ccc_uxv# zq8OPD1>fPx`0to(tZv9l-CRQ@mtQfBM3wYTZnH5Q4{u@^C z@Z+6u%`w01oA3zM6klA^p|zQ3T9O(UiR8SUwj8GKQR)5-ItbiScFz5ssinI&V_$Vd zQ@=bmU1PVZ>(h}h?38aBq4A;ENICQ)Kz+}*&qH{R+-@$`{GXe;_&2=pfxHqPTWrxd z;dxFqDW0GC!ml?;__JS$|y6A$_gGkpJO&<B+{X2O1Zatkyl=wcb4h$>brR+V&@=iFnWmyTci4xr;}6hvtVlrJZG83IItSy#+dKnHpBIX=Rchd| z?Y1$2{=l4q_!e+0aQ}NNs0I`eGd&z|#+!Gkk5;JtuNo-oP3}kGUi$c7SN(}X!&yf6yN zr2Lpw>Gr;r&VI}P>J&ScZzwkSGuoOV{&_zyff1ZXW9-35w>rg%qEDk7xjvgjT-Kf` znE*|cWldN~iaTXjmecs*??tQchuZUZ2@`|G?z^>=j-MVWXo}ru5F38eJA@UH2)d2H zMw#|GjRa@F#VIZDb;-B{9OsjqPXI`qV~F4@5PeR7zzWyre4eK9u(Syh9TU!tr&}hY z!FM3Un7msZO%eO66J)Y~;Hdz9Kd|0|caD~rX=KS(H{Hf5KKZ`jwsU<P1<= zB>=+!hxM#gBnvZyZ~xP;f6IcX^j^&*8tvvd z7NASy533P#j?#EaA;Ko&OWU@Esa)G%``|SJ{Lsl=q0>@nR@e(LBTlc zJ(LxsVQ61fe%3p|wBzI&1Ep7yyT?MG=MjR&;i4FLoy>4vzC=ygK)u;p9Qsm9S1i>O zVSlhEAA6Qm-B%b_R{BM;CyzMmw5mP}D}3l@*p||3dP0e68Wl&W#nwZV4mD?7_}4Vj zbiSv9=Y{@eTyKkO-V%EhvL@tWi`~8Ur%@Qi@d}p~2%v+Ix4csOVg-2$=LtIJH~w-_ zNWHHkxLI|sSXLU)6T`Q=f)JHPyU{RabC=YnIicZNiKYEH7Dm45VRpR#JvTcDQ`Db+ zTnl3WFUWqJi(GuBhgL5)cStJ78bl?Eru445_4eSp7H;uCYs@XSDo!;d-Yx5~ZjUIZ zzS;@omv{aGd`Tb~7ud~>h$og~nLHaYd)p_QL zW;0Z_L!sDZ7JZkwB4^I*$qhJii?GfpV8*OT6D~?$nA54$FeOC#pp?En1rY3Hecy-a zm^W-&4c@xwZs7ir0}JKA1VT9miKcC?#Ma3vSAN;zcZ8KR5ii!2Wy8!zEh1{=cLShl z+&!+2Qzdr`EXd8boCAh}LeAtt1u@j-Clz#<&OZ`6atBumRG4h*$fg{Yw2Cz!Ll@sO z7Dve#d?!$?XESR(t5ka7&|#UF8%YBhQObysx8kty&vXp1Z(g!J37|zArH#ZRKRP4IkRB zZtK?Y^{-yny55=vsuyM`ta;#Tg=g-CEGjC3T$j?H#-h}q2vIpq-f_^_Zqi#s;Hz-T2x8rv3UW2!R~q6_7FP7+4}R}LuKVHclol4<2<`lVzH$V zA4l!Cn)Ln&S?y9M)P@b+K18`XsW5ir3I=_RutX1B=*tD zmS;jHtJfcGKlPUmO%N3>sy47w88Njg-Zm+X%A)T3;eL*b&aarLG8!iuV%p|sW4QTT zv~^0law~&omX;w$(|FXK!MY{ijmfEix2w`KuTH#aIczLjH!b1!E-!o9N|Q=~$C z@zj=`g|6iACt~LZeCx??A1P0_A4@4JY6H7t5i5o zyNlf@((<12>sHPmZR39a>iUxtY~K94Ge9dUukHPggzESyz)00B)dqgqzCNm$BZ!C_ z=_PKrM+Kt`tfu?)OMx0>l3sP9K#CpPdabV@{YxTMCj6(LgaLJPv-_Iw_W+Zp)cqn6 z=}z33PKAEov+JuIlihoFPok?s|EqYm7mI3NE2ZGc|1M`XEm($k*a`B@BfK_0-Yiwn z@vW?$T4eZ^fl?9g{|f|$5RPi3_@^Ml7H(ZGubwjixsirFu+Q=uMjV-$d9rj` zg%dpom%H$a9{>BFr>;9-?!5&gG)e%n-RdH4$@``i zXr|JwPw-o!)zOD)uK>N|;!;;4?WvT#Amf&Wg{~@4SN%_rwWfShIniSIIhAIsb zeZ;N8V9|^Wx0adAl7G~h1F!ZNf|tsKgHkZB>b>xq>fF9gq7Vr%?}!JlL_AJ*WTw;( z5^|Zj06w+Jfsy)rE~9ZNrN;U(v3Bj`>=Srl)}+g7OA4~hE-RNnmSwk8rr8wX&01R# zs9yC>cGXA~>F1-byl-Vl2NZbzjD>Se?mvCQq8x%GlgSjCZ@KEEntfmS?CB<#^yMGB ztJPhHhaqA@th7a@9CJ-_1S$H1g@c2xmcAkKBAC3|M{I@xld-B@NZnBjxX#o?Gs~qJ zgcM7tyeN^k(x`D}kaR1Gn*xeTN)-;Rr2}Qt88mRq6~oJsUd&YT=~FA$Na=9X3xHUZ z=N1|vFWTGG>7jPiS0etjEDP7V!{mK*gI|*k1vseg=Nb#gYrVtfkZAWMhkz;)fp_Zw z#JOGR$yinR=1^_gPsCXLC!Tj~okQwFZtK-=Zy)LyY!QOuRP=~}pw%q+CNMQWf~ZEzc)%7a!~&6?(2CmMEiQGT!2 zAWY|Wz;)t~RABrdC{+UgAY5s?o%xE#j@P0mZEvwsy{EVT+%4~lo3KR(l9RnPse@8C zrKBeiez~sj2u4(@g>#A&yhKJJO$$27IOc>-CE;)A7#y1>O%jMf0Uq6P2K86Q+qOI? z$n~ck;aqC(8&7u-*jC5N`mIbsn@A5e8U2bpsSp)SGa?0)E?umRI)tUn zL%)j|hd*pFA8-TeEUOM`?B`zmY!7aM#ult`5ZE9rgAR{w#tropbIf=@{7qAksZWEX z8NB7_cdI)Ivp5{pHUQ+YCdpmnMiZBgAT)Oy+SHL0BCb4A(0ZV~;*RnfPhkc^gt79p z?^==Oo(?T^4^k>k&bnOXf|$MeA8y< zjjd4t_St#Tm zJxQpe8UqIG9EYB++6NM$v{PiOAf5w5gr0?3xi5`JCoz; z5iHm>?hd=F^Aeb5g25aKy!uz8IX;spM8&CqPWumM_6+bCNP(_!uRkDT0tG4HMOD9B zoO5;o{j?mg?*N;CO6n73%=p)w5TG($d?AVef`0OYoZ=#=1OlrC%0FUiY71A%01kRcR0;SbcBb(u-XcuDef;i}fGKKYb2CGPJa64| z_t)d~*mw=&;WNPLjg5_ciH(K5h+xliv`i&%u8JWn8`>BkwYEf6wR$G2 zP9@-X@w!+`go4NF@)YBu)6$+<$2lCEn|H0A`1&++F)*mcMVO#IL=-sy=Lsv3Yo+hp zFJzr6ZEV{AV>aI+UGRK@P*MLX#e|nX1(7pI1|m?0g~ms;&{{QmnV;#;Iu}|ZbY42` z#H5Qk8~|@v9EAE!G5s+R+^SOhS117g0Ep!Tn)esU-Qh&?*9*kU6n!Y6K)jld0Mb8I zIe0*1j#UrVzO5-Cz6+IXC=_ndH^v;iu)y*JTAy2HwH|uBzdwnei?$~@G_jeQ;!XJ| z)A-_YFa^X_Ws;=;C3ANxSOuI#@AJbZsDod|VvBh9gA`o>8wT9DQlCO5Va*~KQ zQ(9c27Is5Hi+`JFp#^~zFh~QZNOUI1y@B4rr}WST$i%+<7)2`LtOk{M%^|`h0P?1K zQ@h!!Kv|d^0k`{ZEz%*TP+pX=m_$ZomF+C4^Nnq`=wAB3csj=tN}@G{|YdJ_ZZq}mgCWa7nOhq{)5N%20V@f zNUVrq0{$&ogysV1E4UjFc2V&Vj^oh*uM5o1#DT^dd>|EK`seU)fszk26q{lsgAZMv z8kOb&WtNu6O^cQMf8eG&0mJ$c;pQk~~m zD)||?J791apIFBFxwo8>_Wvqy7aumLr|Q7msxA?)+qZv{O`RIQF7-a_kugA%_w$PC_V%_8?Cwd@eW`)A<(V z#OO%|Uz7Jcsf_{rwhS?UVAiRgu11>-+cJflbfz&db3f;+4t*K00_JcgyuBIq=}Qnk za{H7G)I~&CJ%#f!5L^smx3Iy5OBn~azXK!rA0S36*KISP?hbCb!j!lT=vs%$LoPzc ze<$+}eyx+GMqk^C6CWKYZ*KIQdLjQ55JEv1q=L$AO^b|XvSVN^0EvExDiU~fb98M4 z04(lU^RIgb^iEbx!zbYL!%fE_4QfyqyI&+-Xc_wM-%UOqlgaRLdp?w%JOAC}{y{e= zXElcPj_$dfveml&v~BH&6yQ+#0^}!-ww@T71d-w4k4NB|qoVlvPOBn4-43leyVkBL zYt0ZmZYTbnNc%=ghE|qHI0OckfDooG3o=x$rU<B3~6=giX=?xY#q)wd=LkTGR`#N z8hqSPp~jG?^iSUQYb|ci1_6&OeM|A^muz4>1coVgbJ4Jw?hiDS58_JSl01~1A1Y}r z_BKxh-(2bfO#w;?5MOnDCQsmjYfjCw_4Pu2SA*;UsTEDQ0jfbYvJa4ZAghp_pR77h zf62Ty?gGdietw&eI2KcG&Xqt=YZ+8yJ-Gu~rK|mGWA!Q^Ey)z|0R*C~ex0H3<;O?# zn;qs=l=I;7j)sSgfTyX&)idisUf!eSKy9+wjlS)yl}t8kl_Jh7`a_Do*)SC#uQ6Ke1Oow-aiMA zlP$2`7pN|POjn@|T@VOhgaC9kE))MNM5X%y52G*sNBA&C{DTkr&_q)Z;ikRQp*EXh zq*R->PwE2=U#fjzpsiQDh_0a7R_$oJy#o<+D4QV1O0<*bdO8VlZW2DnYUQe*wol7`HQZ#aqy0~U zO<^ohh|1Sv25^<1sj?+!dducg_N`UneH?*t>Hx=7yK~y z3z|7d@Cg3T4*wR|+x%8u00VP0D&A|omtB8g6HueSM6Z2_62(2gdZ^Q{Q_j*hT9mtU z>2PFOk%x4@pG5F4<(H`90qq5$>W%l@A`rAE<7NI#Z)qv=<^!BCT3ii~`^^GZ0&>9_ zAW0{gOmPb2m7Qrv=< zCl(zp0#Q=;+df;4=VBsn0nNiuP^-|JYihqcOd&eVvxQ0_0H}V1oLSAfY_K>A)*w|+Z z{zv|L_4V8M@3vM$3rFz_*0}0*GsL;!?zNVD%A>yA-m0__QMuo|_9bXO7QcAF-pBGH zP`I5r1NNzWk(R?12_L}t_yUA?S$|#zabZ1++Ppu8_Y05fqZuKn;4-7z-F5IEp^h6g zC+B}~HWDr$A06SyDw-vDEfaIG`5;pjY6|~}0EYcW*L@BN3W{am4G$g$5jYxS)1Xjn zQ(kbU2~BH9-F`odY@|Bs`AlV%@ZfgqXt)I-_A~`EASMtV7XY`wx`&XV=94C-1D@># z16Ljc$unofv#>}}g_MVF33mhDBYwE#BuANAIx|>B`mW{m}-YPniTz;#cp zAAc06N^oGQM21?uJ54Bl1b`r8eAez(aQ4UN3f_nX-Je;kJ@^aGTS2f^==dh>IzS|? zS10i5x)BL8e9k_8{k*eWQF!ltWkpeD7s3f$&hWp+M~xnP_3mCNb24)!ZD0ab3btV+y)dOVwM+{-=ybzRFb)=SAZ;<#`I5r+eT;U zO#OH+ZfpC`eIgZ-Zt;&eRsm?R%P46j)-cH3!a4b}cmR93-v>v5Z235u_3w6vZ(oza zw_UGTPz(Uo$CwxZh~@oTjrGl{kvqNv<(TU0y}Uy^&d@scMr5kfU{lNKYE%-WVX;5! zvBt`G8oPBV)$NOxbAZ;&l9h&$>s>S%l8RqL*AGH*310*^oYu7 z`&?&qWiQFa*bp31g{fV=~zBuW!6DfryK~Zn*6# zm~Vi1xpTN{ZC4eH^|F@PscqFR%SnMK)j5X&<_KiZqxN48!3pvBZx01SuHYyd$FQUvB{z&HC}-=1;) z>dEknM6?W=&Qoybx_vPN%%czPOAwCP7#%&LueUnK0VfPUIBWeDkUwDL;3aN3QgU{8 z7|-@pzm@gQa)!I06so0l18GU}^$R7c=DEOt|L$P^!_0#HLgQqO0<(zYmy+M8N~uq< zno9J@8IYCJdE50*Olif$mj1~--k2H<-5JCK*-~9!YihP6^HvZZtA?`N!t;bcbTsH7qylB@*ouM!_V$B3od5j~-{C3phX8p$&fhsknK>AMWD_ zyesI+t3$)`NDo-?+al9i zdxS(zRFGo;@WrDo$I#lhzW|aI#&F&M;C8PY$p?U}y3;qww*krZ9}1K}^x`m+`NTI3 z^mPu2!X`sb7r9zrNqRG;0OSHz%GoyxK_6Dv#b3u16?Trx{;mx#sEeRI83Cei$>kd< z_Kj6IZ|Lh@&(6qOx7f<_L~^B=b_d^%yySF)e<>94;^0kUD?&(Ubfkgz*Xm83 zRXv37yo85zNTZYfmu=L3H$^1wXoJSb9}?51$NAL~`c@Bx#T z@1N5;6mFhoFcZx2fpYj74@yi>s%OsGiY444zBrplsA|`JBu*xbeG;#Qf{nqk_j!L} zRD=df4KfMFr3i}jAprj=eq-WzG0mOhNRY|#k-TrGH>PvCqDEmL;3Im#I2hPYwthWA z{rxjZ!>8@KGT9=~jjnq35(JuJQDMwfnvDY|@$}wIRZ}63*9c@7yc@WY`*t6sX7MRN zu)#wpKT5cLmMild-J?1LSD0i6W&^XWMg>4R8HFXdy#ews1yYRwvceJp;EjGCkRt%z zu2wt0p03D|1kW=wgwZOmr)sHKzA*=CpW%Q(z$f;=A4rKqX^mqS|RwnX(I=SThQYci${EECPCNA7sOEjx@*}@#)s{mUK9UZ$=Gg5r=_l z(TFsEZrGw;u5pSV9=1$m3$MG*`^G_h0JtsWQg}1H9yBlQ?IvmgB6AR6+zf_-0wewI z^((D)Ann*N)wH|*B;BtFaj2okeS%bvcen*Fm<|H*xd&5Xe%cYMIFJKDziuB(yxi_W z-eA1HL#O#Q=R!P&*!5~LsQUUz+%E*4?Peqzd7g@KHM2B|fVwsHO=B0d9-4;3MEEJJ z9S7k4R!s(NpW%YniP|e*<)SEn?c5s{;NAgFTsBf_Tmvbg?eeTimaAW7iSn}B@7~Nt zvAgF*CY9slB4BlJCGfL`jjCS_U4o_&F`MB*)YVysOb)?ra^DHo_I!lw<3B3eor=4G z1e?u!@fHRoJ=mEg7slC|MgRdN%TZ$k7<$e~cM29$X;q8GQsAn4C`wzE43`;KHK*D8 zmlfd)F73ob+b>ll!*O^W)c-IAp+_K=of;@ax!cSpIX5$XQQdRXLq z6Q0OTtE<^DhnW21G0$j@?+5vf6ud34+}nuYsQC3nuwhKBR%yNL@dq_0%cqX-)cyTd z%TeP)U%Q*td1RG7YiDHVXPkHf-IdBUUHvuChaoFD>e4}6KFxBUcyS^zhlPNQ9h+&K zCQrr^98sw+@0&7fGgfL0+Kfgq}qNiOA@6m&tH? zlQE))3$D=PP<$Skd(_*~ z3j6chY>ZhGl*NEc z67t(Iv0+F4Nv$_a^LmC++m_=88+)(~Dtv3acS)zjHi| zdOLrYIehazY!b_2bBUGW8u&taSETUUhz}u2qb2!v>F)48x#CFres|$qTV<3zm_i^q z2m4fK2!Y*jdtRjdlr)QFhb#=@&%Uv~o|~U<*y=MwrB)z|6Az(Sv-FQ*&@ITPa#}PV ze5Ixq6Cdw-;}M+xPcH#+8)E+VMq|G_RV0$eeaC9v<=|VB zEHkoe`hH~lDQcMXdhK^*$7n9lG>+zx~_& z#i_&HMx@Y*Oe`(EG_yeD?og{BoZrf7JZfYtdb-z{rzgv<@U<>NbCE{M`4*Pvf(g(Gjd_uxXX6g@@4HD2-jgu!}~O~afq1n&-EY}{})mHre|uk2^pdnPd7z9 z!?MdtQDBAa*^e$faMm0VXzniAp}SNUAyj@{3yX1QZ3{A*3WtJO!ElD|PY#s(zcUe5 zHVh_i1gSsD)cAHtTOE}tRg>d_cnZ>6I?`n6i2 zZx&iJ+q13iHS8uMG#>=kMJ`1eJfA`KJ9Dv#s)#dm{@E}vs<7DV31`UV4HlxhwZX^4 z?V{H4e9^s)!OqhvRHKng%VhBs*~9a(yTo|-&93VL-9cbm#oPxCghfC8tfTjjqHc=< zkL^)r_?e*!w^#g8y`Ayo?shZY+P^|kj~5Y|t8Z54CJ&AUq*YQm)jdk}4)2;@XQ0ygbsAGT-W`Kybq0>pfTRs7RD*WIJ~j~4%|Q7@gAyZbkJ zg`aYOFPHW3*Xe~(x+01Z*X%997G{1L zOtu==9H$x>1Wc*BAY+r~5tFzdz)IX!Cx_CTz{cC{o;I?*5A#kQ{iYDrnswgc)b6J4 zwO!Ndpx^Q-GqKq>7XD^_I8k<>=kBKd=eL_p;`=*lR_38CmX@QWnj8w ziH-PUMY-r$yI0cNFaOzwY$6Qn+nZ!|3yEKrpI827dgQTiq}Mi~#!pfe06wT8sanDW z+1=t-xuHA{|M`g^*y5d~yK%Wyz!bZ$fdAweHLnfXGWj!`%W-6?z4?}?^qkSvkF_ao zl$GfcY7;eta8$03;Hc{5e2AB2A~Kjd^IjhA9qq&?3?7{L3y-T(VNnSG;q!;ZB+OD))3f%eBB>q;p0v#H&FdNzQlr=40)c!=fk6;AM#|cDeSxu!R=!^6Ac! zzsH`6zweQM31#}Ea`WtrOCpEW``vk6F1R~M!y-vs&uzwHz7#$gmm!f8N6Yh~Yh~x{ z0<^=csiqC3C=caU!`V_x-WIfd+GK!nX_g<6!&qXxv7h8i)T4$ zJ@aLKNkY&!ck8$U=Agq5TdixiU`I@x;9Myicrytce5ctpI-Fx3=ePTY39Gz7EhE2B zBXG?LgG4pHhhWb$w4%fC*iQ1D#$2fc%`4w&FxDx&i9WI6M5^@~Fl@1?XSlp`!1Qg= zx~IO+xX+o&cRo7>5A>lb7nm4jvtSsFPr5>djc&6EHODh!Uyo#=#P*6-N=H5N% z665nQzCgwoO+J?j(Q4{fP~hYtc2*JolE;cbkFr7ip4{uw?%xX3$^4Jo+^Ke|WkIiw7U+T0HO6>I*p56eNhaz?gqj#*Fnn0U7T zTdWo+0+x0QwOxC%8I)0QhR$itja@^c2*?B8Sz1i@+X&dnFWXBY8NLg1FZYT^Y?7`k zlA370jzPG{1H66jmim|4cu($*0I5a(b}uuYt^)B4CMT(*^r6{PzbYUS824jHw6Ge{6bCuR=ve-$&R&F69X z?WWVjANVD~2J8__=%X6s>^;L--VoBvcf41jb~J4LQ<`wvo%i3xLj!5pJh}64?kH!=MHxoGJg#<0(N05uOm)UKxAr4 z2e0875n4-6r8KMhVMJ1{P=C<7IC4I#vTJ3fzMnTY_tzi!p7d1O&v;)XiJQ2XacwuH z=vi3wV89FwxzB{P$d}vn8>KUBC~ry#y;oj2B<}F#8%;RGCe(k^uoC_Ix_7}6gx$F1$heeyIDid`t`Cf|@Uv{bx z315Zuoao>kn_})M^i9w{{mt<+@?_=@)iUE+%Z)Do?X3c@^{+8fn*})0>#N$3X;N+W z{q@U-Q)|U!Ge?#OCrWmEVYwC2+gv6D9q4&Vo0{SJ|JZxas3@AR(RUC96QH0Xl0=e7 z7zN2mM6x6UNE(!!k(`r(AW7m#&Ou3%GmL`doEcyk$$3T?G7P5%pXWXQbJqEI?|av} z>-d4Srn~K~uHLn4*WSO{JIQkRE<#BWetrJPJtFHfG(AwR(+A8XQ72m`Pdh(gxbi-j!z!F3T9x2_tm%F@iO8qZ^L@v>J?wM7Y}fbeV?s@rnDMQ?-S3o) z@hbLDovo(rpW%%k^rT@OvL3|L<8iAjEl|DE;aTykWiu8YRmS{w2UZtQ8^5!Sl;aQ2 zX&jo+cMDhA=%klLhYHUGF)%1KP-bqvJCF^bzA zKz+8=9^)L|-2Jrq>Z~gSD||w-7q3wUVN4lb`L!c{SU=C8qAvLL&c>H4gQ1|h0ezrm zk8Al4Kp0Yx;%IGggHcve3tR5_(b=^;?~vn~j2o2DMRY_c%i%Og8D@r+8VrXIQy)!WT_7_wETJXVk1y@Me3?Y)Yw*Py?M zz2>wU3{f>-K6ga0zLLEQ)iFf2yFQj?(}E;|t~0j0Slg51-Bsa=l=NIH!h;#WgRi1F z?#Y5b1zmj9C>r4rT^F^sia%COY?$zbTeG)FK_j{THZnQ=?x*y--)ycCF+e=SAgb!u zWYn*{6#N9CfJn&s<`7PP`bK*?@R$+L2`6QSL;q6IV!ZNwa6@<>h$K`k&{T*Y6vL zEejdX>z5q9B($y@(zq-iu`l`%>kyXyw-Z0<2B^YZaRp6C%bAPL5VQ`sRVU{=AD@$4 zJjX$}w*u|+Pm?}{e_uG)W~-#jCS0|0ej@1}#8-02#Fu(C@=~{c275pb+hTOhiNWcI zlG(2|u@*5rP&yIiEzh6aPy;WW9!p$jCeGb-r0nke^cKzE58N@7GqI_xs$ik~ENu#Y z2C^r+PP#Yg06!6X`0Gf^0^1W&-u`i;5%%{RHhk+kQhMK5Ho``YHynylI%n<_@04EmbK?nU5G&7_ zIw8-^pZchJdmYyTNiqeR&F!!J3TvUKJ<|bodbm5&n$T00;5|9c5qS&I-~xGcbk^0b zqALH-2glXk>!;luow6!~9rmv<9bq9Nq#FWA2gWZ$|e}&Rr--jW7zz{a#-=zDN0iq@&M&g<=_w3im zP`YqJ?`8bKoYG);<9+`3>33Idyx0o*+FMt-SiQSjEeBq7Dsx4I-C=DLLyh4UitpqB zjWx2PM-!!L%*`hutKN+7SX-`Lg+L&W)|tRK4h16#;w$t7OkK2Kj6ooFf#8S$7zPl? z%oT9N3Wz-f(#!;ojDev8fdoo}Bj3TGfb@%xR-}BY&3W>rT{v9cC(a`Dr^|SCYS0a!dz~ zqHhqs+^2G?4Muxqs#_^AQhhKMs#w&Q_>7?;#Q(+xd{$Te-?34l+kUEp+_VV)$dzKo!CgR_d zy3h;T-)5d|`xEj4+*jv$Kok{U*q?G~^`U}1T-jVx`ynM&rLf^qL%mM$r=E#=R>E*q z_k}{CqKv5t68zp3*ONGgnzle1GirrfmGWROF|>mV8&MJxTGL__szs!Vs; zjyYK$fRZhgwdX)zq|7at*iN|5$H<E$^5?d{3HXe2a4LPV2>?!cFOl4qKxMS0w{7mWxjmJX>Le)YNvAAHe zI-P0Q`fSY(e1!%V5FA>c7`3^qKUrBy+W3OMI-^`Kk^{8wGWB7J{Rh63U zpu^0QcyV$Fnzoz{&Br%5uiBp-I<42>+0FpO*jEu${ELAN#&&Zq@e}#l4%_=qUzK23 zGh1QuI13&`$5LsFD$^BF86T(F5&AnJK)Jd5OQ(BNBrU_i8$Ej2uHezc&ahD-2Z0== z2@oHFOus+n-W8OA-w?$7(AWIFthI`w@_$ZyI*&ZqXO8|-1zc0$0(g4*mOCz8{0sz5 z0VbxwfyjAi-|soE9nk%dDp0Icf%xd-r7Xg;DZpf;s)Azo@f#S7Tqh{Hzz?gWM7}A* z7QU)Ts*Bj=fWJqL23s(hhcDDV+lGT4hY*xsYR_#T|0$=sZRIk-6rlNk{_yM+$iQCt zVVoKC>&X=~5u$g++Lyrpe*X^OOyumlA6lg*%VWDJkD}b%HU@fCZ~RsY`~P-5Bd<~5 z0;<95P&>nLm`A zGT{2&)o+^oj}LdJD{WJTUoZ9~xo;l*3Z|a_4YHLRAE-(mmq{GW1VfFYpGx6vYOcM7P83&(mNC~tL%h!0;Rbk;|OY$T`ngL`h92j_Zc;N>c)EH*;-g{$hC|@}8 zwlG^I>01-){lA?TaFH0iKKB8aa$L4$fPz#-FxdZJ!?ZOq|w^Yh~c z@1+zPT~Ka075n|+-YiT`XG5#}amJX0jRaFQJ~XhLJRrt$03bCSE$J}1N! zVy}^6+e6PyL}X-S8c&Wf-MJ)Wx8#40o}6sL8y$PnL=l2kV`>3ymn8u-A$jzhVmqy? z)EtbU6;^^g#zD^KN=aOww8LVLxYt2hTo{QfQ=0qE+Bok=(DB{z{Hvsrdw%4qDk>wf_3qmPOyH2H z+h!&?ouqdixfQ6b7?*t=t(qv5M&YxSFs)nlR?0t7*kR#oPV#^s$biJJ3PwN3?gm6E zA5ZGK7`73|XNp|^2}B`>48K#^C>)TJm9>mLIVd_AU}8f|+qKdYpv@L3IbX(yW}h8g z0g2_bCphiNOaU~-$t>f^EA~i`t%BmMo7;Yo_ckXDUiAd0T3uZ&(svOH#>hZGxWqxO zaOla<*XOHEU8O*?)Dz@dGVg?_0n}jkIhtkMKG_9%()@@?*Rp@0zrSfJ`_XuE*m#n0 z&%w8b&6*0GB-&0!g$9{tm+5SFDxtbnY7B}4BLReOxor==Qe&8QnOTIy=tV4$?m=wI z6^Wt7mw6gdY|6+ww^BE52t+NMVC)daEu)BIZuX@2@d2m~8lTs2yk;sXYB%#?xXN<0 zP(*+*PDtsA@avV+6>OA_XZQFmM}z`&h`fKiJ9luOBxwFXcUKt8=!H`su3!Q$w)E*b z=Tlm6f9SvWujDa+M@HIX>UUSJ>^iL->|%;CZV6cBx&%3Nc63}6IaBv^TKL@j0|eny zAE-U6Jm!t*QgL$er3Ur8K=}S}9_sToBLoak(FB4h6BGnNlo>Bi|2&qHJcuACqz&0t zbh1E}nTyovH+q}Aej><%0PiKwC}7ry80Sg{1zp;DdH`)WAEhes)mp@9#i$aD$?fx} zszi=R;wEa_y2sq-U^%Co3{_fE;)sU&>~}^+QV!xb=Q}}3wc*l1e(dyT2$X@KRSc4% zB+dG6dsUlxpr>7D$-I^($MNH)Da(jJ(*1Ri3O?)a=G86YE_ot;iq?I@p< z!)Z8|RtcfMJK;+1qkfUO<>R3ju_hvod%vIyJxQFHdoe!aVsl+=%o7|!mP9ZCG2p*6{WjVsKVI#RU@lk)n`e)Ct29B z61;~9CZ`u!dem9Ji>mcHbRO*TI)7|gRIjm3qbU^5LFf$1)6M|it2&_U*mlVoC3Hz+ zqoZ+{>1o{}2`0~!3JG8s#Ky+6qYw}%n{xb6suVGOR=j|9d{oq|)hs#I02I4-nR0@= z#da;GO1h4B$3xSugyOfnlhia7UXShW=^JZR%UNn|x%b_q8 z!SFjWG1_qnE#+ZWaDYF=7%$Av8iogI{@K}(b2MxuUTwakPk~9fy+?JbjZh>YUwWpS z!{Q`@XLj3lex6+UmS9-`$_OCzWMaEzxP%FK)tWjR3ORqhexrdgA z!+HfhNB(&}8H1Sof@x=L_s)4Z99aR_{vh!(4X^!Qi@aNmDhU>#1?Bh+0z2Mdl@U`> zT|DhK`n@fHoSv?)5fj}`wSLvE{gC}U{;!UzB*G^YHAa&6y;EJBV76!qU9WS0;X-^Q z$7d?9DJppq;(+FN9B0<9bPj#$3AFFdd5Re(#Pm>%Lq^rCT-h+JmIz-UNE+s8TgCqw zqW^qzfS0&mabj`|WPN`%ceQDHc)z{9y`uG_%;Zgj%Z*H7jJ=J{{9 zBbX7>7X>S}R@A4+uOUSUubU$1(i23*-u2lmuily1 zb5_<56qL{=j*292@2d5&4AcSdZ|SRFx0KM_?6olhPy~ zdoCc+wZJq(qEaFdMz194gRc|6;~p833v%*H4yjEzQa1JH z*by(r7vb&Yq0~D-k2d7WHwO9%+47<^JJd0Hxui&^c41gkWaU*{y|GfGlZsSL8z3qN z^{k&I>3bN1Dx4w~nB=LE=BzL}1v`nU&*Y%`E3;EFElF|&Lj}%%iU@WvUGL!}_?%YE z^|PqHR-WBTbi0N<&=SQSm4(PuP$6-%;mOm4(MJ1snkH2Naqq*msne%&%8lUWd3669tqgeYr`ug_{}u@X>mu zl8Cce4)?q?`^0;z;U1FyHWTJx=FZ&S_3;dyX9G-p0yCIfAR}X`$?ct-;loChlSqhx>u*YgGc z22uVq!}@Z+f1X9-h+}Px@|5RH;zWg~pAHrLU^BRPyShj9v%Sx6(G%@+oRT4?y~$p? zNlc-%=Q5Qu7E_{Vz(iD5O0z7dAQyZZCuCE0Mxh zz*RbtIqNt8(iY7BPL;A1a=Fu;ASyQ@FKE{htuQdLG4?wPAS)C??pwn=`BAujQXD3) z`tiHsBYl{=j)Ul3;rLuAzBh$b3IgCb|QST65FXK2SGw;1eM9r@01 z$@n)g86k%d`)Rs)+0D)A4c8ez^>(;Ht}UXFtJ7U$5@lj8_xTxAd#>hUDSry{7& z&+Cb*>zl`w@bb-{)Vn*o2n1FDRmFhYDQBAqvbw)%+ib1VU}WT4X`^Q(pCC}+ctAT+ zG`Ltp?77yc@uPzxBV^?W$yw53Bt0y=HlC4S{{A0RZyM6*Xnr?UadCL9O~hp*amV%H z2jH!YJ)tNNQ!GtSPv5@8U54TDV&oTC%I{fi!!{iE-te^Wm!FHGW7T_0mhvKe;c6TQb8EB__JZ{h9CO(O69^dhJ6?m^>mC;hCATQNcM z0WgCYVUS=-(}(wd^&M7L)+=GwCFzcVUwGXNZXeMGEO%?xez$oLMhUmySkSJtu}0Xv zc*hmyT&_%fd|mcn*F?U#&1iq66Z&{{u^?|Xr>yMBZdn2Z$_K>+UbaKMQbF3$&jAhRrWbTagO;kx=(%UXASdsWl45f{ zmyNYP$9sm2aDK)Rd&jct=)u8(Grz@;Ohn-{pQ~=>9<0KiP;Q+`)7NqT!sIUYGjTj+ zsVOk;*F}O{)Yl!Vz)<5hMFHNtce%NT12!4m6dN1rm5FoA77?b#ddD(|1M+B1cz_8r zs6c6R)3t?$->~;l=#BkQzhmQ>g}kS1ce+7!%)E+EbwD&WKrW059XFdbYf&TzvYM+X&@oCq``GIyGh!9=PnZbl%JGC#PP%MT zQ6qj}9DRs!AWo}?t#AJh7<}A5Xz~2Ti_BX8hX@a?SCuzu%H?)PlZcfR70>*h`gWj( zAzu$6tFBIS^9Ez>R2=&0lSe9w+3D(^`i?GHdQYnZs|MfI+NYy;J%XE9E=%IFwrqR5Rt@G_^7B)_8V5V3C zwVNG!9c`@5!SS*OwmkNm)y<{r>r}BR;_!~8(qbCsfEAjMoAFYjgk+pNGv`Nk9(M9Y9G&)8Srdz zPFGk`Yw-zGQ5Ob7-N%ODZWnNvvG>xwqUA=&={V;ApO8GQi^TGgwpniu!$c8#V9ms3 zch2bh{0I--S4trv7J4q)@GOT#%_QP1MyQ_7a!<#l?R6@Vdr~IDA)zQz{o4{KxYus~&p(HtlJ8I{6S0PA`hAAwCMD2RMXF3NaM(Gi-eVl(FWrjqQ4m$C`RC2Z|G; zsI>&IFWP?G+>h*5K)G{j%T3236vt>bIjg!1CG&e64n{YkjX)aR77S1w#kkeYB@W$rB7wUbUK>A6UQW0U17z z&`ed`x!2C=BBDhi-Ddnw9Wha3PMA;0l^hWz#=#~)%KOztwg8K@sDk7Irf%_?zvDaN zG%vYdBX7wT4@BQFw5);dd28moTlGEGvM1MI9XY%(pqp)R4eeB9-n_kK-`ZDx0eA8| z`)Q;}$%7Zxl)8IbW(UB*bwQ)B#mv~*T}jYu@4L&bfFjZB=k=32kf95X-JEi!LfSHQ zEK=p1d&EnNr)ZpWPvYzGHLwYJkU) zwu5b@)M&*zE24FbB+;kfj`z(A*lj>`0e5T;U=|r@0c%fRbPU-8yCT)bMni>M7Zs*vBa9Vv@oX?Z{(Ec*mu)xBk;$5SX2+$>Zlh8&Hw0#x=QHE z_^YGgj;+eRqx8rTDWn0=tbX4-&{KJL?<{94t)&RhO4s)}mUYhRZ*>F-H1)e`#a-o- zzE6Ipu9Y9tVSMB9Mf^v%fmub1nUim;6Dl#PCz>AvtiwaX_m*PMn+3SS8EQDH0YPjG}x*(U=dTp2H*vH^X@M`<`cD+GTZ)d z9Vs_KlHaItRWju4-T{FK@Lm{%Fy@Pu4z0jf%OkqAKK^)XL@2@qlU~~UbD3VqYOL?k z-cQ3#3|I|4oi$5L(iHc37U{M|yHCRDVw{ti8FKT%)8v`DpfZ>mx1iu?|KtsB1Zf6W zZB+7Zt$7>@Z;rJB7)w>PYrrnDqn6h2Ro z$x7?RkJAQQCA`cg{;%oI?pci1*_)Q8aw6o8xLl)SVpuV@5wkb0Q?_VVnB8N%=#{Gv z>a7ZAjrNYd7kbAaQ2;}E4EMAiB?Y7`b%b% zblZ3jzi$`hT%r|JRgy)4SsYfOd2kHYno2G(j{4}bQ! zp$~SJ&6CsN+;68Hk;_{jQ_lUQZm@@aYyjJJOU&31o!=}6vtz`D__J{vkOkhe0EqVM zTmP#6@Ob;RYq&uZf{*w9&E%$q?psRf4h(7B{d&pU>y?_D3G?xqVRf-A7v0QF3maVj zq+2BtjY7{z*`Df+ni;wK2Nt`O_LBq@aJbNIxWd|U|E*R)t6bg1de0k))McZL=W%SX zNCB{BvGjTzliYT^eB@>l$e=Zt?WE+DTWSBfh?9W%n(WfhH zdyfMr;W;9F^evkRhFM)W`lHnF^33nqUMHHoBqC@qm%f)Dww@)?I%IyD`_hYgBQG!^ zBi};xEjQK$mXH;tC66x+X_ zf!yX_J^!*k(+%NNSdOdS&b9-t?}!{Fecklnu1F(d2fO8)L?|wg98LX^PUYo4SYG~8 zRRz_!*!}1}Rx%vgP3y`cT?1nS{26eAn0OsUltnmk@};Kz!Y$N8^V1HAS;tjCei*hq zkM=sp@k-n28|PaE+|pt=V>|Cf{)!d`I&vQ@pwcQcMLc$_z@8u@y>^q^RQkmR)ei3z z?Q2n$y0Mzbt|ZY~uiAw z6vrk?j7zO#4u}J)f0u69!pRg4mO$~ucp*Dau#R2@m53D-g5m-P`HK=??`@Kmpn9sR#w2&-dnD%Pw6wMn&MR{^s8iFWL&92AT_w$m^&RQ~{jqBS_ zvgUNOHG=103Nta}KAamW~WpQ?XGan*MDIzb;S$E>VpXJtA-#|PZ*zr8VwGd z--Ly)gzZMwvP22da>454RUzabMz?A)mmmCQO1Fdw|G>&%wjWw?E1Ux^LuXsPZ^#&$ z2YhGs{z%K{uEQRO)rNw&3-0>@D+Cy6+CCvsNLq=)AN1(=znhTTg7h30yfym;!V6Eqm}qb-3k?Brgijw}9*^Jf zlVJ^-{I~ULKRDkGU`Oc+C|wv3c`|6SMw9CtVLbnke9Er=t7>CJV)e%cBBYn~p(~#k1)l%F4>2{S>k4a^cw;9`}B1L-XZI z?#+?K2wmldyR&l;7Td_4x6(Ce;BPS$?Cccfz97(I5nb4rzqp!W3jGuuOwOp{%*egw zOsr{Q{nfeK!)0YiVZW|Q^B0)F4|YKn9mhy0O&saC%G%W!qYlEZ7j|z^H((tv{z>X? zYik3(@Bq~9lJ6yL#g$D7BE+MZSg(Sw+DDcjs~!_17m!Pc@+ioWeXBF5$6F!?L3E$` zR!@)X0E-_3n_5qUydC`S4VwHP<23nznfTolv`Kw;-7FG-5D-X&{ItT#(f%2LU?4~> z_z*c+teyI)bN@j_s1v9N)_A}e>Pj${wS-aY94{{O{AAE=LC zg%ynE@}?YtYq^!9Vxl;0KH5VDIEz!t4{$1t_Dr#*Zs48}$P1?>G*2PpS%|kr0uhW( z==*a#IVS0=_m*cDjgJb(wRa$KHCo=U)B27eHX_9k^!>VO9A$%3$g#z--M zp)4_X`S_-6h#=S$*9+JmSjKzxwO>b)#S`XjUjj8c1)!%?e23(g*Gj~D9f6|zN$J4j zJst{>oNIDKu=zQ<=?3f2{b1H|>(!O_q|PA9WiQ4Y^OtpnX{*|{A z#zG+8*IkHUNn&y4J)i#Knwu}JQDrmo9YpD(5CkPyKj>AdMaIN!;Dh1;6qr2%*q8x7 zREb-b2Y{%Cx_LHmwO#idnoppp? z5`f{xDx6}RH17QcwY|P|C&XgL8k{l7-5eZ;jlJCsj?3v&f&lb`iIMD;nr=C7yu#p{ zWKYrG`4kcRIRrqM*1Rbph}Idf-?PDUx$lSG0%im-ZO_QrO^maBQWChoHl(62|9wJz zKA)*QP$PC?vu!AG&7PSHyB{?=9@RFTg<$i9E@FTEk`b*a8kE$mb+%~}_38l1007w3 zP?)@`q?N%@e@4LC4G2X46CC1$_xu5#f-8FA?CdWNz5v2hQAvq>Z91(!0}TH03?c}g zikb*N^Kqa+=P~^2M75!won6}Ll|{U&nbwooB*!Q)7nT8D&og_6*wf8zyfBG&u6H18 zeU(RlK@lJ+ke7ff)E8EKu=?d+O$aDC0e_pMlJv~Xu~Ulg^?h+}Ej~}6pa_=`NIxj- z?o`3nViHf>#3>uQD@V`N%iCO&dCW?ze$3|3W%PEJB4aRWX98 zYB9MZQu48fvn`l&L4RbWqC;>=6*gMNKqc)-Ta)ZjwT5>vrbR~||3mmScxr{ixS-Hd zLKmchYz1Hv*uUh6`*$id^M!J75N>vQ(tZF2PI4W8|2t2b`AWL*OaqV*>xU%6ddxx9 zM1anNAQF^m8>2R!OA!&AFa*f!zZs9{q9$r6E5m`Lw5wox09EU50q_m1Z9uraP^Ypm zJ^iOeTynAmS0tdSHpYr)K}fu(h4(6b0w6}{3dtG7at#5+WKNjps@MpCTq)SY%bMyn|XCDAnu1kGPriTDL8VjY{XgtA_(F%NNX~b_AIqhvs)zS}E830I61O!GK z2y?ox<*8wm*GtrppPz3x+rs88heN-oFflQS&uJ#d&J@;ehycJzfDJ(3zB&Y^#9JeU z_-L%yP#KF~)jt6cP>SMCPj@%S@VLQ<$CC1`0JMi+^#Kt+LXJU8-yA?myk8G7jrwwA zH}5~9r9u+{hM2&OOXdMQVy->PWzsG{b_qCe;c&QUE7JqQ65y()ezCw_!&5}CVo=Ewy&IJ`CP-+E}KrmMT=dJm{E zqtm`U02?AYJAD9*X(^V>Zy75tQgf52!RG`ovH~(h4Ke}C`{T!t3qqc8^{osZPfwsE zh<0!%@d5E70CBG+%b6v0JDJ>a2LQk z4Knfa)7xXwKS&u|ANfyu?M{tI;lZ73{%r{eRA#DcB})3Bge}01?}awESx^U|4`i|Z zh_7rsE{J=rOnmg^fuW;3oJa3zX;T2jYSw0cLKi_)PJ=C=H6Vi1pUai#5=2Fr(|va~ zMibUwTugk#*DJ>STp;S3IRQ#7h#OJ`8{mMX^v3gZva)8aY=LH^6EN|V;a-S74CW1} zDa#jx&*Tu@3c=KcCW?5DC99fegu1May$PiRaf{L7g$jpskWF8tkK@YuDt2HUq5ppI z6e1XbhP)Ln>bUeeJkMyRMifyCHCWss5PQhItzOB>)^qPc>qm=cTS6Vk0U#Q|vg5H) zE!X-@6jL`?ZOaHg@tuD^F>qcy_~SZJVZF2Tef2UHpsjr^3pKbFp!vObYp0iAIfRRJ z{&-JH&7+%3T}%Z!x|Cqpz4*I~tt4uws@fB5+z&OG1&=TVNlUV_vU*4WO`;tCu%Be~ z1{!qcx6ogKSDCx8o169yxkq{~Q^v4eTc5*(%hl@@6LVS0v!nVxdOF^Ksx!F_War7> zaY>7oF))|`EU08ch}UA0i^R#%7BPr_4}Bd3V@A^3GL5e@hSr8N8cE}>Gjt5kJF38;g6t)DZYcZ(%Sjs4o z&Ul0-1;nkOi+`TFFEv0eDI|~H4lBTh)1(7{#Xq<=S?v%miLaWv`?RFq@EhoZh`&{9QQeHlfq}OiGZ*I^rj-g z-!vQGmz7h_3Fb#YUwHrdMeQsK^oji6g`*^b7A*Gwg9{ueutsk$dwY8~x9X{x1Ax-5 zqv}ODINC+VQQYk8a<O8R%51`gBw zzkj$+Gk}7fajbEgL3OG#$T1rWhypdan44NE->; z#QMrG=lazjXaXuU1m>4}_wM0NicMdku9-jA*0u<*Vue{uWXo)^?ewia9}F!;t>P?N z7ilFdDbA_w{uLQ(U%2O;`u%e$J(Y4F>ifP`hm;4jI7*)(P;Hl&rZwY*Nh#6Hg|T~A zMG=gW$xM8WZ-1Y$-0MD%kxBCB0H6xt0R*-to(2G11F5FrHamej zA0a-MuAu$?`uJsicUSz+?n%8zeJ06KNpd95mnOoOf~O~lE`k@>3A_vJzrZ|JZ!lZ6 zw=3{EcP*)pV0CVwE3L*SL8MRtlb)Zy_Di53GwZCAsgfabsGZk&{?j~o&b-Oly?C3% zZ3ZHS+`kTwvwQb|srLLCd5SRA8_l<65I|mz4-GMcyfNgcR7eWQ1kLU8H8C*}1c1DL zB}m;-D~q|JtfJ!F>{fBcQ&zU6+F|jVz=fbH?aOhD7Yc#;D3N>e4EXdy!Bnz-^<}&0 zbvyr<{D};?=krO!0i%p)r*{uZvRa)s$g!f%R`AaYPcarfiK?0Nz~`Ds9Fwnbp|?AM z|EqIMhiGJSvWB|)oAC2dIi1co4b$HB=b$eGv-DT-bP4camDnpP6&+igDZ$T6-}2B% z67z00>c3;tTl|rlowcO{WSO44Nt+Wn%&KHm?+zT9Pseqge;E40!@Gj0UJ4|al<>OA z8E`egU7nyEwIoMG;Jry)Ne+Qg4K^AYu^#X3K2@#x7X9NF(M;^_Wqe@p1?OU1HMFWH zikYfy9Y@fE?0LOQ6FKk-TU7p^#Y>XGSJ6OiWpOL>TO#G<;vkJS>+zcU%ejUVm6o+u z%at#4uIiRsWL+L5QBDw(bn2Kh^I8(H)^5iorOCV0G;A+ClFLI^6&o6*R<|X3u&$xX z2gmUMkrA}HcAp|EgNR#%2Tk zAi+e&HOTvI?|HwcmvbM8#A)E>pHX zd@PY9srs`(U}?0nvN_0%2)60Zzx7Pt`%iV)Fr*RWQ3dd2jGz*JdTDLWcv;(% zln-fi>9x@k-oP^&GF}vk#^fr`n z3+Uj6!=5Wi!-1>Q242lN#%-$9ra$yFxKSW+1pILGZWosI9aS(kHrc_9 z%IT}zVrOSww*VQzVXxL3ZCk}vwNR`4BuLr zn+B7?Wx6p{kskxQ%9EA7WyNPaVX$Np&7jH1&@3p=I%_sb`q{nt&PLk4#v*46kKGlX z6UK*2Wb|3}u($?CXI3kDIp?L%m0861h+Qx!<817D>#Q`zBAUF_WW)I0U}hF z(@$4kjV*BmYzmnM2}4}!_aXapNwa7)j3Xs@JX@u%>RgVg|b zgn8qexWhsS%cHocVnc~9BF6RsJ87VPxgeP%t5WRe>$HZfy?}nxEMckVc+S@DDB|pD zKIhRy9C402}%IEiQw*Ks{w9EHV!kVZbEUVRT=Z+*T4yQwey zE-R-TY%~cK!^xMu^H#vEz++LA`BC@_@-$t+esG$Uo`yU#~tu zbed2Z9_ut5I|ubtSkabzUE1NI%^aU|^hgGQlCpJSe`Q+1vp<*1R58h$Fw6Lu-q{8( zkMbX2FO6}aC&T>V9KrFfnfWrW5cXXSF7Fc*b^R`sJ3{m(ihTO!m_A9zq8R`yHXGU; z2w1f!Ldh_wYEzG0iK+CO`+H31NdF$^mDazxR16f5UF$dC)A z6_u4lrg&}^AIS-j?9W72O8d95ckT}k1pda5Jf^lY<57)(j|4Wn{xNKe3G9Rc2jxbV zz$OO|503!%``?z)ihORRiNt;fA5Pagy~K5ns;rUKN}_?;#cXA6TZJ&V1pjs*PFbhi z%B03$!+|$!z)PS)A}04dQSkzD2uI%1}TkIRYeUxCGEp zkE&K}{g*b}SKH2K{g{}1sATf$T_tX`BRXYaaMn560!gP}y>(|lraHu^>I8<7W>fvZW z$zBOlyQnJ`O*AMAdz{QlG<@049;7ZhKoA`ptuWOm;z?6^)oYz)%xPotZ zcsNJFZKxXV`E51Wln5BLp*!f?O1`!Uh6pd{HmKerNw&jJ-d18EV@cFGTyH$sycE}y zT8X+(dI@a9z-KV`AgUkcqAF0GZoZH6u7c666&gM$(uI^;_Ie9}t_g2~>FLQ7>N8xIy>o z*)z)XnAf8Ll%?~(0kEB9eeqJS(qh%bswPq>sk@g#s3&h!@0n;tfcpeNFA>9`Cn^jp z>Lg&|AK-@@KYXqR?9M2G$gJ%&K4t|*@hd25mvX23cij|7;@@ZSW4i#3U)sI1K4F9o z7|f1mx~drrQ=A%q%Er0;U?ymWT_y3edQYX{+82ts<%Ka5D5n;T6TLSq-M)|}xeFHC zf~3NnTGx5sf;|yZy{nT7yNJfgVm&~t&KHCWD1A#$f6NAy@{>3}fSpowBNmjy{9i7` zn&`Qxp)9rP-3^?*@ihBBQc`|2wH7azM(>n1mLWOLassaHPfUttbe*VQM`xQ-P1FfU zy*o1g$Z>T<=7!kJe(wf^(c@SLw|IYb3ZBYvo}PXX#^^qtY)mGqq{`FxtC;5nM?zkH zRCF|_zHbRrR;I4Fk*wnyHj*oUU-&8941ZMJBkv@W&Js7f7JYh3^%J3FM(q`@Ea&NT~h3ar@O~x`%Zr@bUis8=4GLC~j z8uYHeXzNA}3;V~C$?|iqc@9O3nhAqK{5O?UHNyu5STCr#_g)RZhrz9M7HA6$o`8(` zhvh0oXPY7(#p+FjyPt>$WgnazA5szj9<(V~UKQTYo?h#c2X4n`c*SN;UGCQ1<3P={ z(Q-lK4i-L^8H?MfYWengHyl&c!$Kc~De&_FqfS1Un_Mxj@ie=)xBRk1DuxrYl_g0W)HX5`1d-gC7#9{~hN z5}DRX3isZ_nd{%c7RVxY{#P$EASc@FQ!ii5dm>-YXW?yJ0h`v7)Ww9`Jh?C{-6~cI zY!q34uo>kdBOLXZs@A6Rs?V+aj!F(iXQf9@y5a`6tIPn4roVpe9mAPAoBIBHGyG=? z`4Dmm*2nv@MS~m09z5R^Nmpio<92&xP!|BZ-SgLzjL*4R2H_rk*_f&kA^ipyt*G?_ zM5id_pXWD35h<5Qw*$AJeAC&yW3{GZdnb(bfTE8H|MN({EQj>aXr1TB}HE6OyHYw?2|TPqW=866_;-RBuz- z>?)2M(p{23Dt>=Oqi`_jj`YF6#Bn08c$;n^qTvXtkP__>;1vMr3e3TRh~=|9-*V9e z_dBWfvKn2=ob5@??n*#Zx=ZoyDvsp+n8HFZc3`r4S%0w}##sogv`rfgYuA|M3+bex zo!S;RUD+fhe+r+ME!RQ5JGVvrn=cmlv%eBKO(FS-hZ^ehyY}`ih(82MY;Be7w9kQfl!V%+9`})?;3G{6F)15)*rt2VV)< zO~U&7t6r0{Hp6iL1z7_|6CfOPtOF%5)l6BFDw0Cd=XVN{>#*$jkm z&SCd?<`#XGFcHkO#f>S~!kl6cXc83dgQu~YF1nqKnU?dq;A zZBGbG^#K@^HO|Wq%?Hkke{nv*nVY~fHgCG#DFn2@HSuQ%d1Ym?s*MlSA@=thv;W*W zx3|pb-{V(#=Mz^pOySU<)HQr{vi&0VN5^F-=BtxFkF+KdteET_q;nj?Z{iE5((j4C z*W;}gPk%Nok*05vDCN?#)+fHG{BM^*+a;XW%<_~kkJ6x-dz|g;BI@KQGq=-P*TQYHchF4PxOdcSjV4A$Mz;K}o&H6RO`ljz`(QQ2pz8Cq?muKM7$z<~I#KE^(ho1~ z@pFOJ4rH+OEG)XXmm98uar>{xQPAvc^!d(e5B_+&c0nIkRb0Ij>2%$_JFzDt!oj{| zqRf1_L}n1c|3inEb3b=UT*yPlR?AC_Wo69*+`tYDkzHF;V-tbPV$8(dSCoL%A$Ip$?fdjf^h8`rHlp(3GbaS1#ZROuJ8!*9-u=VlwIn?TV%6qhULjBUx~<4j1%&{k zHjl#K&ncoVJXa({M=$pilO+z~N+S6`xqx|yPsvlr`kM`aG(9ym%sRZ>cgy_cLg+Mn zaV0%TlSC4)-1J0t&()BT$saW6GzL|fV?C-3V-D$;Fmmp3BK4nZX*g*!lh0XiP^tNh&~oAerU8dHCi%%KyVyivm(P|MrccwK#nJz|%9Vmp1O!^=mp4P_YxM8tU$# zR=8sIv+7OV$*EB3RZb<9Y%k#1Da&&^tl;+8sVlV{O)$4#R!-P3xACkL82I{}ofB;Q z(_UOW9PAnOa7RI(GP?|;&1<#OsqRRo)1({K)Yhg`*fuP;?5!6>O$Z(@1ti=Hy-+h- z*?N4iKsK5)SSOA{s}A5y=nLn#L*1tC`1|Y*1luVwc%t!D-yTkl+7s#PPfjrA8MKDM zXLAO~5f$y~=i5|9-0~{G8LPdjuLX9)*Ro!ii_N%9Z_k_3amssa&-!P@d@rz*n`w_- zAW>qs_d)v|k~6FYIo2Il=p=jEA3B8!lh5wB*pqFOpOI*O&rrRWWIKJOu{h7tM{FI_ zuo_l{7*v&Xn(x)yxlY{Bu*Z+78vkPyoRCDG98T@MRlJ1VpVSbUmzvCQHED^F++DLB z{Q7XPEU?&QrV)PAZvJq0eX3?Y$W=JA*CqLAxB_=QsABi-lgic(RYCs3{b5c0>TM4} zIY{fLgImN6s2|J&J1?DweZy^3;rn=TmzMSEKOLRR%xML|TgF2ZQ)rFujO#Oa*!Bzf zccSyIzPwaecxZ^{&faXJS#mt5z}?!S#MZnk*zFeK-sko&ZvCN2qM5I-M~u?`kM^!C zsL3k|{~?ZXK>|_;f&^Ozq_!*;1Z5FMLH5AN7%)Uu5hWl+wkS!!jYXgoK`2EC3PF&4 zGm)^05J3cnJrVxRTlQnqv6LCD=ujTJeGIz2OpO*-gSW#*$iO3Gb&PkbZ14b942YM|o&X)2EKs zZq1)X#%AYdatbbBln6X)d|qvyvL3nokLP9s+zy@DLj88%FrjQUl}B@F5RTg1TFA6Ts2g(Y=-UtftRIzO=NcHu2tJ!!+@T!EB-C^I7+aaRj_uaS=GuO> z+9&Lb;29ZMYw^e{@3FB|?~w7O8!)FtNfMkM*M3M+NV>#gr1B;s?}J5YuK$E-U$;GH z?JDRLu&KVMi3!{ys>N17AT^q@8tBN9inxr?&N4058d7id9OQRgO&2ACUt1tWnYS_;UL!ClsXo%uI|KvMX(fC2!fvJeQ-2Zh(h z@4p@o3S2(h1w}gWD2C*^};VJZxFM7!U>OHm=xKvm6Cd+}io5Lx5g=F(h$A?AZpDAeWCBfrVz$ zmdEY)$v6rC_@@LV1qJ=gzP-lVl7oAKaKL**JmwKJvlKAQ{s+?GN1p8zkXKKG4KAFb z_JWXQn{jO|6j_mo#R7JxszJ_XiHRd8A9T=Zpz5SPv3%K*1V&;TO)=?)4>yVY_^?N$ zLoj#fyL<>xzc}*5h%pS!Wm&=qi@3qvi6seAE_qlDT=T4h!^!n>?Ia0Y_VOi@UrSY2 zDiiQ(r;6TzITQO-c$I8emd=kx&=R*vTQEE#UGus2O9FwR5fm_J*pDc0rEjYhI;SMS zK}(*$Oxbs(WM_L-&M&;HM9bS~8bAr5k1@=fk!MZY-2&gvM5b~QPH(PfaZ$*&Nld3! zfEH23CqG@<2N=}!_lt-+lClq4u!O%Qb#-Hi-8%LxM5kPz0gn-&1Q(8<-V`M0>OQNQ zNMkZxd(#zw@FOi49;bJ|s%r4vyIX=zXhA4kf=mO+UpFjxR7r4VpVkv)?Dr`3f5j`du4>O9T(#O^} zH%BQ2oEItpC7!pNFveni7RSWi;B10jL;^LE12y`R4|{p3nwlbTEtn|RJ%D-$$T#p( pKzad5ihJ*iM!UXgw;Ov01uRRlR=P#?QLhAqk literal 0 HcmV?d00001 diff --git a/docs/content/guides/inference_guide.md b/docs/content/guides/inference_guide.md new file mode 100644 index 000000000..14ee35bf8 --- /dev/null +++ b/docs/content/guides/inference_guide.md @@ -0,0 +1,708 @@ +# From Platform Engineer to Inference Engineer: An Open Field Guide to Inference Engineering + +*Numbers, prices and tool versions are as of October 2026.* + +This guide is for platform engineers who have been handed a GPU and a model and told to make it work. It covers how inference works, how to run it in production, which tools to learn, and how to turn that experience into a career. + +This guide builds on people who have written about inference in the open: Viktor Farcic's [*LLM Inference Explained*](https://devopstoolkit.live/ai/llm-inference-explained-12-concepts-you-actually-need-to-know/), Philip Kiely's [*Inference Engineering*](https://www.baseten.co/inference-engineering/), work from [The Pragmatic Engineer](https://newsletter.pragmaticengineer.com/p/what-is-inference-engineering), [ByteByteGo](https://blog.bytebytego.com/p/a-guide-to-ai-inference-engineering), [Data Gravity](https://www.datagravity.dev/p/the-rise-of-inference-engineering) and [Spheron](https://www.spheron.network/blog/inference-engineering-guide-2026/), Vishakha Sadhwani's [inference roadmap](https://www.tech5ense.com/p/the-ai-inference-engineering-roadmap), and Upbound's [Project Champagne](https://www.upbound.io/blog/drinking-our-own-champagne-what-it-takes-to-own-your-intelligence) series. Each is credited where the guide draws on it, and Section 7 lists them all. If you spot something that should be credited or corrected, open an issue or a pull request. + +## 1. Why this matters now + +Self-hosted inference is landing on platform teams. The coding assistant can't send source code to a third party, frontier model bills keep growing, and the support tool needs internal data. Each of those becomes the same request: run the model ourselves. + +Open models made that realistic. [Hugging Face](https://huggingface.co/models) hosts more than two million of them, and since DeepSeek V3 (December 2024) and R1 (January 2025), open models have kept pace with closed releases within months. Running your own lets you tune latency, aim for four nines of uptime, keep data on your network and, at volume, cut costs by around 80% ([*Inference Engineering*](https://www.baseten.co/inference-engineering/)). + +The work suits platform engineers. The price of a fixed level of AI capability falls roughly 10x a year, and most of that comes from how models are served, not from better GPUs ([Data Gravity](https://www.datagravity.dev/p/the-rise-of-inference-engineering)). Upbound, the company behind Crossplane and Modelplane, saw this firsthand. In its [Project Champagne](https://www.upbound.io/blog/drinking-our-own-champagne-what-it-takes-to-own-your-intelligence) blog series, the team ran its own models on [Modelplane](https://github.com/modelplaneai/modelplane) and found that vLLM and scheduling worked. The time went to TLS, identity, timeouts, storage, IAM and observability. This guide draws on that series throughout, and Section 5 collects its lessons. + +### You are closer than you think + +An inference engineer picks a model, gets it onto the right GPU, keeps it serving, makes it faster and cheaper, and gets paged when it breaks. That is SRE for a new workload, and most of your skills carry over in a new shape: + +| What you already know | What it becomes in inference | +|------------------------------|--------------------------------------------------------------------------------------| +| Scheduling and node pools | GPU placement with device selectors, taints and exclusive devices | +| Capacity planning | GPU memory math: weights, KV cache and runtime overhead, checked at runtime | +| Load balancing | Cache-aware routing and long-lived streaming connections | +| Autoscaling on CPU | Scaling on queue depth and KV cache use, with cold starts measured in minutes | +| Rollouts | Rollouts of exclusive GPUs, and a version matrix of model, hardware and engine build | +| SLOs and latency percentiles | Time to first token, time per output token and goodput | +| Observability | Engine metrics, real GPU activity and speculative decoding acceptance | +| FinOps | Cost per million tokens and cost per user session | +| Identity and access | Credentials that last as long as an agent session | + +A hosted API is still the right call early in a product's life ([ByteByteGo](https://blog.bytebytego.com/p/a-guide-to-ai-inference-engineering)). Self-hosting pays off when API spend becomes a real line item, when latency or reliability needs outgrow vendors, or when data can't leave your network. And the field is wide open: in August 2026, [Chris Zeoli counted](https://www.datagravity.dev/p/the-rise-of-inference-engineering) about 58 open inference roles on public careers boards, with no settled job title. + +## 2. Inference 101 + +Every inference concept answers one of two questions: what is the engine holding on the GPU, and what limits how fast it can work? This section walks through the concepts in roughly the order you will meet them. + +For each concept, it also helps to ask when it starts to matter to you, a framing from Viktor Farcic's *LLM Inference Explained*. Some concepts matter the moment you deploy a model. Others only show up once many people use it, and some you may never need. Knowing which is which tells you what to deal with now and what to leave for later, and the table near the end of this section sums it up. + +### What an inference engine is + +An inference engine, such as [vLLM](https://github.com/vllm-project/vllm) or [SGLang](https://github.com/sgl-project/sglang), is the server that turns model weights into an API. For a platform engineer, the closest mental model is a stateful database server with a GPU attached. It loads a large dataset into memory at startup, keeps per-session state, schedules work internally and answers over HTTP. [Ollama](https://github.com/ollama/ollama) on a laptop is the same idea at small scale, and every hosted model API runs one behind the scenes. + +| You know this from databases and services | The engine's equivalent | +|---------------------------------------------------|--------------------------------------------------------------------| +| Loading a dataset into the buffer pool at startup | Loading model weights into GPU memory, which can take many minutes | +| Per-session state in a session store | The KV cache, one for each conversation in flight | +| A query planner and connection pool | The scheduler that decides which requests share each GPU pass | +| A wire protocol clients already speak | An OpenAI- or Anthropic-compatible HTTP API | +| Warming caches after a restart | Cold starts: pulling a large image and reloading the weights | + +The analogy breaks in one important place. A database's working set lives in plentiful RAM, while an engine's lives in scarce, expensive GPU memory. Most tuning comes down to how much of that memory each request gets. + +### The four layers + +The engine is one layer of four. At the bottom, **kernels** do the arithmetic; they are benchmarked often and rarely chosen. Above them sits the **engine**. Above that, a **control plane** deploys and manages engines, because sooner or later you run more than one. On top, a **gateway** routes traffic across models and replicas. + +![Inference stack: four layers and the clients they serve](inference-stack.png) + +*Inference stack: four layers and the clients they serve* + +Requests flow down the left column. The control plane sits beside it: it deploys and manages the gateway and engines, but no request passes through it. + +### Tokens + +A token is the unit an LLM works in, roughly a word or word fragment. Context windows, prices and speeds are all counted in tokens. When someone quotes tokens per second, they mean this unit. + +### The two costs on the GPU + +GPU memory holds two very different things, and sizing a deployment means budgeting for both. If you have sized a database, the split will feel familiar: a fixed footprint you pay once, and a working set that grows with load. + +**Weights are the fixed footprint.** Most models ship at 2 bytes per parameter (BF16 or FP16), so an 8B model needs about 16 GB, and it has to live in GPU memory, not system RAM. On a 48 GB L40S, that leaves about two-thirds of the card for everything else. Whether one person uses the model or a hundred, this number does not move. + +**The KV cache is the working set.** Each new token is computed against every token before it. Rather than recompute that history, the engine keeps the attention keys and values for every token in flight. Its size scales with context length multiplied by the number of concurrent conversations, much like a session store that grows with both session size and user count. + +You can estimate the KV cache per token from a model's config: + +$$ +\text{KV bytes per token} = 2 \times \text{layers} \times \text{KV heads} \times \text{head dimension} \times \text{bytes per value} +$$ + +For Llama 3 70B at FP16 (80 layers, 8 KV heads, head dimension 128), that is about 320 KB per token. One 128K-token conversation needs about 40 GB of cache. This is why the KV cache, not the model, usually decides how many users a GPU can serve. + +Three details catch people out: + +- **Mixture-of-experts (MoE) models keep every weight resident.** A model with 320B total parameters and 18B active still needs about 320 GB at FP8. Active parameters set compute, not memory. + +- **Read the checkpoint size, not the dtype table.** While self-hosting its own models for [Project Champagne](https://www.upbound.io/blog/drinking-our-own-champagne-part-2-finding-the-corks), Upbound estimated one MoE model at about 315 GB from its dtypes (Part 2). The real checkpoint was 177.7 GB, because the experts were stored in a 4-bit format. + +- **Runtime overhead is real.** CUDA graphs, activations and memory fragmentation all take space. A model that fits at startup can still run out of memory days later ([Project Champagne, Part 2](https://www.upbound.io/blog/drinking-our-own-champagne-part-2-finding-the-corks)). + +### Two phases, two bottlenecks + +Every request runs in two phases on the same GPU, and they are limited by different things ([ByteByteGo](https://blog.bytebytego.com/p/a-guide-to-ai-inference-engineering)). + +| Phase | What it does | Limited by | Metric it sets | +|---------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------------|---------------------------------------------| +| Prefill | Reads the whole input in parallel: system prompt, tool definitions, conversation history. Produces the first token and the KV cache. | Compute | Time to first token (TTFT) | +| Decode | Writes the answer one token at a time. Each token needs a pass that reads all active weights, plus the KV cache, from memory. At long context, reading the cache can cost more than reading the weights. | Memory bandwidth | Time per output token and tokens per second | + +Decode is the less intuitive one. Generating one token does one or two floating-point operations per byte read from memory, depending on weight precision. An H100 needs about 295 operations per byte to stay busy. So a single request uses a small fraction of the chip while the compute units wait on memory ([Data Gravity](https://www.datagravity.dev/p/the-rise-of-inference-engineering)). + +The gap is growing. Peak compute has grown about 3x every two years, against about 1.6x for memory bandwidth. Nearly every optimization does one of two things: move fewer bytes per token, or spread each byte across more tokens. + +Because the phases have opposite bottlenecks, a technique that speeds one often does little for the other. That is why benchmarks report TTFT and tokens per second separately. + +### The metrics you will live with + +| Metric | What it measures | Watch out for | +|-----------------------------------------------|------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------| +| Time to first token (TTFT) | Queueing plus prefill: how long the user waits before anything appears | With reasoning models, the engine's first token may be hidden thinking. Measure the first token the user sees. | +| Time per output token, or inter-token latency | Decode speed per stream | Speculative decoding and memory bandwidth show up here | +| Tokens per second per user | Interactivity, or the model's typing speed | Falls as batch size rises | +| Throughput | Total tokens per second across all users | Rises as batch size rises | +| Goodput | Throughput that still meets your latency targets | The number to plan capacity with | +| End-to-end latency | Total request time | Depends on how much the model writes. A healthy 190-second answer is normal for long generations. | +| Percentiles | P50, P90, P99 of each latency | Users feel the tail, not the average | +| Errors and refusals | Failed or rejected requests | Count refusals as errors, or a context cap can look like a speedup | +| Cost per million tokens | GPU cost divided by tokens served | Compare at the load you actually run | + +Cost per million tokens converts a GPU price and a throughput into one comparable number: + +$$ +\text{Cost per million tokens} = \frac{\text{GPU cost per hour}}{\text{tokens per second} \times 3600} \times 10^6 +$$ + +[Spheron's](https://www.spheron.network/blog/inference-engineering-guide-2026/) worked example: a GPU at \$2.90 an hour sustaining 1,200 tokens per second costs about \$0.67 per million tokens. + +### Latency versus throughput + +A GPU is good at doing many things at once and ordinary at doing one thing quickly. Batching more requests together raises total throughput and slows each individual user. Every product picks a point on that curve. Interactive chat and coding favor latency. Overnight batch jobs favor throughput. + +It is the same choice you make when tuning a database for transactions or for analytics. You cannot optimize for both at once, so start from the workload and write its constraints down. The more specific they are, the easier every later trade-off becomes. + +### When each concept starts to matter + +The table below turns that question into signals you can watch for. Each row names the symptom that tells you it is time to act, in roughly the order teams meet them. + +| Concept | What it does | The signal that you need it | +|----------------------------------------|-------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------| +| Fitting the weights | Choose a model, precision and GPU so the weights fit with room to spare | The engine fails to start, or starts with almost no KV cache left | +| Continuous batching and paged KV cache | Share each GPU pass across requests, and hand out cache in small blocks | Already on by default in vLLM and SGLang. Check it, don't build it. | +| Quantization | Store weights in fewer bits, and give the freed memory to the KV cache | Requests queue while the GPU still has compute to spare | +| Prefix caching | Reuse the work already done on repeated prompt openings | Already on by default in vLLM and SGLang. Watch the hit rate once your clients are agents or chat apps that resend long histories | +| Speculative decoding | Draft several tokens and verify them in one pass | Individual responses stream too slowly, even at low load | +| Gateway and control plane | Route between replicas, and manage engines declaratively | You add a second model, a second replica or a second cluster | +| Sharding across GPUs | Split one model with tensor, pipeline or expert parallelism | The model will not fit on one GPU. Plan this before you deploy. | +| Autoscaling | Add and remove replicas on queue depth and KV cache use | Idle GPUs show up on the bill, or queues build at peak hours | +| Disaggregation | Run prefill and decode on separate GPUs | Long prompts stall streaming at scale, and you have RDMA-class networking | + +Most teams need only the first few rows on day one. Treat the rest as a backlog that the signals pull forward, not a checklist to finish before launch. + +### What agents change + +Coding agents and other agentic workloads stress every part of this picture. Upbound measured this in [Part 5](https://www.upbound.io/blog/drinking-our-own-champagne-part-5-what-does-it-take-to-grow-a-self-hosted-coding-model-from-pilot-to-team) of its Project Champagne series (introduced in Section 1), where about 10 of its engineers ran Claude Code and other coding agents against a self-hosted model. Their traffic looked like this: + +- **Prompts are long.** The median prompt was 59K tokens and the P90 was 125K. + +- **Prompts are mostly repeated.** About two-thirds of prompt tokens hit the prefix cache, because agents resend their transcript on every turn. + +- **Thinking is hidden.** A reasoning model can think for more than a minute before its first visible token. Every timeout between the agent and the model has to allow for that. + +- **Load depends on the client.** Short-prompt sessions launch subagents and can keep more requests in flight than long-prompt sessions. Plan capacity in requests in flight rather than seats. + +- **Sessions run for hours.** Streams, tool calls and credentials all have to survive that long. + +If your inference platform works for agents, it will work for most other clients. + +## 3. Running inference in production + +With the concepts in place, the next question is how to run them in production. Every choice trades latency, throughput, cost and quality against each other, usually under GPU scarcity. The practices below draw on the sources cited throughout. Where one comes from Upbound’s Project Champagne, the specifics are in the case study in Section 5. + +### 3.1 Start from use cases, not hardware + +Start from the workloads, not the GPUs. Each use case writes different requirements: + +- **Coding agents** needed long context without quality loss, reliable tool-call parsing, streaming, sessions that last hours, and an Anthropic-compatible API for Claude Code. + +- **Chat** needed low time to first token, real concurrency, and an OpenAI-compatible API for existing tools. + +- **The platform** needed TLS on a real hostname, identity tied to the company domain, observability that tells a healthy engine from a busy one, and correct placement without babysitting. + +Separate interactive workloads from batch workloads. They need different targets, and often different deployments. Write the pass and fail bar down before you test anything. + +### 3.2 Pick models deliberately + +- **Evaluate on your own tasks.** Public benchmarks measure someone else's workload. Section 3.11 covers how. + +- **Start small, then grow.** A small MoE model with a few billion active parameters builds operating experience faster and costs less to get wrong. + +- **The newest model is not always the right one.** Model releases move faster than hardware plans. A new ‘small’ flagship can need several of the largest cards, while your current model may be running at half its supported context. The cheapest gain may need no new model at all. + +- **Check that your engine can actually load it.** A model listed as supported in release notes may still fail to load on your hardware and engine build. + +- **Check the license.** Most open models use permissive licenses, but some restrict commercial use. + +- **Check the chat template and parsers.** Tool calls and reasoning output depend on settings each model needs in the engine. A wrong one breaks tool use while every request still returns 200. + +### 3.3 Size for runtime, not startup + +A model that fits at startup can still run out of memory under real traffic, often days later and mid-request, when fragmentation pushes past the profiled peak. + +- **Budget all three costs:** weights, KV cache, and runtime overhead such as CUDA graphs and activations. + +- **Know what your engine's memory setting covers, and check again on every upgrade.** vLLM sizes its KV cache from a startup estimate of everything else, including CUDA graph pools, and real traffic can go past that estimate. If you see out-of-memory errors under load, lower the memory fraction slightly, enable expandable segments and cap concurrent sequences, then measure free memory again. + +- **Cap every request.** Set a maximum output length on every call. Without one, a single request can take a model down for everyone, and abandoned requests keep generating after the client gives up. + +- **Treat context length as a lever, carefully.** A shorter maximum context frees KV cache for more users. It also rejects every longer conversation, so count those rejections as errors. + +### 3.4 Know the optimization techniques + +Most engines turn on continuous batching and paged KV cache by default. The rest are choices you make per model and workload. + +| Technique | What it improves | Cost or risk | Use it when | +|-----------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------| +| Continuous batching | Throughput. [Orca](https://www.usenix.org/conference/osdi22/presentation/yu) reported 36.9x over a static baseline at the same latency. | Per-user speed drops as batches grow | Always. It is the default in modern engines. | +| Paged KV cache ([PagedAttention](https://arxiv.org/abs/2309.06180)) | Concurrency. vLLM cut KV cache waste from 60 to 80% to under 4%. | Little in practice | Always. It is the default in vLLM. | +| Prefix caching | TTFT and prefill cost on repeated openings | Only helps up to the first token that differs | System prompts, RAG, multi-turn chat and agents | +| Weight quantization | Decode always, and prefill only when activations are quantized too, as with FP8. Weight-only formats such as AWQ and GPTQ speed up decode alone. One step down in precision is typically 30 to 50% faster (Inference Engineering). | Quality loss, which you must measure | Use the lowest precision your hardware runs natively and your evals accept. FP8 runs natively on Ada, Hopper and Blackwell GPUs. | +| KV cache quantization | More cache fits, and it moves faster | Errors compound from token to token | When KV cache is the limit | +| [Speculative decoding](https://arxiv.org/abs/2211.17192), multi-token prediction | Tokens per second at low batch sizes | Wasted compute when guesses are rejected. Its value shrinks as load rises and spare compute runs out. | Latency matters and output is predictable | +| [Tensor parallelism](https://arxiv.org/abs/1909.08053) | Fits large models, faster per user | Needs fast GPU-to-GPU links such as NVLink | Large models within one node | +| Expert parallelism | Throughput for [MoE](https://arxiv.org/abs/1701.06538) models | Routing between experts | MoE models, including across nodes | +| Pipeline parallelism | Fits a model across nodes on ordinary networking | Adds latency to each pass | The model does not fit in one node | +| Prefill and decode disaggregation | Removes interference between phases. [DistServe](https://arxiv.org/abs/2401.09670) reported 7.4x more requests within latency limits. | Needs RDMA-class networking to move the KV cache | Large scale, long prompts, fast interconnect | + +A few rules of thumb: + +- **Structure prompts for the cache.** Prefix caching stops at the first token that differs. Put shared content such as system prompts and tool definitions first, and variable user input last. Hosted APIs price cached input at 10 to 50% of fresh input, as of 2026, for the same reason ([Data Gravity](https://www.datagravity.dev/p/the-rise-of-inference-engineering)). + +- **Quantize from least to most sensitive.** Linear weights tolerate quantization best, then activations, then the KV cache. Attention layers are the most sensitive, because small errors compound across a long sequence. Keep the most sensitive parts at higher precision until your evals show you can go lower. Floating-point formats such as FP8 handle outlier values better than integer formats. + +- **Watch speculative decoding acceptance.** Acceptance depends on temperature, subject matter and the draft model. If acceptance collapses, every pass still pays for the guesses and the model slows while other panels look normal. + +- **Beware interactions when applying multiple techniques.** Quantizing the KV cache helps disaggregation. Larger batches leave less spare compute for speculation. [One Baseten engineer tried 77 configurations](https://newsletter.pragmaticengineer.com/p/what-is-inference-engineering) before finding one that doubled throughput for a customer (Inference Engineering). Change one thing at a time and measure (Section 3.10). + +- **Keep long prompts from stalling other streams.** Chunked prefill splits a long prompt across several passes, so decode keeps moving for everyone else. Current vLLM turns it on by default, and its token budget per pass is the setting to tune before you consider disaggregation. + +### 3.5 Get GPUs, and place them correctly + +- **Capacity is not quota**. Quota is permission to ask. Each instance size is its own capacity pool, even with the same GPU, and no cloud API tells you what you can actually launch. + +- **Design for a shape that can change.** Expect to get the GPU you planned for only some of the time. A model may end up split across smaller nodes in different zones, using pipeline parallelism over ordinary networking. Allow multi-zone pools, and keep the model definition independent of the node shape. + +- **Match models to hardware explicitly.** Kubernetes Dynamic Resource Allocation (DRA) device selectors can pin a model to a GPU architecture or a minimum memory size. Verify placement against what the GPU driver publishes, not against a status field that echoes what you declared. + +- **Watch the cloud, not just the controller.** A managed node group can report failure while the cloud keeps trying to launch instances, and GPUs that arrive hours later can be terminated by a later change to the pool. Track instances and node joins, and list a pool's instances before you change it. + +- **Know your interconnect.** SXM datacenter GPUs such as H100, H200 and B200 link GPUs with NVLink. PCIe cards such as the L40S and RTX PRO 6000 do not, so multi-GPU communication goes over PCIe and host memory. That changes which parallelism strategies pay off. + +- **Size the system pool too.** GPU nodes are tainted for inference. Monitoring, controllers and gateways compete for whatever general capacity is left. + +### 3.6 Gateways, routing and timeouts + +- **Do not round-robin.** Requests are not interchangeable. One replica may already hold a conversation's prefix in cache while another has never seen it. Use cache-aware or prefix-aware routing, such as the [Gateway API Inference Extension](https://github.com/kubernetes-sigs/gateway-api-inference-extension)'s endpoint picker. + +- **Route by model name.** A single hostname can serve many models, routed on the request's model field. + +- **Speak both common APIs.** Most tools expect the OpenAI API. Claude Code expects the Anthropic Messages API, which [vLLM](https://github.com/vllm-project/vllm) serves natively, including streaming and tool calls. + +- **Raise every timeout on the path.** Long generations and reasoning models break the defaults, because each hop has a default timeout sized for web requests, not for generations that run for minutes. Typical culprits are a gateway route timeout that bounds the whole response, a load balancer idle timeout, and retries that multiply the wait. Check the route, each load balancer, the gateway, the client and any helper calls the client makes. + +- **Detect truncated streams.** A cut stream can return HTTP 200 with valid tokens and simply stop. To the client, it looks like a finished answer. + +- **Meter usage per caller.** Token-based rate limits and quotas at the gateway keep one heavy user or runaway agent from starving everyone else. + +### 3.7 Security and identity + +- **Never ship an open endpoint.** Apply your auth policy before the gateway gets an address, and remove it last when you tear down. + +- **Use real TLS on a real hostname,** with a plan for certificate issuance and renewal. + +- **Plan for three kinds of client.** People are easy. Browsers cannot attach a bearer token to page loads or preflight requests, so handle browser login with OIDC at the gateway. Agents run for hours and read their credential once, so a 60-minute identity token will kill them mid-session. Give agents long-lived API keys or a credential that refreshes. + +- **Do not stack two expiring sessions.** An outer gateway login in front of an app with its own login causes an outage when the outer session expires. + +- **Test what should be refused.** Keep a small probe matrix: no token, a garbage token, a key you did not sign, an expired key, a revoked key, a forged caller header. Include what should stay open, such as the health check. Run it on a schedule and in a loop during changes. + +- **Carry identity into usage records,** so you can see who sends what. Overwrite any caller header a client tries to forge. + +- Treat model weights like code. Prefer safetensors over pickle-based checkpoints, don't enable remote code from a model repo you haven't read, and pin each model to a specific revision. + +### 3.8 Observability + +- **Scrape the engines.** Engines expose detailed Prometheus metrics. Nothing collects them unless something is set up to scrape them, such as a ServiceMonitor and a named port. + +- **Build the core panels:** requests running against the concurrency limit, requests waiting, KV cache use, TTFT, time per output token, output tokens per request, prefix cache hit rate, speculative decoding acceptance, errors by status code, and cancellations. + +- Measure quality on the server. Broken tool calls, cut-off answers and empty responses usually reach you as user complaints while the serving dashboards look normal. Count them on the server instead, including tool calls that fail to parse, responses cut off by the length limit, empty outputs, refusals and hidden reasoning tokens per request. When the engine doesn't export one of these, the gateway sees every response and can count it. + +- **Look past GPU "utilization."** It reports whether anything ran, not how much of the chip worked. A GPU can report near 100% “utilization” while SM activity is around half and tensor cores sit mostly idle. Graph SM activity and tensor activity as well. + +- **Learn the shapes of "slow."** Queued: running is at the limit, waiting is above zero, and the KV cache is full. Thinking: TTFT looks fine but each request writes many hidden tokens. Genuinely slow: time per output token is up. + +- **Watch the watcher.** Something outside the monitoring stack must check that data is fresh and that new metrics arrive from every cluster. A metrics adapter can report its API as available while holding no data. + +- **Keep the evidence.** Metric scrapes can make up nearly all of a retained engine log. Keep per-request access logs with the caller, ship them to a log store, and remember kubectl logs --previous after a crash. + +- **Normalize metric names across engines,** so a vLLM deployment and an SGLang deployment are judged on the same signals. + +Set SLOs on what users feel, measured per request at the front door: + +- **Responsive:** time to the first token the user actually sees, under a threshold. + +- **Fast enough:** time per output token, under a threshold. + +- **Answered:** the request completed without an error, and any tool call in it parsed. + +Do not set an SLO on total request time, because it depends on how much the model writes. Low-traffic models need a fleet-wide SLO, longer windows or synthetic traffic. Missing data reads as 100% good, so keep a separate alert for metrics that stop arriving. Open source tools such as [Sloth](https://github.com/slok/sloth) and [Pyrra](https://github.com/pyrra-dev/pyrra) turn SLO definitions into burn-rate alerts. For background, see [*Observability Engineering*](https://www.oreilly.com/library/view/observability-engineering/9781492076438/). + +### 3.9 Autoscaling and cold starts + +- **Scale on inference signals,** such as queue depth, KV cache use and requests in flight. CPU usage tells you almost nothing here. + +- **Combine traffic and utilization.** Traffic lets you scale ahead of demand, and utilization lags. The knobs are the ones you already tune on any autoscaler: a floor and a ceiling, how far back the window looks, how long to wait before scaling down, and how many concurrent requests one replica should hold. [Kiely's book](https://www.baseten.co/inference-engineering/) covers each in depth for inference. + +- **Respect the cold-start wall.** A new GPU node must boot, pull a large engine image and load the weights. Loading weights alone can take half an hour for a large model. Cache weights on shared storage in each cluster so new replicas do not download them again. + +- **Scale-to-zero is harder than it sounds.** By the time a node is ready, the spike may be over, and scarce GPU capacity may not come back. + +- **Plan for multiple clusters at scale.** Past a few hundred GPUs, capacity pushes workloads across regions and providers. A control plane that treats separate pools as one avoids stranded capacity. + +### 3.10 Change management + +- **Track a three-way version matrix:** model, hardware and engine build. New models often need a nightly or model-specific engine build on day zero. + +- **Plan rollouts for exclusive GPUs.** A GPU claim is exclusive, so a rolling update on a single replica deadlocks: the new pod cannot schedule and the old one never leaves. Use a Recreate strategy or allow one unavailable, and budget the downtime. + +- **Ready is not routable.** A gateway can lag a ready replica by a minute or two. + +- **Test serving changes like code.** Make each change a pull request and change one thing at a time. Write the bar down first. Sweep load upward until a step misses the bar, and report capacity as one number with a confidence interval. + +- **Test the test rig first.** A rig bug can waste a scarce GPU window that a few minutes on a laptop would have caught. + +- **Test on your own hardware with traffic like yours.** Agent prompts are long and mostly cached. A benchmark of short, fresh prompts measures a different system. + +Promote every serving change through tiers. Each tier costs more than the one before and catches a different kind of failure, so most problems show up before users see them: + +- **Offline.** Validate the config and run the test rig's own checks. This takes minutes on a laptop and no GPU. + +- **Rehearsal.** Run the full test end to end against a fake gateway or a small model, so rig bugs don't cost a GPU window. + +- **Smoke.** Send a few real requests on the target hardware: a long prompt, a tool call, a streamed answer and a request that should be refused. + +- **Load with evals.** Sweep concurrency up to your bar with the eval suite running alongside, and compare against the current version on the same traffic. Evals can pass one request at a time while tool calls break under concurrent load, with nothing moving on the serving dashboards. + +- **Canary.** Send a small share of real traffic to the new version and compare its server-side signals with the current one: TTFT, time per output token, errors, tool-call parse failures and length-limit stops. A gateway that routes by model name can split traffic by weight. Write the rollback trigger down before you start. + +- **Full rollout.** Keep the old version ready to take traffic back until the new one's signals stay within the bar through a peak period. + +### 3.11 Quality and evals + +When an open model scores low, check three places before you blame the model: + +1. **The harness.** Is the model the only thing that changed? Run your current hosted model first. If it scores below a weaker model, the harness is broken. + +2. **The infrastructure.** Did the model get a chance to answer? Timeouts, expired credentials and routing errors show up as model failures. Mark those runs "not measured." + +3. **The grader.** Does the check measure the outcome you care about? Grade the end state, not the agent's description of it, and test your graders against answers people already accepted or rejected. + +Also: pin the data that decides the right answer, give every model the same context, keep test markers where the model cannot read them, run each test more than once, and prefer deterministic checks over LLM judges. A batching server can return different tokens for the same request even at temperature 0, so expect some variance unless the engine runs in a slower batch-invariant mode. + +### 3.12 Incidents and debugging + +- **Expect silent failures.** The costly problems often throw no errors: a truncated stream returns 200, a metrics API reports available, node groups report ready. Part of the platform’s job is to make those failures loud. + +- **Read the number.** A failure at a round number of seconds is a timeout. A multiple of it, such as 239 seconds after four 60-second tries, points to retries. + +- **Bypass a layer.** Take one hop out of the path and see whether the problem moves. + +- **Verify status against reality.** A status is one component's view of its own job. Check the disk, the instance or the route itself. + +- **Work additively, and upstream the fix.** Keep workarounds beside the platform, labelled for teardown, rather than editing objects a controller owns. A forked fix helps nobody else. + +### Checklist before your first users + +- [ ] Use cases defined, with a written bar for time to first token, speed per user and error rate + +- [ ] Model chosen on your own evals, with your current hosted model as the baseline + +- [ ] Engine build confirmed to load the model on your hardware + +- [ ] Memory budget covers weights, KV cache and runtime overhead, with max output tokens and max concurrent sequences set + +- [ ] GPU placement verified against the driver, with a fallback instance shape or zone + +- [ ] Auth policy applied before the gateway gets an address, and TLS on a real hostname + +- [ ] Credentials that last as long as an agent session + +- [ ] Timeouts raised at every hop: route, load balancers, gateway retries, client and helper calls + +- [ ] Engine metrics scraped, GPU SM and tensor activity graphed, and an alert when metrics stop arriving + +- [ ] Server-side counts of tool-call parse failures, length-limit stops and empty responses, so users aren't your first alert + +- [ ] A probe matrix of requests that must be refused and paths that must stay open + +- [ ] A rollout strategy for exclusive GPUs, with downtime budgeted + +- [ ] Previous container logs preserved after a crash + +- [ ] Serving changes promoted through tiers, with evals under concurrent load and a canary with a written rollback trigger + +- [ ] Workarounds kept beside the platform, labelled for teardown, with fixes sent upstream + +## 4. The stack + +Learn the engine layer deeply. Learn the Kubernetes, gateway and observability layers you already know in their inference form. Learn the kernel layer well enough to read a profile. Most inference engineering happens at the higher levels of abstraction, but a working model of the layers next to yours guides every decision. + +### Tools by layer + +| Layer | Tools to know | Master first | +|------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------| +| GPUs on Kubernetes | [NVIDIA GPU Operator](https://github.com/NVIDIA/gpu-operator), [Dynamic Resource Allocation](https://kubernetes.io/docs/concepts/scheduling-eviction/dynamic-resource-allocation/) and its [NVIDIA driver](https://github.com/NVIDIA/k8s-dra-driver-gpu), [DCGM exporter](https://github.com/NVIDIA/dcgm-exporter) | How GPUs are advertised, requested and placed | +| Kernels and frameworks | [CUDA](https://developer.nvidia.com/cuda-toolkit), [PyTorch](https://pytorch.org), [Triton](https://github.com/triton-lang/triton), [FlashAttention](https://github.com/Dao-AILab/flash-attention), torch.compile | What a kernel is, kernel fusion, and how to read a profile | +| Inference engines | [vLLM](https://github.com/vllm-project/vllm), [SGLang](https://github.com/sgl-project/sglang), [TensorRT-LLM](https://github.com/NVIDIA/TensorRT-LLM); [Ollama](https://github.com/ollama/ollama) and [llama.cpp](https://github.com/ggml-org/llama.cpp) for local work | vLLM's core flags: memory utilization, max sequences, max model length, tensor parallel size, quantization, prefix caching, chunked prefill | +| Distributed serving | [NVIDIA Dynamo](https://github.com/ai-dynamo/dynamo), [llm-d](https://github.com/llm-d/llm-d), [KServe](https://github.com/kserve/kserve), [LeaderWorkerSet](https://github.com/kubernetes-sigs/lws) | Multi-GPU and multi-node layouts, and prefill and decode split | +| Gateway and routing | [Envoy Gateway](https://github.com/envoyproxy/gateway), [Agent Router](https://theagentrouter.ai) (formerly Envoy AI Gateway), [Gateway API Inference Extension](https://github.com/kubernetes-sigs/gateway-api-inference-extension) | Streaming, timeouts, auth policies, model-name and cache-aware routing | +| Control plane | [Modelplane](https://github.com/modelplaneai/modelplane), built on [Crossplane](https://github.com/crossplane/crossplane) | Declaring clusters, hardware classes, deployments and caches across environments | +| Observability | [Prometheus](https://prometheus.io), [Grafana](https://github.com/grafana/grafana), [OpenTelemetry](https://opentelemetry.io), [Loki](https://github.com/grafana/loki), [Sloth](https://github.com/slok/sloth) or [Pyrra](https://github.com/pyrra-dev/pyrra) | Engine metrics, real GPU activity, and SLOs on user-facing latency | +| Benchmarking | [vLLM's benchmark tools](https://github.com/vllm-project/vllm), [NVIDIA AIPerf](https://github.com/ai-dynamo/aiperf), [GuideLLM](https://github.com/vllm-project/guidellm), [SemiAnalysis AgentX](https://inferencex.semianalysis.com/agentx/methodology) traces | Concurrency sweeps and latency percentiles under realistic prompts | +| Evals | [Inspect](https://inspect.aisi.org.uk) from the UK AI Security Institute; Hugging Face upskill for testing skills across models | Deterministic graders on the end state of a task | +| Models | [Hugging Face Hub](https://huggingface.co/models), [safetensors](https://github.com/huggingface/safetensors) checkpoints | Reading a model config for layers, heads and context, and checking real file size | + +### Choosing an engine + +The engine controls batching, memory management, kernel selection and API compatibility. It often matters more to cost and throughput than a hardware upgrade. + +| Engine | Why teams pick it | What it costs to operate | +|--------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------| +| [vLLM](https://github.com/vllm-project/vllm) | The broadest model coverage, plus OpenAI and Anthropic APIs out of the box, so it is the usual first deployment. Hosted by the PyTorch Foundation. | Frequent releases with new flags. Pin versions and re-benchmark on every upgrade. | +| [SGLang](https://github.com/sgl-project/sglang) | A radix-tree prefix cache and a scheduler that favors requests sharing a cached prefix, which suits agents and long multi-turn sessions. Upbound measured its biggest single gain here (Project Champagne, Part 5, 2026). | A smaller community, and new models may need nightly builds before a tagged release works. | +| [TensorRT-LLM](https://github.com/NVIDIA/TensorRT-LLM) | Kernels tuned for NVIDIA hardware, including FP4 on Blackwell | NVIDIA hardware only, which ties your serving stack to one vendor. | +| [NVIDIA Dynamo](https://github.com/ai-dynamo/dynamo) | A layer above engines that coordinates many nodes, including split prefill and decode | Another distributed system to run. Worth it only at multi-node scale. | +| [Ollama](https://github.com/ollama/ollama), [llama.cpp](https://github.com/ggml-org/llama.cpp) | Easy local serving for learning and prototyping | Not designed for many concurrent users | + +Engines rise and fade. Hugging Face's TGI was a common choice and is now in maintenance mode. Check release cadence, maintainers and foundation backing before you commit to one. Engines ship new flags every few weeks. Re-run your benchmarks when you upgrade, rather than trusting last quarter's numbers. + +Upbound's own comparison in [Project Champagne, Part 5](https://www.upbound.io/blog/drinking-our-own-champagne-part-5-what-does-it-take-to-grow-a-self-hosted-coding-model-from-pilot-to-team) is a good example. The team was serving a coding model to about 10 engineers on two 8-GPU nodes and tested how to grow to a team of 15. On the same hardware and the same agent traffic, switching from vLLM to SGLang was the largest single gain it measured. Adding SGLang's host-memory cache on top added nothing measurable, because the KV cache never filled. + +### Where Modelplane fits + +[Modelplane](https://github.com/modelplaneai/modelplane) is an open source project from Upbound, built on [Crossplane](https://github.com/crossplane/crossplane) and released under the Apache 2.0 license. It orchestrates models, inference engines and AI infrastructure across clouds, neoclouds and your own GPUs, with Kubernetes-style declarative APIs. It provisions inference clusters or uses ones you bring. Documentation lives at [docs.modelplane.ai](https://docs.modelplane.ai). + +Modelplane splits the work into three layers of ownership. This split is a useful mental model even if you use different tools. + +| Who | Owns | Modelplane resources | +|---------------------------------------|-------------------------------------------------------|----------------------------------------------------| +| Platform team | The fleet: clusters, hardware classes and the gateway | InferenceCluster, InferenceClass, InferenceGateway | +| Model operator, or inference engineer | The models: deployments, services and weight caches | ModelDeployment, ModelService, ModelCache | +| Modelplane | Everything in between, composed from the two above | ModelEndpoint, ModelReplica | + +The hierarchy mirrors core Kubernetes one scope up. ModelDeployment → ModelReplica → ModelService → ModelEndpoint parallels Deployment → Pod → Service → Endpoint, across a fleet instead of within a single cluster. You bring the engine container and its flags, and Modelplane shapes the topology around them. People and agents never create any of these. They call the InferenceGateway by model name. + +Modelplane maps vLLM and SGLang metrics onto common names out of the box, and a MetricMapping adds any other engine, so different engines are judged on the same signals. + +For a platform engineer, the second layer is where the new learning is. Upbound found the same thing when it started self-hosting in [Project Champagne, Part 1](https://www.upbound.io/blog/drinking-our-own-champagne-what-it-takes-to-own-your-intelligence): the cluster and gateway work was familiar, and the model operator layer held the unknowns. + +### A suggested order to learn the stack + +1. Run vLLM on one GPU behind its OpenAI-compatible API, and read its metrics endpoint. + +2. Deploy it on Kubernetes with the NVIDIA GPU Operator, and place it with device selectors. + +3. Put a gateway in front of it with TLS, auth and timeouts sized for long streams. + +4. Scrape engine and GPU metrics into Prometheus and build the core panels from Section 3.8. + +5. Benchmark it with a concurrency sweep, and run a small eval suite at the same time. + +6. Serve the same model on SGLang and compare on your own traffic. + +7. Split a larger model across GPUs with tensor and pipeline parallelism, then try a distributed serving layer. + +8. Learn enough CUDA and profiling to understand why a kernel or a setting made the difference. + +## 5. Learn from teams that share their work + +Much of what you need to know about running inference isn’t in the docs. It’s in write-ups from teams who ran it and shared what broke. Reading those cases well is a skill worth building, and writing your own is one of the strongest ways to show what you know (Section 6). + +### How to read a case study + +- **Find the setup.** Model, hardware, engine version and traffic shape. Note the date, because numbers age fast. + +- **Find the bar.** What counted as good enough, and who decided. + +- **Follow the failures.** What broke, how the team found out, and how long it stayed hidden. + +- **Check the method.** Did they change one thing at a time and measure it on their own traffic? + +- **Separate what transfers.** A timeout lesson applies everywhere. A specific flag value applies to that engine release on that hardware. + +#### Case study: Project Champagne (2026) + +Upbound, the company behind Crossplane and Modelplane, self-hosted chat, coding agents and a CVE-fixing build agent on open models with Modelplane, across AWS and Nebius GPUs. The team published what broke in a five-part series. Highlights, with numbers as of 2026: + +- **Serving was the easy part.** vLLM and scheduling worked. TLS, identity, timeouts, storage and observability took the time. (Part 2) +- **Quota did not mean capacity.** Approved quota for 768 vCPUs of H100 instances never produced one, and the planned GPU arrived about half the time. One model ended up split across two nodes in two zones. (Part 2) +- **“It fits” was a startup-time claim.** A chat model that fit cleanly ran out of memory nine times in four days. Lowering the memory fraction from 0.92 to 0.88, enabling expandable segments and capping concurrent sequences at 16 freed 4.6 GiB. (Part 2) +- **The expensive failures were silent.** A truncated stream returned 200, a metrics API reported available with no data, and monitoring sat pending for six days. (Part 2) +- **Timeouts stacked up.** A 15-second route default, then a 60-second load balancer idle timeout, cut off answers that were still streaming. (Parts 2 and 3) +- **Agents were the hardest clients.** Hour-long sessions outlived 60-minute tokens, and a reasoning model thought for more than a minute before its first visible token. (Parts 2 and 3) +- **Low scores were often not the model’s fault.** A fake shell in the test harness scored Claude 0. A 60-second idle timeout failed a model that completed every run once the limit was raised. (Part 3) +- **“Slow” had shapes.** An engineer’s “wow, is it slow” traced back to hidden maximum-effort reasoning while every dashboard looked fine. One GPU showed 99% “utilization” with tensor cores busy about 10% of the time. (Part 4) +- **Load depended on the client.** Agent prompts had a median of 59K tokens, about two-thirds hit the prefix cache, and short-prompt sessions kept more requests in flight by launching subagents. (Part 5) +- **The engine mattered more than expected.** On the same two 8-GPU nodes, moving from vLLM to SGLang was the largest single gain. A 128K context cap only looked faster because it rejected long conversations. (Part 5) + +### Read the full series + +Each post on the Upbound blog tells these stories in detail, with the configs, queries and numbers behind them: + +- [**Part 1: What it takes to own your intelligence.**](https://www.upbound.io/blog/drinking-our-own-champagne-what-it-takes-to-own-your-intelligence) Why Upbound self-hosts inference, and how the use cases picked the models and hardware. + +- [**Part 2: Finding the corks.**](https://www.upbound.io/blog/drinking-our-own-champagne-part-2-finding-the-corks) The inference lifecycle stage by stage, from pick and place to change and fail. + +- [**Part 3: The build security agent.**](https://www.upbound.io/blog/drinking-our-own-champagne-part-3-use-case-1-the-build-security-agent) How to tell whether an open model can do a real job as well as a hosted one. + +- [**Part 4: Candling the bottles.**](https://www.upbound.io/blog/drinking-our-own-champagne-part-4-candling-the-bottles) What the team could see at each step, what it missed, and how it plans to alert. + +- [**Part 5: From pilot to team.**](https://www.upbound.io/blog/drinking-our-own-champagne-part-5-what-does-it-take-to-grow-a-self-hosted-coding-model-from-pilot-to-team) A repeatable way to test serving changes and size capacity for a growing team. + +## 6. Career advice: making the move + +You do not need to start over. Keep your platform skills, add the inference concepts, and prove both in public. The field is young enough that newcomers can become experts quickly. + +### Pick a direction + +Pick the direction closest to the work you already do. Most platform engineers start in the first two and grow toward the third. + +| Direction | Focus | Good fit if you | +|-----------------------|-------------------------------------------------------------------------|--------------------------------------------------| +| Inference platform | Clusters, GPU scheduling, gateways, autoscaling, multi-cloud capacity | Enjoy Kubernetes and distributed systems | +| Inference reliability | SLOs, observability, capacity planning, rollouts, incident response | Have carried a pager and run SRE practices | +| Runtime performance | Engine tuning, quantization, speculative decoding, parallelism, kernels | Like profiling and working close to the hardware | + +### Follow a learning path + +Vishakha Sadhwani's *AI Inference Engineering Roadmap 2026* is a thorough 13-stage map of the field. Coming from platform engineering, you can move through it faster, because you already own much of the infrastructure half. A shorter path that plays to that head start: + +1. **Learn the model, not just the container.** Transformer basics, attention and the KV cache, dense versus MoE models, and how to read a model config. The goal is to estimate the memory any model needs in five minutes. + +2. **Learn one engine deeply.** Pick vLLM, learn what each core flag trades away, and measure time to first token and per-token latency yourself. + +3. **Translate your platform skills to GPUs.** Device allocation, gateways that hold long streams, autoscaling on inference signals, and observability for engines. This is mostly translation, not new learning. + +4. **Learn to judge quality and cost.** Build an eval, run a capacity sweep and calculate cost per million tokens. This is where platform engineers most often have a gap, and where the role earns its keep. + +### Build projects that prove it + +Each project below fits on a rented GPU or two, and each produces a repo with real numbers. Shut instances down between runs, and use spot capacity for batch experiments. + +| Project | What it proves | What to publish | +|---------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------|--------------------------------------------------------------| +| Serve an 8B model with vLLM on one GPU. Sweep 1, 10 and 50 concurrent users. | You can measure TTFT, time per output token and throughput honestly | Configs, the load script, charts and cost per million tokens | +| Predict how many 32K-token sessions fit on your GPU, then test it | You understand weights, KV cache and runtime overhead | Your prediction, the result and why they differ | +| Compare FP16 and FP8, and a 4-bit format if you can | You can trade quality against speed with evidence | Eval scores next to latency and throughput | +| Run the same prompts with shared content first, then last | You understand prefix caching and prompt structure | Cache hit rate and TTFT for each | +| Deploy on Kubernetes with the GPU Operator, a gateway with TLS and auth, autoscaling on queue depth, and dashboards | You can run inference as a platform | Manifests or compositions, and dashboard definitions | +| Compare vLLM and SGLang on agent-like traffic, one change at a time | You make engine choices with data | The bar, the method and results with intervals | +| Build an eval harness for one agent task with deterministic graders | You can tell a model failure from a harness failure | Tasks, graders and negative controls | +| Reproduce a failure: a timeout, an out-of-memory crash or a stuck rollout | You can operate it, not just deploy it | A short postmortem and runbook | + +### Show your work in public + +Many inference roles fill through networks before they reach a careers board. In August 2026, [Zeoli found](https://www.datagravity.dev/p/the-rise-of-inference-engineering) one serving company valued at \$17.5B listing no dedicated inference performance openings. Public work is how people outside those networks get found. + +- **Write READMEs like lab reports.** State the question, the setup (model, GPU, engine version and flags), the method, the results with numbers, what surprised you, and how to reproduce it. + +- **Publish what failed.** The Champagne posts are useful precisely because they show the corks. A clear account of a failure shows more skill than a clean demo. + +- **Pin versions and show cost.** Results without an engine version and a price are hard to trust and harder to reuse. + +- **Write and speak about it.** Turn each project into a short post. Submit talks to meetups, CNCF events and community live streams. + +### Contribute to open source + +Much of what inference projects still need is platform work, which plays to your strengths. Upbound's own upstream list from Project Champagne reads like a platform backlog: configurable load balancers, certificate issuance, Recreate rollouts for exclusive devices, an honest Ready signal, engine metrics scraped out of the box, and auth that suits long-running agents. + +Projects worth watching include [vLLM](https://github.com/vllm-project/vllm), [SGLang](https://github.com/sgl-project/sglang), [llm-d](https://github.com/llm-d/llm-d), [KServe](https://github.com/kserve/kserve), [NVIDIA Dynamo](https://github.com/ai-dynamo/dynamo), [Agent Router](https://theagentrouter.ai) (formerly Envoy AI Gateway), the [Gateway API Inference Extension](https://github.com/kubernetes-sigs/gateway-api-inference-extension), [Modelplane](https://github.com/modelplaneai/modelplane) and [Crossplane](https://github.com/crossplane/crossplane). Good first contributions: + +- **Run it and report what breaks,** with a reproduction and numbers. This is how Champagne contributed most. + +- **Write operator documentation:** deployment, tuning and troubleshooting guides drawn from your own runs. + +- **Add metrics, dashboards and alerts** that make silent failures loud. + +- **Improve manifests, Helm charts and compositions** for real environments. + +- **Contribute benchmark recipes and tests** for the hardware you have. + +Read each project's contributing guide, start with a small change, and comment on an issue before taking on a large one. + +### Keep your skills current + +This field moves faster than most. Engines add flags every few weeks, and research reaches production within months. + +- **Read release notes** for the engine you run, every release, and re-run your benchmark when you upgrade. + +- **Read the foundational papers:** [Orca](https://www.usenix.org/conference/osdi22/presentation/yu) on continuous batching, [PagedAttention](https://arxiv.org/abs/2309.06180), [DistServe](https://arxiv.org/abs/2401.09670) on disaggregation, the original [speculative decoding](https://arxiv.org/abs/2211.17192) paper, and [FlashAttention](https://arxiv.org/abs/2205.14135). + +- **Read Philip Kiely's [*Inference Engineering*](https://www.baseten.co/inference-engineering/).** The e-book is free from Baseten. + +- **Follow practitioners.** Newsletters such as [The Pragmatic Engineer](https://newsletter.pragmaticengineer.com), [ByteByteGo](https://blog.bytebytego.com) and [Data Gravity](https://www.datagravity.dev) cover the field well. Join the community channels for the engines you use. + +- **Learn evals.** Whether an open model can replace a hosted one is decided by evals, and few engineers can build good ones. Start with Anthropic's [*Demystifying evals for AI agents*](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents) and Hamel Husain and Shreya Shankar's [evals FAQ](https://hamel.dev/blog/posts/evals-faq/). + +- **Learn the economics.** Be able to explain cost per million tokens and cost per user session to a finance partner. + +### Find the roles + +Don't expect a consistent job title yet. The same work shows up as inference engineer, performance engineer, model serving engineer or ML infrastructure engineer, and sometimes simply as AI platform engineer, so search for all of them. Pay reflects how scarce the skills are: in August 2026, [Zeoli found](https://www.datagravity.dev/p/the-rise-of-inference-engineering) an Anthropic inference performance role posted at \$350,000 to \$850,000, and an OpenAI equivalent at \$295,000 to \$555,000. + +You may not need to change jobs to get started, though. If your company is planning to self-host its first model, put your hand up. That project will probably land on your team anyway, and running it is the most direct experience you can get. + +### Prepare for interviews + +Practice answering these. Each one points to the section of this guide that covers it. Several are adapted from Lamhot Siagian's Efficient LLM Inference: Interview Pocket Notes. + +- **Prefill and decode.** Why is decode usually limited by memory bandwidth rather than compute, and what changes at long context? (Section 2) + +- **Hardware trends.** If compute keeps growing faster than memory bandwidth, will inference soon become compute-bound? (Section 2) + +- **Memory math.** How much memory do a model and its KV cache need at a given context length and concurrency? (Section 2) + +- **Goodput.** What is goodput, and why plan capacity with it? (Section 2) + +- **Prefix caching.** Which applications benefit most from prefix caching? (Section 2) + +- **Precision.** You need to serve a 70B model on two 80 GB GPUs. What precision would you choose, and why? (Sections 2 and 3.4) + +- **Quantization.** When does weight-only quantization speed things up, and when does it not? (Section 3.4) + +- **Speculative decoding.** When would you not use speculative decoding? (Section 3.4) + +- **Chunked prefill.** How do long prompts slow other users' streams, and what does chunked prefill change? (Section 3.4) + +- **Parallelism.** Which parallelism strategy would you pick for a given model and GPU type, and when does tensor parallelism beat pipeline parallelism? (Sections 3.4 and 3.5) + +- **Profiling.** How would you tell whether a workload is limited by compute or by memory bandwidth? (Section 3.8) + +- **Debugging "slow".** P99 time to first token just spiked. How do you tell queued, thinking and genuinely slow apart? (Section 3.8) + +- **Cost.** What does a million tokens cost on your GPUs, and how does that compare with a hosted API? (Section 2) + +- **Capacity planning.** Your user count will grow 10x in six months. How do you plan capacity? (Section 3.10) + +- **Rollouts and SLOs.** How would you roll out a new model version safely, and what SLO would you set for the service? (Sections 3.8 and 3.10) + +- **Failure stories.** Tell one failure story with real numbers: what broke, how you found it, and what you changed. + +## 7. Sources and further reading + +### Upbound and Modelplane + +- Upbound, *Drinking Our Own Champagne* series on the Upbound blog: [Part 1: What it takes to own your intelligence](https://www.upbound.io/blog/drinking-our-own-champagne-what-it-takes-to-own-your-intelligence); [Part 2: Finding the corks](https://www.upbound.io/blog/drinking-our-own-champagne-part-2-finding-the-corks); [Part 3: Use case \#1, the build security agent](https://www.upbound.io/blog/drinking-our-own-champagne-part-3-use-case-1-the-build-security-agent); [Part 4: Candling the bottles](https://www.upbound.io/blog/drinking-our-own-champagne-part-4-candling-the-bottles); [Part 5: What does it take to grow a self-hosted coding model from pilot to team?](https://www.upbound.io/blog/drinking-our-own-champagne-part-5-what-does-it-take-to-grow-a-self-hosted-coding-model-from-pilot-to-team) + +- Viktor Farcic, *LLM Inference Explained: 12 Concepts You Actually Need to Know*. + +- [Modelplane on GitHub](https://github.com/modelplaneai/modelplane) and [Modelplane documentation](https://docs.modelplane.ai). + +### Books and articles + +- Philip Kiely, [*Inference Engineering*](https://www.baseten.co/inference-engineering/), Baseten Books, 2026. Free as an e-book. + +- Gergely Orosz & Philip Kiely, [*What is inference engineering? Deepdive*](https://newsletter.pragmaticengineer.com/p/what-is-inference-engineering), The Pragmatic Engineer, March 31, 2026. + +- ByteByteGo, [*A Guide to AI Inference Engineering*](https://blog.bytebytego.com/p/a-guide-to-ai-inference-engineering), June 15, 2026. + +- Chris Zeoli, [*The Rise of Inference Engineering*](https://www.datagravity.dev/p/the-rise-of-inference-engineering), Data Gravity, August 13, 2026. + +- Spheron, [*What Is Inference Engineering? The 2026 GPU Cloud Guide*](https://www.spheron.network/blog/inference-engineering-guide-2026/), updated October 19, 2026. + +- Vishakha Sadhwani, [*AI Inference Engineering Roadmap 2026: Complete Learning Path*.](https://www.tech5ense.com/p/the-ai-inference-engineering-roadmap) + +- Charity Majors, Liz Fong-Jones and George Miranda, [*Observability Engineering*](https://www.oreilly.com/library/view/observability-engineering/9781492076438/), O'Reilly. + +- Lamhot Siagian, Efficient LLM Inference: Interview Pocket Notes, second edition, 2026. + +### Evals and benchmarks + +- Anthropic, [*Demystifying evals for AI agents*](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents), January 2026. + +- Hamel Husain and Shreya Shankar, [*AI Evals FAQ*](https://hamel.dev/blog/posts/evals-faq/). + +- UK AI Security Institute, [Inspect](https://inspect.aisi.org.uk). + +- Hugging Face, [upskill](https://github.com/huggingface/upskill). + +- SemiAnalysis, [AgentX methodology and datasets](https://inferencex.semianalysis.com/agentx/methodology). + +### Papers + +- Yu et al., [Orca: A Distributed Serving System for Transformer-Based Generative Models](https://www.usenix.org/conference/osdi22/presentation/yu), OSDI 2022. + +- Kwon et al., [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://arxiv.org/abs/2309.06180), SOSP 2023. + +- Zhong et al., [DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving](https://arxiv.org/abs/2401.09670), OSDI 2024. + +- Leviathan et al., [Fast Inference from Transformers via Speculative Decoding](https://arxiv.org/abs/2211.17192), ICML 2023. + +- Dao et al., [FlashAttention](https://arxiv.org/abs/2205.14135), 2022, and Shah et al., [FlashAttention-3](https://arxiv.org/abs/2407.08608), 2024. + +- Shoeybi et al., [Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism](https://arxiv.org/abs/1909.08053), 2019. + +- Shazeer et al., [Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer](https://arxiv.org/abs/1701.06538), 2017. \ No newline at end of file From ac5ddb88388dffeaf93bf60d1ba2654322e15029 Mon Sep 17 00:00:00 2001 From: Rae Sharp Date: Wed, 28 Oct 2026 12:17:02 -0400 Subject: [PATCH 2/5] Update prose and formatting Focusing on vale errors and clarity. Signed-off-by: Rae Sharp --- docs/content/guides/inference_guide.md | 1083 ++++++++++++----- docs/utils/vale/.vale.ini | 2 +- .../utils/vale/styles/Modelplane/Spelling.yml | 1 + .../config/vocabularies/Modelplane/accept.txt | 107 +- 4 files changed, 894 insertions(+), 299 deletions(-) diff --git a/docs/content/guides/inference_guide.md b/docs/content/guides/inference_guide.md index 14ee35bf8..fde95c7a9 100644 --- a/docs/content/guides/inference_guide.md +++ b/docs/content/guides/inference_guide.md @@ -1,22 +1,78 @@ -# From Platform Engineer to Inference Engineer: An Open Field Guide to Inference Engineering +--- +title: An Open Field Guide to Inference Engineering +weight: 40 +description: A field guide to inference concepts, production operations, tooling, and career development for platform engineers. +--- -*Numbers, prices and tool versions are as of October 2026.* - -This guide is for platform engineers who have been handed a GPU and a model and told to make it work. It covers how inference works, how to run it in production, which tools to learn, and how to turn that experience into a career. - -This guide builds on people who have written about inference in the open: Viktor Farcic's [*LLM Inference Explained*](https://devopstoolkit.live/ai/llm-inference-explained-12-concepts-you-actually-need-to-know/), Philip Kiely's [*Inference Engineering*](https://www.baseten.co/inference-engineering/), work from [The Pragmatic Engineer](https://newsletter.pragmaticengineer.com/p/what-is-inference-engineering), [ByteByteGo](https://blog.bytebytego.com/p/a-guide-to-ai-inference-engineering), [Data Gravity](https://www.datagravity.dev/p/the-rise-of-inference-engineering) and [Spheron](https://www.spheron.network/blog/inference-engineering-guide-2026/), Vishakha Sadhwani's [inference roadmap](https://www.tech5ense.com/p/the-ai-inference-engineering-roadmap), and Upbound's [Project Champagne](https://www.upbound.io/blog/drinking-our-own-champagne-what-it-takes-to-own-your-intelligence) series. Each is credited where the guide draws on it, and Section 7 lists them all. If you spot something that should be credited or corrected, open an issue or a pull request. - -## 1. Why this matters now - -Self-hosted inference is landing on platform teams. The coding assistant can't send source code to a third party, frontier model bills keep growing, and the support tool needs internal data. Each of those becomes the same request: run the model ourselves. - -Open models made that realistic. [Hugging Face](https://huggingface.co/models) hosts more than two million of them, and since DeepSeek V3 (December 2024) and R1 (January 2025), open models have kept pace with closed releases within months. Running your own lets you tune latency, aim for four nines of uptime, keep data on your network and, at volume, cut costs by around 80% ([*Inference Engineering*](https://www.baseten.co/inference-engineering/)). +{{< hint "note" >}} -The work suits platform engineers. The price of a fixed level of AI capability falls roughly 10x a year, and most of that comes from how models are served, not from better GPUs ([Data Gravity](https://www.datagravity.dev/p/the-rise-of-inference-engineering)). Upbound, the company behind Crossplane and Modelplane, saw this firsthand. In its [Project Champagne](https://www.upbound.io/blog/drinking-our-own-champagne-what-it-takes-to-own-your-intelligence) blog series, the team ran its own models on [Modelplane](https://github.com/modelplaneai/modelplane) and found that vLLM and scheduling worked. The time went to TLS, identity, timeouts, storage, IAM and observability. This guide draws on that series throughout, and Section 5 collects its lessons. - -### You are closer than you think +*Numbers, prices and tool versions are as of October 2026.* -An inference engineer picks a model, gets it onto the right GPU, keeps it serving, makes it faster and cheaper, and gets paged when it breaks. That is SRE for a new workload, and most of your skills carry over in a new shape: +{{< /hint >}} + +This guide is for platform engineers new to inference. If you need to understand +how models and GPUs work together and how to run inference in production, this +guide will help. You'll also learn about the tools to use and how to grow into +an inference engineering role. + +This guide builds on the work of others in the inference community. If you're +interested in inference engineering, continue your learning with these: + + +- Viktor Farcic's [*LLM Inference + Explained*](https://devopstoolkit.live/ai/llm-inference-explained-12-concepts-you-actually-need-to-know/) +- Philip Kiely's [*Inference + Engineering*](https://www.baseten.co/inference-engineering/) +- [The Pragmatic Engineer, What Is Inference + Engineering?](https://newsletter.pragmaticengineer.com/p/what-is-inference-engineering) +- [ByteByteGo's Guide to AI Inference + Engineering](https://blog.bytebytego.com/p/a-guide-to-ai-inference-engineering) +- [Data Gravity on The Rise of Inference + Engineering](https://www.datagravity.dev/p/the-rise-of-inference-engineering) +- [Spheron's Inference Engineering + Guide](https://www.spheron.network/blog/inference-engineering-guide-2026/) +- Vishakha Sadhwani's [*The AI Inference Engineering + Roadmap*](https://www.tech5ense.com/p/the-ai-inference-engineering-roadmap) +- Upbound, [*Project + Champagne*](https://www.upbound.io/blog/drinking-our-own-champagne-what-it-takes-to-own-your-intelligence) + series + + +## 1. Why now? + +In an AI-forward organization, platform teams become inference teams. Coding +assistants can't send source code to third parties, and frontier model bills +increase month to month. Support tools also need internal data that a hosted +model can't reach. + +Organizations want to run these models themselves, and open models now make that +practical. + +[Hugging Face](https://huggingface.co/models) hosts more than two million open +models, and the best of them compete with closed models. DeepSeek released R1 in +January 2025, and [it matched OpenAI's o1 on math and coding benchmarks about a +month after o1 came out](https://arxiv.org/abs/2501.12948). When you run your +own model, you control latency, uptime, and where your data goes. At high +volume, you can also [cut costs by about +80%](https://www.baseten.co/inference-engineering/). + +The work suits platform engineers. The price of a fixed level of AI capability +falls roughly 10x a year, and most of that comes from how models are served, +[not from better +GPUs](https://www.datagravity.dev/p/the-rise-of-inference-engineering). Upbound, +the company behind Crossplane and Modelplane, saw this firsthand. In its +[Project +Champagne](https://www.upbound.io/blog/drinking-our-own-champagne-what-it-takes-to-own-your-intelligence) +blog series, the team ran its own models on +[Modelplane](https://github.com/modelplaneai/modelplane) and found that vLLM and +scheduling worked. The time went to TLS, identity, timeouts, storage, IAM and +observability. + +### Skills that carry over + +An inference engineer picks a model, gets it onto the right GPU, keeps it +serving, makes it faster and cheaper, and gets paged when it breaks. That's SRE +for a new workload, and most of your skills carry over: | What you already know | What it becomes in inference | |------------------------------|--------------------------------------------------------------------------------------| @@ -30,82 +86,139 @@ An inference engineer picks a model, gets it onto the right GPU, keeps it servin | FinOps | Cost per million tokens and cost per user session | | Identity and access | Credentials that last as long as an agent session | -A hosted API is still the right call early in a product's life ([ByteByteGo](https://blog.bytebytego.com/p/a-guide-to-ai-inference-engineering)). Self-hosting pays off when API spend becomes a real line item, when latency or reliability needs outgrow vendors, or when data can't leave your network. And the field is wide open: in August 2026, [Chris Zeoli counted](https://www.datagravity.dev/p/the-rise-of-inference-engineering) about 58 open inference roles on public careers boards, with no settled job title. +A hosted API is still [the right call early in a product's +life](https://blog.bytebytego.com/p/a-guide-to-ai-inference-engineering). +Self-hosting pays off when API spend becomes a real line item, when latency or +reliability needs outgrow vendors, or when data can't leave your network. In +August 2026, [Chris Zeoli +counted](https://www.datagravity.dev/p/the-rise-of-inference-engineering) about +58 open inference roles on public careers boards, with no settled job title. ## 2. Inference 101 -Every inference concept answers one of two questions: what is the engine holding on the GPU, and what limits how fast it can work? This section walks through the concepts in roughly the order you will meet them. +This section covers the concepts in roughly the order you'll meet them. Most are +about what the engine keeps in GPU memory or what limits its speed. -For each concept, it also helps to ask when it starts to matter to you, a framing from Viktor Farcic's *LLM Inference Explained*. Some concepts matter the moment you deploy a model. Others only show up once many people use it, and some you may never need. Knowing which is which tells you what to deal with now and what to leave for later, and the table near the end of this section sums it up. +[Viktor +Farcic](https://devopstoolkit.live/ai/llm-inference-explained-12-concepts-you-actually-need-to-know/) +suggests asking when each concept starts to matter. Some matter as soon as you +deploy, others only once many people use the model, and some you may never need. +The table near the end of this section gives the signal for each. ### What an inference engine is -An inference engine, such as [vLLM](https://github.com/vllm-project/vllm) or [SGLang](https://github.com/sgl-project/sglang), is the server that turns model weights into an API. For a platform engineer, the closest mental model is a stateful database server with a GPU attached. It loads a large dataset into memory at startup, keeps per-session state, schedules work internally and answers over HTTP. [Ollama](https://github.com/ollama/ollama) on a laptop is the same idea at small scale, and every hosted model API runs one behind the scenes. +An inference engine, like [vLLM](https://github.com/vllm-project/vllm) or +[SGLang](https://github.com/sgl-project/sglang), is the server that turns model +weights into an API. For a platform engineer, the closest mental model is a +stateful database server with a GPU attached. It loads a large dataset into +memory at startup and keeps per-session state. It also schedules work internally +and answers over HTTP. [Ollama](https://github.com/ollama/ollama) on a laptop is +the same idea at small scale, and every hosted model API runs one behind the +scenes. -| You know this from databases and services | The engine's equivalent | +| You know this from databases and services | The engine's counterpart | |---------------------------------------------------|--------------------------------------------------------------------| | Loading a dataset into the buffer pool at startup | Loading model weights into GPU memory, which can take many minutes | | Per-session state in a session store | The KV cache, one for each conversation in flight | -| A query planner and connection pool | The scheduler that decides which requests share each GPU pass | +| A query planner and connection pool | The scheduler, which batches requests into each GPU pass | | A wire protocol clients already speak | An OpenAI- or Anthropic-compatible HTTP API | | Warming caches after a restart | Cold starts: pulling a large image and reloading the weights | -The analogy breaks in one important place. A database's working set lives in plentiful RAM, while an engine's lives in scarce, expensive GPU memory. Most tuning comes down to how much of that memory each request gets. +Unlike a database, the engine keeps its working set in GPU memory, not RAM. GPU +memory is small and expensive, so most tuning is about deciding how much of it +each request gets. -### The four layers +### The inference stack -The engine is one layer of four. At the bottom, **kernels** do the arithmetic; they are benchmarked often and rarely chosen. Above them sits the **engine**. Above that, a **control plane** deploys and manages engines, because sooner or later you run more than one. On top, a **gateway** routes traffic across models and replicas. +The engine is one layer of four. At the bottom, **kernels** do the arithmetic. +You rarely choose them directly. Above them sits the **engine**. Above that, a +**control plane** deploys and manages engines, because sooner or later you run +more than one. On top, a **gateway** routes traffic across models and replicas. ![Inference stack: four layers and the clients they serve](inference-stack.png) *Inference stack: four layers and the clients they serve* -Requests flow down the left column. The control plane sits beside it: it deploys and manages the gateway and engines, but no request passes through it. +Requests pass through the gateway to an engine, which runs kernels on the GPU. +The control plane deploys and manages the gateway and engines, but no request +passes through it. ### Tokens -A token is the unit an LLM works in, roughly a word or word fragment. Context windows, prices and speeds are all counted in tokens. When someone quotes tokens per second, they mean this unit. +A token is the unit a language model works in, roughly a word or word fragment. +Context windows, prices and speeds are all counted in tokens. When someone +quotes tokens per second, they mean this unit. -### The two costs on the GPU +### Weights and the KV cache -GPU memory holds two very different things, and sizing a deployment means budgeting for both. If you have sized a database, the split will feel familiar: a fixed footprint you pay once, and a working set that grows with load. +The weights are fixed, and the KV cache grows with load. Size for both, as you +would a database's footprint and working set. -**Weights are the fixed footprint.** Most models ship at 2 bytes per parameter (BF16 or FP16), so an 8B model needs about 16 GB, and it has to live in GPU memory, not system RAM. On a 48 GB L40S, that leaves about two-thirds of the card for everything else. Whether one person uses the model or a hundred, this number does not move. +**Weights are the fixed footprint.** Most models are released at 2 bytes per +parameter (BF16 or FP16), so an 8B model needs about 16 GB, and it has to live +in GPU memory, not system RAM. On a 48 GB L40S, that leaves about two-thirds of +the card for everything else. That number is the same for one user or a hundred. -**The KV cache is the working set.** Each new token is computed against every token before it. Rather than recompute that history, the engine keeps the attention keys and values for every token in flight. Its size scales with context length multiplied by the number of concurrent conversations, much like a session store that grows with both session size and user count. +**The KV cache is the working set.** Each new token is computed against every +token before it. Rather than recompute that history, the engine keeps the +attention keys and values for every token in flight. Its size scales with +context length multiplied by the number of concurrent conversations, much like a +session store that grows with both session size and user count. You can estimate the KV cache per token from a model's config: -$$ -\text{KV bytes per token} = 2 \times \text{layers} \times \text{KV heads} \times \text{head dimension} \times \text{bytes per value} -$$ +```text +KV bytes per token = 2 × layers × KV heads × head dimension × bytes per value +``` -For Llama 3 70B at FP16 (80 layers, 8 KV heads, head dimension 128), that is about 320 KB per token. One 128K-token conversation needs about 40 GB of cache. This is why the KV cache, not the model, usually decides how many users a GPU can serve. +For Llama 3 70B at FP16 (80 layers, 8 KV heads, head dimension 128), that is +about 320 KB per token. One 128K-token conversation needs about 40 GB of cache. +At that size, the KV cache usually limits how many users a GPU can serve. -Three details catch people out: +Watch for these sizing mistakes: -- **Mixture-of-experts (MoE) models keep every weight resident.** A model with 320B total parameters and 18B active still needs about 320 GB at FP8. Active parameters set compute, not memory. +- **Mixture-of-experts (MoE) models keep every weight resident.** A model with + 320B total parameters and 18B active still needs about 320 GB at FP8. Only + compute scales with the active parameters. -- **Read the checkpoint size, not the dtype table.** While self-hosting its own models for [Project Champagne](https://www.upbound.io/blog/drinking-our-own-champagne-part-2-finding-the-corks), Upbound estimated one MoE model at about 315 GB from its dtypes (Part 2). The real checkpoint was 177.7 GB, because the experts were stored in a 4-bit format. +- **Read the checkpoint size, not the dtype table.** While self-hosting its own + models for [Project + Champagne](https://www.upbound.io/blog/drinking-our-own-champagne-part-2-finding-the-corks), + Upbound estimated one MoE model at about 315 GB from its dtypes (Part 2). The + real checkpoint was 177.7 GB, because the experts were stored in a 4-bit + format. -- **Runtime overhead is real.** CUDA graphs, activations and memory fragmentation all take space. A model that fits at startup can still run out of memory days later ([Project Champagne, Part 2](https://www.upbound.io/blog/drinking-our-own-champagne-part-2-finding-the-corks)). +- **Leave room for runtime overhead.** CUDA graphs, activations and memory + fragmentation all take space. A model that fits at startup can still run out + of memory days later ([Project Champagne, Part + 2](https://www.upbound.io/blog/drinking-our-own-champagne-part-2-finding-the-corks)). ### Two phases, two bottlenecks -Every request runs in two phases on the same GPU, and they are limited by different things ([ByteByteGo](https://blog.bytebytego.com/p/a-guide-to-ai-inference-engineering)). +Each request runs in two phases on the same GPU, and [they are limited by +different +things](https://blog.bytebytego.com/p/a-guide-to-ai-inference-engineering). | Phase | What it does | Limited by | Metric it sets | |---------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------------|---------------------------------------------| -| Prefill | Reads the whole input in parallel: system prompt, tool definitions, conversation history. Produces the first token and the KV cache. | Compute | Time to first token (TTFT) | +| Prefill | Reads the whole input in parallel, including the system prompt and tool definitions as well as the conversation history. Produces the first token and the KV cache. | Compute | Time to first token (TTFT) | | Decode | Writes the answer one token at a time. Each token needs a pass that reads all active weights, plus the KV cache, from memory. At long context, reading the cache can cost more than reading the weights. | Memory bandwidth | Time per output token and tokens per second | -Decode is the less intuitive one. Generating one token does one or two floating-point operations per byte read from memory, depending on weight precision. An H100 needs about 295 operations per byte to stay busy. So a single request uses a small fraction of the chip while the compute units wait on memory ([Data Gravity](https://www.datagravity.dev/p/the-rise-of-inference-engineering)). +Generating one token does one or two floating-point operations per byte read +from memory, depending on weight precision. An H100 needs about 295 operations +per byte to stay busy. That means one request uses a small fraction of the chip +while the compute units wait on memory ([Data +Gravity](https://www.datagravity.dev/p/the-rise-of-inference-engineering)). -The gap is growing. Peak compute has grown about 3x every two years, against about 1.6x for memory bandwidth. Nearly every optimization does one of two things: move fewer bytes per token, or spread each byte across more tokens. +Peak compute has grown about 3x every two years, against about 1.6x for memory +bandwidth. Nearly every optimization either moves fewer bytes per token or +spreads each byte across more tokens. -Because the phases have opposite bottlenecks, a technique that speeds one often does little for the other. That is why benchmarks report TTFT and tokens per second separately. +Because the phases have opposite bottlenecks, a technique that speeds one often +does little for the other. That is why benchmarks report TTFT and tokens per +second separately. -### The metrics you will live with +### Metrics to track | Metric | What it measures | Watch out for | |-----------------------------------------------|------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------| @@ -115,27 +228,34 @@ Because the phases have opposite bottlenecks, a technique that speeds one often | Throughput | Total tokens per second across all users | Rises as batch size rises | | Goodput | Throughput that still meets your latency targets | The number to plan capacity with | | End-to-end latency | Total request time | Depends on how much the model writes. A healthy 190-second answer is normal for long generations. | -| Percentiles | P50, P90, P99 of each latency | Users feel the tail, not the average | +| Percentiles | p50, p90, p99 of each latency | Users feel the tail, not the average | | Errors and refusals | Failed or rejected requests | Count refusals as errors, or a context cap can look like a speedup | | Cost per million tokens | GPU cost divided by tokens served | Compare at the load you actually run | -Cost per million tokens converts a GPU price and a throughput into one comparable number: +Cost per million tokens converts a GPU price and a throughput into one +comparable number: -$$ -\text{Cost per million tokens} = \frac{\text{GPU cost per hour}}{\text{tokens per second} \times 3600} \times 10^6 -$$ +```text +Cost per million tokens = GPU cost per hour / (tokens per second × 3600) × 1,000,000 +``` -[Spheron's](https://www.spheron.network/blog/inference-engineering-guide-2026/) worked example: a GPU at \$2.90 an hour sustaining 1,200 tokens per second costs about \$0.67 per million tokens. +[Spheron's](https://www.spheron.network/blog/inference-engineering-guide-2026/) +worked example: a GPU at \$2.90 an hour sustaining 1,200 tokens per second costs +about \$0.67 per million tokens. ### Latency versus throughput -A GPU is good at doing many things at once and ordinary at doing one thing quickly. Batching more requests together raises total throughput and slows each individual user. Every product picks a point on that curve. Interactive chat and coding favor latency. Overnight batch jobs favor throughput. +Batching more requests together raises total throughput and slows each +individual user. Interactive chat and coding favor latency. Overnight batch jobs +favor throughput. -It is the same choice you make when tuning a database for transactions or for analytics. You cannot optimize for both at once, so start from the workload and write its constraints down. The more specific they are, the easier every later trade-off becomes. +It's the same trade-off as tuning a database for transactions or for analytics. +Write down the workload's constraints before you tune anything. -### When each concept starts to matter +### When to adopt each concept -The table below turns that question into signals you can watch for. Each row names the symptom that tells you it is time to act, in roughly the order teams meet them. +Each row gives the symptom that tells you it's time to act, in roughly the order +teams need them. | Concept | What it does | The signal that you need it | |----------------------------------------|-------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------| @@ -149,69 +269,110 @@ The table below turns that question into signals you can watch for. Each row nam | Autoscaling | Add and remove replicas on queue depth and KV cache use | Idle GPUs show up on the bill, or queues build at peak hours | | Disaggregation | Run prefill and decode on separate GPUs | Long prompts stall streaming at scale, and you have RDMA-class networking | -Most teams need only the first few rows on day one. Treat the rest as a backlog that the signals pull forward, not a checklist to finish before launch. +Most teams need only the first few rows on day one. Add the rest when their +signals show up. ### What agents change -Coding agents and other agentic workloads stress every part of this picture. Upbound measured this in [Part 5](https://www.upbound.io/blog/drinking-our-own-champagne-part-5-what-does-it-take-to-grow-a-self-hosted-coding-model-from-pilot-to-team) of its Project Champagne series (introduced in Section 1), where about 10 of its engineers ran Claude Code and other coding agents against a self-hosted model. Their traffic looked like this: - -- **Prompts are long.** The median prompt was 59K tokens and the P90 was 125K. +Coding agents and other agentic workloads stress every part of this picture. +Upbound measured this in [Part +5](https://www.upbound.io/blog/drinking-our-own-champagne-part-5-what-does-it-take-to-grow-a-self-hosted-coding-model-from-pilot-to-team) +of its Project Champagne series (introduced in Section 1), where about 10 of its +engineers ran Claude Code and other coding agents against a self-hosted model. +Their traffic looked like this: -- **Prompts are mostly repeated.** About two-thirds of prompt tokens hit the prefix cache, because agents resend their transcript on every turn. +- **Prompts are long.** The median prompt was 59K tokens and the p90 was 125K. -- **Thinking is hidden.** A reasoning model can think for more than a minute before its first visible token. Every timeout between the agent and the model has to allow for that. +- **Prompts are mostly repeated.** About two-thirds of prompt tokens hit the + prefix cache, because agents resend their transcript on every turn. -- **Load depends on the client.** Short-prompt sessions launch subagents and can keep more requests in flight than long-prompt sessions. Plan capacity in requests in flight rather than seats. +- **Thinking is hidden.** A reasoning model can think for more than a minute + before its first visible token. Every timeout between the agent and the model + has to allow for that. -- **Sessions run for hours.** Streams, tool calls and credentials all have to survive that long. +- **Load depends on the client.** Short-prompt sessions launch subagents and can + keep more requests in flight than long-prompt sessions. Plan capacity in + requests in flight rather than seats. -If your inference platform works for agents, it will work for most other clients. +- **Sessions run for hours.** Streams, tool calls and credentials all have to + keep working for that long. ## 3. Running inference in production -With the concepts in place, the next question is how to run them in production. Every choice trades latency, throughput, cost and quality against each other, usually under GPU scarcity. The practices below draw on the sources cited throughout. Where one comes from Upbound’s Project Champagne, the specifics are in the case study in Section 5. +In production, every choice trades latency, throughput, cost and quality against +each other, usually with too few GPUs. Where a practice comes from Project +Champagne, Section 5 has the details. ### 3.1 Start from use cases, not hardware -Start from the workloads, not the GPUs. Each use case writes different requirements: +Each use case sets different requirements. In Project Champagne: -- **Coding agents** needed long context without quality loss, reliable tool-call parsing, streaming, sessions that last hours, and an Anthropic-compatible API for Claude Code. +- **Coding agents** needed long context without quality loss, reliable tool-call + parsing, streaming, sessions that last hours, and an Anthropic-compatible API + for Claude Code. -- **Chat** needed low time to first token, real concurrency, and an OpenAI-compatible API for existing tools. +- **Chat** needed low time to first token, real concurrency, and an + OpenAI-compatible API for existing tools. -- **The platform** needed TLS on a real hostname, identity tied to the company domain, observability that tells a healthy engine from a busy one, and correct placement without babysitting. +- **The platform** needed TLS on a real hostname and identity tied to the + company domain. It also needed observability that tells a healthy engine from + a busy one, and correct placement without babysitting. -Separate interactive workloads from batch workloads. They need different targets, and often different deployments. Write the pass and fail bar down before you test anything. +Separate interactive workloads from batch workloads. They need different +targets, and often different deployments. Write the pass and fail bar down +before you test anything. ### 3.2 Pick models deliberately -- **Evaluate on your own tasks.** Public benchmarks measure someone else's workload. Section 3.11 covers how. +- **Evaluate on your own tasks.** Public benchmarks measure someone else's + workload. Section 3.11 covers how. -- **Start small, then grow.** A small MoE model with a few billion active parameters builds operating experience faster and costs less to get wrong. +- **Start small, then grow.** A small MoE model with a few billion active + parameters builds operating experience faster and costs less to get wrong. -- **The newest model is not always the right one.** Model releases move faster than hardware plans. A new ‘small’ flagship can need several of the largest cards, while your current model may be running at half its supported context. The cheapest gain may need no new model at all. +- **The newest model is not always the right one.** Model releases move faster + than hardware plans. A new ‘small’ flagship can need several of the largest + cards, while your current model may be running at half its supported context. + The cheapest gain may not need a new model at all. -- **Check that your engine can actually load it.** A model listed as supported in release notes may still fail to load on your hardware and engine build. +- **Check that your engine can actually load it.** A model listed as supported + in release notes may still fail to load on your hardware and engine build. -- **Check the license.** Most open models use permissive licenses, but some restrict commercial use. +- **Check the license.** Most open models use permissive licenses, but some + restrict commercial use. -- **Check the chat template and parsers.** Tool calls and reasoning output depend on settings each model needs in the engine. A wrong one breaks tool use while every request still returns 200. +- **Check the chat template and parsers.** Tool calls and reasoning output + depend on settings each model needs in the engine. A wrong one breaks tool use + while every request still returns 200. ### 3.3 Size for runtime, not startup -A model that fits at startup can still run out of memory under real traffic, often days later and mid-request, when fragmentation pushes past the profiled peak. +A model that fits at startup can still run out of memory under real traffic when +fragmentation pushes past the profiled peak. That can happen days later, in the +middle of a request. -- **Budget all three costs:** weights, KV cache, and runtime overhead such as CUDA graphs and activations. +- **Budget all three costs:** weights, KV cache, and runtime overhead such as + CUDA graphs and activations. -- **Know what your engine's memory setting covers, and check again on every upgrade.** vLLM sizes its KV cache from a startup estimate of everything else, including CUDA graph pools, and real traffic can go past that estimate. If you see out-of-memory errors under load, lower the memory fraction slightly, enable expandable segments and cap concurrent sequences, then measure free memory again. +- **Know what your engine's memory setting covers, and check again on every + upgrade.** vLLM sizes its KV cache from a startup estimate of everything else, + including CUDA graph pools, and real traffic can go past that estimate. If you + see out-of-memory errors under load, lower the memory fraction slightly, + enable expandable segments and cap concurrent sequences, then measure free + memory again. -- **Cap every request.** Set a maximum output length on every call. Without one, a single request can take a model down for everyone, and abandoned requests keep generating after the client gives up. +- **Cap every request.** Set a maximum output length on every call. Without one, + any request can take a model down for everyone, and abandoned requests keep + generating after the client gives up. -- **Treat context length as a lever, carefully.** A shorter maximum context frees KV cache for more users. It also rejects every longer conversation, so count those rejections as errors. +- **Shorten context with care.** A shorter maximum context frees KV cache for + more users. It also rejects every longer conversation, so count those + rejections as errors. ### 3.4 Know the optimization techniques -Most engines turn on continuous batching and paged KV cache by default. The rest are choices you make per model and workload. +Most engines turn on continuous batching and paged KV cache by default. The rest +are choices you make per model and workload. | Technique | What it improves | Cost or risk | Use it when | |-----------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------| @@ -228,195 +389,356 @@ Most engines turn on continuous batching and paged KV cache by default. The rest A few rules of thumb: -- **Structure prompts for the cache.** Prefix caching stops at the first token that differs. Put shared content such as system prompts and tool definitions first, and variable user input last. Hosted APIs price cached input at 10 to 50% of fresh input, as of 2026, for the same reason ([Data Gravity](https://www.datagravity.dev/p/the-rise-of-inference-engineering)). - -- **Quantize from least to most sensitive.** Linear weights tolerate quantization best, then activations, then the KV cache. Attention layers are the most sensitive, because small errors compound across a long sequence. Keep the most sensitive parts at higher precision until your evals show you can go lower. Floating-point formats such as FP8 handle outlier values better than integer formats. - -- **Watch speculative decoding acceptance.** Acceptance depends on temperature, subject matter and the draft model. If acceptance collapses, every pass still pays for the guesses and the model slows while other panels look normal. - -- **Beware interactions when applying multiple techniques.** Quantizing the KV cache helps disaggregation. Larger batches leave less spare compute for speculation. [One Baseten engineer tried 77 configurations](https://newsletter.pragmaticengineer.com/p/what-is-inference-engineering) before finding one that doubled throughput for a customer (Inference Engineering). Change one thing at a time and measure (Section 3.10). - -- **Keep long prompts from stalling other streams.** Chunked prefill splits a long prompt across several passes, so decode keeps moving for everyone else. Current vLLM turns it on by default, and its token budget per pass is the setting to tune before you consider disaggregation. +- **Structure prompts for the cache.** Prefix caching stops at the first token + that differs. Put shared content such as system prompts and tool definitions + first, and variable user input last. [Hosted APIs price cached input at 10 to + 50% of fresh input, as of 2026, for the same + reason](https://www.datagravity.dev/p/the-rise-of-inference-engineering)). + +- **Quantize from least to most sensitive.** Linear weights tolerate + quantization best, then activations, then the KV cache. Attention layers are + the most sensitive, because small errors compound across a long sequence. Keep + the most sensitive parts at higher precision until your evals show you can go + lower. Floating-point formats such as FP8 handle outlier values better than + integer formats. + +- **Watch speculative decoding acceptance.** Acceptance depends on temperature, + subject matter and the draft model. If acceptance collapses, every pass still + pays for the guesses and the model slows while other panels look normal. + +- **Beware interactions when applying multiple techniques.** Quantizing the KV + cache helps disaggregation. Larger batches leave less spare compute for + speculation. [One Baseten engineer tried 77 + configurations](https://newsletter.pragmaticengineer.com/p/what-is-inference-engineering) + before finding one that doubled throughput for a customer (Inference + Engineering). Change one thing at a time and measure (Section 3.10). + +- **Keep long prompts from stalling other streams.** Chunked prefill splits a + long prompt across several passes, so decode keeps moving for everyone else. + Current vLLM turns it on by default, and its token budget per pass is the + setting to tune before you consider disaggregation. ### 3.5 Get GPUs, and place them correctly -- **Capacity is not quota**. Quota is permission to ask. Each instance size is its own capacity pool, even with the same GPU, and no cloud API tells you what you can actually launch. +- **Quota doesn't guarantee capacity.** Quota only lets you request instances. + Each instance size is its own capacity pool, even with the same GPU, and no + cloud API tells you what you can actually launch. -- **Design for a shape that can change.** Expect to get the GPU you planned for only some of the time. A model may end up split across smaller nodes in different zones, using pipeline parallelism over ordinary networking. Allow multi-zone pools, and keep the model definition independent of the node shape. +- **Design for whatever GPUs you can get.** Expect to get the GPU you planned + for only some of the time. A model may end up split across smaller nodes in + different zones, using pipeline parallelism over ordinary networking. Allow + multi-zone pools, and keep the model definition independent of the instance + type and node count. -- **Match models to hardware explicitly.** Kubernetes Dynamic Resource Allocation (DRA) device selectors can pin a model to a GPU architecture or a minimum memory size. Verify placement against what the GPU driver publishes, not against a status field that echoes what you declared. +- **Match models to hardware explicitly.** Kubernetes Dynamic Resource + Allocation (DRA) device selectors can pin a model to a GPU architecture or a + minimum memory size. Verify placement against what the GPU driver publishes, + not against a status field that echoes what you declared. -- **Watch the cloud, not just the controller.** A managed node group can report failure while the cloud keeps trying to launch instances, and GPUs that arrive hours later can be terminated by a later change to the pool. Track instances and node joins, and list a pool's instances before you change it. +- **Watch the cloud, not just the controller.** A managed node group can report + failure while the cloud keeps trying to launch instances, and GPUs that arrive + hours later can be shut down by a later change to the pool. Track instances + and node joins, and list a pool's instances before you change it. -- **Know your interconnect.** SXM datacenter GPUs such as H100, H200 and B200 link GPUs with NVLink. PCIe cards such as the L40S and RTX PRO 6000 do not, so multi-GPU communication goes over PCIe and host memory. That changes which parallelism strategies pay off. +- **Know your interconnect.** SXM datacenter GPUs such as H100, H200 and B200 + link GPUs with NVLink. PCIe cards such as the L40S and RTX PRO 6000 do not, so + multi-GPU communication goes over PCIe and host memory. That changes which + parallelism strategies pay off. -- **Size the system pool too.** GPU nodes are tainted for inference. Monitoring, controllers and gateways compete for whatever general capacity is left. +- **Size the system pool too.** GPU nodes are tainted for inference. Monitoring, + controllers and gateways compete for whatever general capacity is left. ### 3.6 Gateways, routing and timeouts -- **Do not round-robin.** Requests are not interchangeable. One replica may already hold a conversation's prefix in cache while another has never seen it. Use cache-aware or prefix-aware routing, such as the [Gateway API Inference Extension](https://github.com/kubernetes-sigs/gateway-api-inference-extension)'s endpoint picker. +- **Do not round-robin.** One replica may already have a conversation's prefix + in cache while another has never seen it. Use cache-aware or prefix-aware + routing, such as the endpoint picker in the [Gateway API Inference + Extension](https://github.com/kubernetes-sigs/gateway-api-inference-extension). -- **Route by model name.** A single hostname can serve many models, routed on the request's model field. +- **Route by model name.** One hostname can serve many models, routed on the + request's model field. -- **Speak both common APIs.** Most tools expect the OpenAI API. Claude Code expects the Anthropic Messages API, which [vLLM](https://github.com/vllm-project/vllm) serves natively, including streaming and tool calls. +- **Speak both common APIs.** Most tools expect the OpenAI API. Claude Code + expects the Anthropic Messages API, which + [vLLM](https://github.com/vllm-project/vllm) serves natively, including + streaming and tool calls. -- **Raise every timeout on the path.** Long generations and reasoning models break the defaults, because each hop has a default timeout sized for web requests, not for generations that run for minutes. Typical culprits are a gateway route timeout that bounds the whole response, a load balancer idle timeout, and retries that multiply the wait. Check the route, each load balancer, the gateway, the client and any helper calls the client makes. +- **Raise every timeout on the path.** Long generations and reasoning models + break the defaults, because each hop has a default timeout sized for web + requests, not for generations that run for minutes. Typical culprits are a + gateway route timeout that bounds the whole response, a load balancer idle + timeout, and retries that multiply the wait. Check the route, each load + balancer, the gateway, the client and any helper calls the client makes. -- **Detect truncated streams.** A cut stream can return HTTP 200 with valid tokens and simply stop. To the client, it looks like a finished answer. +- **Detect truncated streams.** A cut stream can return HTTP 200 with valid + tokens and then stop. To the client, it looks like a finished answer. -- **Meter usage per caller.** Token-based rate limits and quotas at the gateway keep one heavy user or runaway agent from starving everyone else. +- **Meter usage per caller.** Token-based rate limits and quotas at the gateway + keep one heavy user or runaway agent from starving everyone else. ### 3.7 Security and identity -- **Never ship an open endpoint.** Apply your auth policy before the gateway gets an address, and remove it last when you tear down. +- **Never expose an open endpoint.** Apply your auth policy before the gateway + gets an address, and remove it last when you tear down. -- **Use real TLS on a real hostname,** with a plan for certificate issuance and renewal. +- **Use real TLS on a real hostname,** with a plan for certificate issuance and + renewal. -- **Plan for three kinds of client.** People are easy. Browsers cannot attach a bearer token to page loads or preflight requests, so handle browser login with OIDC at the gateway. Agents run for hours and read their credential once, so a 60-minute identity token will kill them mid-session. Give agents long-lived API keys or a credential that refreshes. +- **Plan for three kinds of client.** People are the simplest case. Browsers + cannot attach a bearer token to page loads or preflight requests, so handle + browser login with OIDC at the gateway. Agents run for hours and read their + credential once, so a 60-minute identity token will end them mid-session. Give + agents long-lived API keys or a credential that refreshes. -- **Do not stack two expiring sessions.** An outer gateway login in front of an app with its own login causes an outage when the outer session expires. +- **Do not stack two expiring sessions.** An outer gateway login in front of an + app with its own login causes an outage when the outer session expires. -- **Test what should be refused.** Keep a small probe matrix: no token, a garbage token, a key you did not sign, an expired key, a revoked key, a forged caller header. Include what should stay open, such as the health check. Run it on a schedule and in a loop during changes. +- **Test what should be refused.** Build a probe matrix of requests the gateway + must reject, such as a missing, garbage, unsigned, expired or revoked key, or + a forged caller header. Include paths that must remain reachable, such as the + health check. Run it on a schedule and in a loop during changes. -- **Carry identity into usage records,** so you can see who sends what. Overwrite any caller header a client tries to forge. +- **Carry identity into usage records,** so you can see who sends what. + Overwrite any caller header a client tries to forge. -- Treat model weights like code. Prefer safetensors over pickle-based checkpoints, don't enable remote code from a model repo you haven't read, and pin each model to a specific revision. +- Treat model weights like code. Prefer safetensors over pickle-based + checkpoints, don't enable remote code from a model repo you haven't read, and + pin each model to a specific revision. ### 3.8 Observability -- **Scrape the engines.** Engines expose detailed Prometheus metrics. Nothing collects them unless something is set up to scrape them, such as a ServiceMonitor and a named port. +- **Scrape the engines.** Engines expose detailed Prometheus metrics. Nothing + collects them unless something is set up to scrape them, such as a + ServiceMonitor and a named port. -- **Build the core panels:** requests running against the concurrency limit, requests waiting, KV cache use, TTFT, time per output token, output tokens per request, prefix cache hit rate, speculative decoding acceptance, errors by status code, and cancellations. +- **Build the core panels:** requests running against the concurrency limit, + requests waiting, KV cache use, TTFT, time per output token, output tokens per + request, prefix cache hit rate, speculative decoding acceptance, errors by + status code, and cancellations. -- Measure quality on the server. Broken tool calls, cut-off answers and empty responses usually reach you as user complaints while the serving dashboards look normal. Count them on the server instead, including tool calls that fail to parse, responses cut off by the length limit, empty outputs, refusals and hidden reasoning tokens per request. When the engine doesn't export one of these, the gateway sees every response and can count it. +- Measure quality on the server. Broken tool calls, cut-off answers and empty + responses usually reach you as user complaints while the serving dashboards + look normal. Count them on the server instead, including tool calls that fail + to parse, responses cut off by the length limit, empty outputs, refusals and + hidden reasoning tokens per request. The gateway sees every response, so it + can count anything the engine doesn't export. -- **Look past GPU "utilization."** It reports whether anything ran, not how much of the chip worked. A GPU can report near 100% “utilization” while SM activity is around half and tensor cores sit mostly idle. Graph SM activity and tensor activity as well. +- **Look past `GPU-Util`.** It reports whether anything ran, not how much of the + chip worked. A GPU can report near 100% `GPU-Util` while SM activity is around + half and tensor cores sit mostly idle. Graph SM activity and tensor activity + as well. -- **Learn the shapes of "slow."** Queued: running is at the limit, waiting is above zero, and the KV cache is full. Thinking: TTFT looks fine but each request writes many hidden tokens. Genuinely slow: time per output token is up. +- **Tell the kinds of slow apart.** When requests are queued, running is at the + limit, waiting is above zero and the KV cache is full. When the model is + thinking, TTFT looks fine but each request writes many hidden tokens. When + generation is slow, time per output token is up. -- **Watch the watcher.** Something outside the monitoring stack must check that data is fresh and that new metrics arrive from every cluster. A metrics adapter can report its API as available while holding no data. +- **Watch the watcher.** Something outside the monitoring stack must check that + data is fresh and that new metrics arrive from every cluster. A metrics + adapter can report its API as available while returning empty results. -- **Keep the evidence.** Metric scrapes can make up nearly all of a retained engine log. Keep per-request access logs with the caller, ship them to a log store, and remember kubectl logs --previous after a crash. +- **Keep logs you can debug from.** Metric scrapes can make up nearly all of a + retained engine log. Keep per-request access logs with the caller, send them + to a log store, and remember `kubectl logs --previous` after a crash. -- **Normalize metric names across engines,** so a vLLM deployment and an SGLang deployment are judged on the same signals. +- **Normalize metric names across engines,** so a vLLM deployment and an SGLang + deployment are judged on the same signals. Set SLOs on what users feel, measured per request at the front door: -- **Responsive:** time to the first token the user actually sees, under a threshold. +- **Responsive:** time to the first token the user actually sees, under a + threshold. - **Fast enough:** time per output token, under a threshold. -- **Answered:** the request completed without an error, and any tool call in it parsed. +- **Answered:** the request completed without an error, and any tool call in it + parsed. -Do not set an SLO on total request time, because it depends on how much the model writes. Low-traffic models need a fleet-wide SLO, longer windows or synthetic traffic. Missing data reads as 100% good, so keep a separate alert for metrics that stop arriving. Open source tools such as [Sloth](https://github.com/slok/sloth) and [Pyrra](https://github.com/pyrra-dev/pyrra) turn SLO definitions into burn-rate alerts. For background, see [*Observability Engineering*](https://www.oreilly.com/library/view/observability-engineering/9781492076438/). +Do not set an SLO on total request time, because it depends on how much the +model writes. Low-traffic models need a fleet-wide SLO, longer windows or +synthetic traffic. Missing data reads as 100% good, so keep a separate alert for +metrics that stop arriving. Open source tools such as +[Sloth](https://github.com/slok/sloth) and +[Pyrra](https://github.com/pyrra-dev/pyrra) turn SLO definitions into burn-rate +alerts. For background, see [*Observability +Engineering*](https://www.oreilly.com/library/view/observability-engineering/9781492076438/). ### 3.9 Autoscaling and cold starts -- **Scale on inference signals,** such as queue depth, KV cache use and requests in flight. CPU usage tells you almost nothing here. +- **Scale on inference signals,** such as queue depth, KV cache use and requests + in flight. CPU usage tells you almost nothing here. -- **Combine traffic and utilization.** Traffic lets you scale ahead of demand, and utilization lags. The knobs are the ones you already tune on any autoscaler: a floor and a ceiling, how far back the window looks, how long to wait before scaling down, and how many concurrent requests one replica should hold. [Kiely's book](https://www.baseten.co/inference-engineering/) covers each in depth for inference. +- **Combine traffic and load.** Traffic lets you scale ahead of demand, and load + lags behind it. The settings are the ones you already tune on any autoscaler. + Set minimum and maximum replicas and the length of the metrics window. Then + set the scale-down delay and how many concurrent requests one replica should + hold. [Kiely's book](https://www.baseten.co/inference-engineering/) covers + each in depth for inference. -- **Respect the cold-start wall.** A new GPU node must boot, pull a large engine image and load the weights. Loading weights alone can take half an hour for a large model. Cache weights on shared storage in each cluster so new replicas do not download them again. +- **Plan for slow cold starts.** A new GPU node must boot, pull a large engine + image and load the weights. Loading weights alone can take half an hour for a + large model. Cache weights on shared storage in each cluster so new replicas + do not download them again. -- **Scale-to-zero is harder than it sounds.** By the time a node is ready, the spike may be over, and scarce GPU capacity may not come back. +- **Be careful with scale-to-zero.** By the time a node is ready, the spike may + be over, and scarce GPU capacity may not come back. -- **Plan for multiple clusters at scale.** Past a few hundred GPUs, capacity pushes workloads across regions and providers. A control plane that treats separate pools as one avoids stranded capacity. +- **Plan for multiple clusters at scale.** Past a few hundred GPUs, capacity + pushes workloads across regions and providers. A control plane that treats + separate pools as one avoids leaving capacity idle in one pool while another + runs short. ### 3.10 Change management -- **Track a three-way version matrix:** model, hardware and engine build. New models often need a nightly or model-specific engine build on day zero. +- **Track a three-way version matrix:** model, hardware and engine build. New + models often need a nightly or model-specific engine build on day zero. -- **Plan rollouts for exclusive GPUs.** A GPU claim is exclusive, so a rolling update on a single replica deadlocks: the new pod cannot schedule and the old one never leaves. Use a Recreate strategy or allow one unavailable, and budget the downtime. +- **Plan rollouts for exclusive GPUs.** A GPU claim is exclusive, so a rolling + update on a one-replica deployment deadlocks: the new pod cannot schedule and + the old one never leaves. Use a Recreate strategy or allow one unavailable, + and budget the downtime. -- **Ready is not routable.** A gateway can lag a ready replica by a minute or two. +- **Ready doesn't mean routable.** A gateway can lag a ready replica by a minute + or two. -- **Test serving changes like code.** Make each change a pull request and change one thing at a time. Write the bar down first. Sweep load upward until a step misses the bar, and report capacity as one number with a confidence interval. +- **Test serving changes like code.** Make each change a pull request and change + one thing at a time. Write the bar down first. Sweep load upward until a step + misses the bar, and report capacity as one number with a confidence interval. -- **Test the test rig first.** A rig bug can waste a scarce GPU window that a few minutes on a laptop would have caught. +- **Test the test rig first.** A rig bug can waste a scarce GPU window that a + few minutes on a laptop would have caught. -- **Test on your own hardware with traffic like yours.** Agent prompts are long and mostly cached. A benchmark of short, fresh prompts measures a different system. +- **Test on your own hardware with traffic like yours.** Agent prompts are long + and mostly cached. A benchmark of short, fresh prompts measures a different + system. -Promote every serving change through tiers. Each tier costs more than the one before and catches a different kind of failure, so most problems show up before users see them: +Promote every serving change through tiers. Each tier costs more than the one +before and catches a different kind of failure, so most problems show up before +users see them: -- **Offline.** Validate the config and run the test rig's own checks. This takes minutes on a laptop and no GPU. +- **Offline.** Validate the config and run the test rig's own checks. This takes + minutes on a laptop and no GPU. -- **Rehearsal.** Run the full test end to end against a fake gateway or a small model, so rig bugs don't cost a GPU window. +- **Rehearsal.** Run the full test end to end against a fake gateway or a small + model, so rig bugs don't cost a GPU window. -- **Smoke.** Send a few real requests on the target hardware: a long prompt, a tool call, a streamed answer and a request that should be refused. +- **Smoke.** Send a few real requests on the target hardware. Include a long + prompt, a tool call and a streamed answer, plus one request that should be + refused. -- **Load with evals.** Sweep concurrency up to your bar with the eval suite running alongside, and compare against the current version on the same traffic. Evals can pass one request at a time while tool calls break under concurrent load, with nothing moving on the serving dashboards. +- **Load with evals.** Sweep concurrency up to your bar with the eval suite + running alongside, and compare against the current version on the same + traffic. Evals can pass one request at a time while tool calls break under + concurrent load, with nothing moving on the serving dashboards. -- **Canary.** Send a small share of real traffic to the new version and compare its server-side signals with the current one: TTFT, time per output token, errors, tool-call parse failures and length-limit stops. A gateway that routes by model name can split traffic by weight. Write the rollback trigger down before you start. +- **Canary.** Send a small share of real traffic to the new version and compare + its server-side signals with the current one: TTFT, time per output token, + errors, tool-call parse failures and length-limit stops. A gateway that routes + by model name can split traffic by weight. Write the rollback trigger down + before you start. -- **Full rollout.** Keep the old version ready to take traffic back until the new one's signals stay within the bar through a peak period. +- **Full rollout.** Keep the old version ready to take traffic back until the + new one's signals stay within the bar through a peak in traffic. ### 3.11 Quality and evals When an open model scores low, check three places before you blame the model: -1. **The harness.** Is the model the only thing that changed? Run your current hosted model first. If it scores below a weaker model, the harness is broken. +1. **The harness.** Is the model the only thing that changed? Run your current + hosted model first. If it scores below a weaker model, the harness is + broken. -2. **The infrastructure.** Did the model get a chance to answer? Timeouts, expired credentials and routing errors show up as model failures. Mark those runs "not measured." +2. **The infrastructure.** Did the model get a chance to answer? Timeouts, + expired credentials and routing errors show up as model failures. Mark those + runs "not measured." -3. **The grader.** Does the check measure the outcome you care about? Grade the end state, not the agent's description of it, and test your graders against answers people already accepted or rejected. +3. **The grader.** Does the check measure the outcome you care about? Grade the + end state, not the agent's description of it, and test your graders against + answers people already accepted or rejected. -Also: pin the data that decides the right answer, give every model the same context, keep test markers where the model cannot read them, run each test more than once, and prefer deterministic checks over LLM judges. A batching server can return different tokens for the same request even at temperature 0, so expect some variance unless the engine runs in a slower batch-invariant mode. +Also pin the data the right answer depends on and give every model the same +context. Keep test markers where the model cannot read them. Run each test more +than once, and prefer deterministic checks over using a model as the judge. A +batching server can return different tokens for the same request even at +temperature 0, so expect some variance unless the engine runs in a slower +batch-invariant mode. ### 3.12 Incidents and debugging -- **Expect silent failures.** The costly problems often throw no errors: a truncated stream returns 200, a metrics API reports available, node groups report ready. Part of the platform’s job is to make those failures loud. +- **Expect silent failures.** The costly problems often don't throw errors: a + truncated stream returns 200, a metrics API reports available, node groups + report ready. Part of the platform’s job is to make those failures loud. -- **Read the number.** A failure at a round number of seconds is a timeout. A multiple of it, such as 239 seconds after four 60-second tries, points to retries. +- **Look at the duration.** A failure at a round number of seconds is a timeout. + A multiple of it, such as 239 seconds after four 60-second tries, points to + retries. -- **Bypass a layer.** Take one hop out of the path and see whether the problem moves. +- **Bypass a layer.** Take one hop out of the path and see whether the problem + moves. -- **Verify status against reality.** A status is one component's view of its own job. Check the disk, the instance or the route itself. +- **Verify status against reality.** A status is one component's view of its own + job. Check the disk, the instance or the route itself. -- **Work additively, and upstream the fix.** Keep workarounds beside the platform, labelled for teardown, rather than editing objects a controller owns. A forked fix helps nobody else. +- **Work additively, and upstream the fix.** Keep workarounds beside the + platform, labelled for teardown, rather than editing objects a controller + owns. ### Checklist before your first users -- [ ] Use cases defined, with a written bar for time to first token, speed per user and error rate +- [ ] Use cases defined, with a written bar for time to first token, speed per + user and error rate -- [ ] Model chosen on your own evals, with your current hosted model as the baseline +- [ ] Model chosen on your own evals, with your current hosted model as the + baseline - [ ] Engine build confirmed to load the model on your hardware -- [ ] Memory budget covers weights, KV cache and runtime overhead, with max output tokens and max concurrent sequences set +- [ ] Memory budget covers weights, KV cache and runtime overhead, with max + output tokens and max concurrent sequences set -- [ ] GPU placement verified against the driver, with a fallback instance shape or zone +- [ ] GPU placement verified against the driver, with a fallback instance shape + or zone -- [ ] Auth policy applied before the gateway gets an address, and TLS on a real hostname +- [ ] Auth policy applied before the gateway gets an address, and TLS on a real + hostname - [ ] Credentials that last as long as an agent session -- [ ] Timeouts raised at every hop: route, load balancers, gateway retries, client and helper calls +- [ ] Timeouts raised at every hop: route, load balancers, gateway retries, + client and helper calls -- [ ] Engine metrics scraped, GPU SM and tensor activity graphed, and an alert when metrics stop arriving +- [ ] Engine metrics scraped, GPU SM and tensor activity graphed, and an alert + when metrics stop arriving -- [ ] Server-side counts of tool-call parse failures, length-limit stops and empty responses, so users aren't your first alert +- [ ] Server-side counts of tool-call parse failures, length-limit stops and + empty responses, so users aren't your first alert -- [ ] A probe matrix of requests that must be refused and paths that must stay open +- [ ] A probe matrix of requests that must be refused and paths that must stay + open - [ ] A rollout strategy for exclusive GPUs, with downtime budgeted - [ ] Previous container logs preserved after a crash -- [ ] Serving changes promoted through tiers, with evals under concurrent load and a canary with a written rollback trigger +- [ ] Serving changes promoted through tiers, with evals under concurrent load + and a canary with a written rollback trigger -- [ ] Workarounds kept beside the platform, labelled for teardown, with fixes sent upstream +- [ ] Workarounds kept beside the platform, labelled for teardown, with fixes + sent upstream ## 4. The stack -Learn the engine layer deeply. Learn the Kubernetes, gateway and observability layers you already know in their inference form. Learn the kernel layer well enough to read a profile. Most inference engineering happens at the higher levels of abstraction, but a working model of the layers next to yours guides every decision. +Go deep on the engine. You already know Kubernetes, gateways and observability, +so learn what changes for inference. For kernels, reading a profile is enough. +Most inference engineering happens higher up the stack. ### Tools by layer | Layer | Tools to know | Master first | |------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------| | GPUs on Kubernetes | [NVIDIA GPU Operator](https://github.com/NVIDIA/gpu-operator), [Dynamic Resource Allocation](https://kubernetes.io/docs/concepts/scheduling-eviction/dynamic-resource-allocation/) and its [NVIDIA driver](https://github.com/NVIDIA/k8s-dra-driver-gpu), [DCGM exporter](https://github.com/NVIDIA/dcgm-exporter) | How GPUs are advertised, requested and placed | -| Kernels and frameworks | [CUDA](https://developer.nvidia.com/cuda-toolkit), [PyTorch](https://pytorch.org), [Triton](https://github.com/triton-lang/triton), [FlashAttention](https://github.com/Dao-AILab/flash-attention), torch.compile | What a kernel is, kernel fusion, and how to read a profile | -| Inference engines | [vLLM](https://github.com/vllm-project/vllm), [SGLang](https://github.com/sgl-project/sglang), [TensorRT-LLM](https://github.com/NVIDIA/TensorRT-LLM); [Ollama](https://github.com/ollama/ollama) and [llama.cpp](https://github.com/ggml-org/llama.cpp) for local work | vLLM's core flags: memory utilization, max sequences, max model length, tensor parallel size, quantization, prefix caching, chunked prefill | +| Kernels and libraries | [CUDA](https://developer.nvidia.com/cuda-toolkit), [PyTorch](https://pytorch.org), [Triton](https://github.com/triton-lang/triton), [FlashAttention](https://github.com/Dao-AILab/flash-attention), torch.compile | Kernel basics, kernel fusion, and how to read a profile | +| Inference engines | [vLLM](https://github.com/vllm-project/vllm), [SGLang](https://github.com/sgl-project/sglang), [TensorRT-LLM](https://github.com/NVIDIA/TensorRT-LLM); [Ollama](https://github.com/ollama/ollama) and [llama.cpp](https://github.com/ggml-org/llama.cpp) for local work | vLLM's core flags: `--gpu-memory-utilization`, `--max-num-seqs`, `--max-model-len`, tensor parallel size, quantization, prefix caching, chunked prefill | | Distributed serving | [NVIDIA Dynamo](https://github.com/ai-dynamo/dynamo), [llm-d](https://github.com/llm-d/llm-d), [KServe](https://github.com/kserve/kserve), [LeaderWorkerSet](https://github.com/kubernetes-sigs/lws) | Multi-GPU and multi-node layouts, and prefill and decode split | | Gateway and routing | [Envoy Gateway](https://github.com/envoyproxy/gateway), [Agent Router](https://theagentrouter.ai) (formerly Envoy AI Gateway), [Gateway API Inference Extension](https://github.com/kubernetes-sigs/gateway-api-inference-extension) | Streaming, timeouts, auth policies, model-name and cache-aware routing | | Control plane | [Modelplane](https://github.com/modelplaneai/modelplane), built on [Crossplane](https://github.com/crossplane/crossplane) | Declaring clusters, hardware classes, deployments and caches across environments | @@ -427,25 +749,43 @@ Learn the engine layer deeply. Learn the Kubernetes, gateway and observability l ### Choosing an engine -The engine controls batching, memory management, kernel selection and API compatibility. It often matters more to cost and throughput than a hardware upgrade. +The engine controls batching, memory management, kernel selection and API +compatibility. It often matters more to cost and throughput than a hardware +upgrade. | Engine | Why teams pick it | What it costs to operate | |--------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------| | [vLLM](https://github.com/vllm-project/vllm) | The broadest model coverage, plus OpenAI and Anthropic APIs out of the box, so it is the usual first deployment. Hosted by the PyTorch Foundation. | Frequent releases with new flags. Pin versions and re-benchmark on every upgrade. | | [SGLang](https://github.com/sgl-project/sglang) | A radix-tree prefix cache and a scheduler that favors requests sharing a cached prefix, which suits agents and long multi-turn sessions. Upbound measured its biggest single gain here (Project Champagne, Part 5, 2026). | A smaller community, and new models may need nightly builds before a tagged release works. | | [TensorRT-LLM](https://github.com/NVIDIA/TensorRT-LLM) | Kernels tuned for NVIDIA hardware, including FP4 on Blackwell | NVIDIA hardware only, which ties your serving stack to one vendor. | -| [NVIDIA Dynamo](https://github.com/ai-dynamo/dynamo) | A layer above engines that coordinates many nodes, including split prefill and decode | Another distributed system to run. Worth it only at multi-node scale. | -| [Ollama](https://github.com/ollama/ollama), [llama.cpp](https://github.com/ggml-org/llama.cpp) | Easy local serving for learning and prototyping | Not designed for many concurrent users | - -Engines rise and fade. Hugging Face's TGI was a common choice and is now in maintenance mode. Check release cadence, maintainers and foundation backing before you commit to one. Engines ship new flags every few weeks. Re-run your benchmarks when you upgrade, rather than trusting last quarter's numbers. - -Upbound's own comparison in [Project Champagne, Part 5](https://www.upbound.io/blog/drinking-our-own-champagne-part-5-what-does-it-take-to-grow-a-self-hosted-coding-model-from-pilot-to-team) is a good example. The team was serving a coding model to about 10 engineers on two 8-GPU nodes and tested how to grow to a team of 15. On the same hardware and the same agent traffic, switching from vLLM to SGLang was the largest single gain it measured. Adding SGLang's host-memory cache on top added nothing measurable, because the KV cache never filled. +| [NVIDIA Dynamo](https://github.com/ai-dynamo/dynamo) | A layer above engines that coordinates many nodes, including split prefill and decode | Another distributed system to run, so use it only at multi-node scale. | +| [Ollama](https://github.com/ollama/ollama), [llama.cpp](https://github.com/ggml-org/llama.cpp) | Local serving for learning and prototyping | Not designed for many concurrent users | + +Hugging Face's TGI was a common choice and is now in maintenance mode, so check +release cadence, maintainers and foundation backing before you commit to one. +Engines add new flags every few weeks. Re-run your benchmarks when you upgrade, +rather than trusting last quarter's numbers. + +Upbound's own comparison in [Project Champagne, Part +5](https://www.upbound.io/blog/drinking-our-own-champagne-part-5-what-does-it-take-to-grow-a-self-hosted-coding-model-from-pilot-to-team) +shows this. The team was serving a coding model to about 10 engineers on two +8-GPU nodes and tested how to grow to a team of 15. On the same hardware and the +same agent traffic, switching from vLLM to SGLang was the largest single gain it +measured. Adding SGLang's host-memory cache on top didn't add a measurable gain, +because the KV cache never filled. ### Where Modelplane fits -[Modelplane](https://github.com/modelplaneai/modelplane) is an open source project from Upbound, built on [Crossplane](https://github.com/crossplane/crossplane) and released under the Apache 2.0 license. It orchestrates models, inference engines and AI infrastructure across clouds, neoclouds and your own GPUs, with Kubernetes-style declarative APIs. It provisions inference clusters or uses ones you bring. Documentation lives at [docs.modelplane.ai](https://docs.modelplane.ai). +[Modelplane](https://github.com/modelplaneai/modelplane) is an open source +project from Upbound, built on +[Crossplane](https://github.com/crossplane/crossplane) and released under the +Apache 2.0 license. It orchestrates models, inference engines and AI +infrastructure across clouds, neoclouds and your own GPUs, with Kubernetes-style +declarative APIs. It provisions inference clusters or uses ones you bring. +Documentation is at [docs.modelplane.ai](https://docs.modelplane.ai). -Modelplane splits the work into three layers of ownership. This split is a useful mental model even if you use different tools. +Modelplane splits the work into three layers of ownership. You can use the same +split as a mental model even with different tools. | Who | Owns | Modelplane resources | |---------------------------------------|-------------------------------------------------------|----------------------------------------------------| @@ -453,82 +793,149 @@ Modelplane splits the work into three layers of ownership. This split is a usefu | Model operator, or inference engineer | The models: deployments, services and weight caches | ModelDeployment, ModelService, ModelCache | | Modelplane | Everything in between, composed from the two above | ModelEndpoint, ModelReplica | -The hierarchy mirrors core Kubernetes one scope up. ModelDeployment → ModelReplica → ModelService → ModelEndpoint parallels Deployment → Pod → Service → Endpoint, across a fleet instead of within a single cluster. You bring the engine container and its flags, and Modelplane shapes the topology around them. People and agents never create any of these. They call the InferenceGateway by model name. +The hierarchy mirrors core Kubernetes one scope up. ModelDeployment → +ModelReplica → ModelService → ModelEndpoint parallels Deployment → Pod → Service +→ Endpoint, across a fleet instead of within one cluster. You bring the engine +container and its flags, and Modelplane shapes the topology around them. People +and agents never create any of these. They call the InferenceGateway by model +name. -Modelplane maps vLLM and SGLang metrics onto common names out of the box, and a MetricMapping adds any other engine, so different engines are judged on the same signals. +Modelplane maps vLLM and SGLang metrics onto common names out of the box, and a +MetricMapping adds any other engine, so different engines are judged on the same +signals. -For a platform engineer, the second layer is where the new learning is. Upbound found the same thing when it started self-hosting in [Project Champagne, Part 1](https://www.upbound.io/blog/drinking-our-own-champagne-what-it-takes-to-own-your-intelligence): the cluster and gateway work was familiar, and the model operator layer held the unknowns. +For a platform engineer, the second layer is where the new learning is. Upbound +found the same thing when it started self-hosting in [Project Champagne, Part +1](https://www.upbound.io/blog/drinking-our-own-champagne-what-it-takes-to-own-your-intelligence): +the cluster and gateway work was familiar, and the model operator layer held the +unknowns. ### A suggested order to learn the stack -1. Run vLLM on one GPU behind its OpenAI-compatible API, and read its metrics endpoint. +1. Run vLLM on one GPU behind its OpenAI-compatible API, and read its metrics + endpoint. -2. Deploy it on Kubernetes with the NVIDIA GPU Operator, and place it with device selectors. +2. Deploy it on Kubernetes with the NVIDIA GPU Operator, and place it with + device selectors. -3. Put a gateway in front of it with TLS, auth and timeouts sized for long streams. +3. Put a gateway in front of it with TLS, auth and timeouts sized for long + streams. -4. Scrape engine and GPU metrics into Prometheus and build the core panels from Section 3.8. +4. Scrape engine and GPU metrics into Prometheus and build the core panels from + Section 3.8. -5. Benchmark it with a concurrency sweep, and run a small eval suite at the same time. +5. Benchmark it with a concurrency sweep, and run a small eval suite alongside + it. 6. Serve the same model on SGLang and compare on your own traffic. -7. Split a larger model across GPUs with tensor and pipeline parallelism, then try a distributed serving layer. +7. Split a larger model across GPUs with tensor and pipeline parallelism, then + try a distributed serving layer. -8. Learn enough CUDA and profiling to understand why a kernel or a setting made the difference. +8. Learn enough CUDA and profiling to understand why a kernel or a setting made + the difference. ## 5. Learn from teams that share their work -Much of what you need to know about running inference isn’t in the docs. It’s in write-ups from teams who ran it and shared what broke. Reading those cases well is a skill worth building, and writing your own is one of the strongest ways to show what you know (Section 6). +Much of what you need to know about running inference comes from teams that +wrote up what broke. Learn to read those write-ups critically, and write your +own (Section 6). ### How to read a case study -- **Find the setup.** Model, hardware, engine version and traffic shape. Note the date, because numbers age fast. +- **Find the setup.** Model, hardware, engine version and traffic shape. Note + the date, because numbers age fast. - **Find the bar.** What counted as good enough, and who decided. -- **Follow the failures.** What broke, how the team found out, and how long it stayed hidden. +- **Follow the failures.** What broke, how the team found out, and how long it + took to detect. -- **Check the method.** Did they change one thing at a time and measure it on their own traffic? +- **Check the method.** Did they change one thing at a time and measure it on + their own traffic? -- **Separate what transfers.** A timeout lesson applies everywhere. A specific flag value applies to that engine release on that hardware. +- **Separate what transfers.** A timeout lesson applies everywhere, but a + specific flag value only holds for that engine release on that hardware. #### Case study: Project Champagne (2026) -Upbound, the company behind Crossplane and Modelplane, self-hosted chat, coding agents and a CVE-fixing build agent on open models with Modelplane, across AWS and Nebius GPUs. The team published what broke in a five-part series. Highlights, with numbers as of 2026: - -- **Serving was the easy part.** vLLM and scheduling worked. TLS, identity, timeouts, storage and observability took the time. (Part 2) -- **Quota did not mean capacity.** Approved quota for 768 vCPUs of H100 instances never produced one, and the planned GPU arrived about half the time. One model ended up split across two nodes in two zones. (Part 2) -- **“It fits” was a startup-time claim.** A chat model that fit cleanly ran out of memory nine times in four days. Lowering the memory fraction from 0.92 to 0.88, enabling expandable segments and capping concurrent sequences at 16 freed 4.6 GiB. (Part 2) -- **The expensive failures were silent.** A truncated stream returned 200, a metrics API reported available with no data, and monitoring sat pending for six days. (Part 2) -- **Timeouts stacked up.** A 15-second route default, then a 60-second load balancer idle timeout, cut off answers that were still streaming. (Parts 2 and 3) -- **Agents were the hardest clients.** Hour-long sessions outlived 60-minute tokens, and a reasoning model thought for more than a minute before its first visible token. (Parts 2 and 3) -- **Low scores were often not the model’s fault.** A fake shell in the test harness scored Claude 0. A 60-second idle timeout failed a model that completed every run once the limit was raised. (Part 3) -- **“Slow” had shapes.** An engineer’s “wow, is it slow” traced back to hidden maximum-effort reasoning while every dashboard looked fine. One GPU showed 99% “utilization” with tensor cores busy about 10% of the time. (Part 4) -- **Load depended on the client.** Agent prompts had a median of 59K tokens, about two-thirds hit the prefix cache, and short-prompt sessions kept more requests in flight by launching subagents. (Part 5) -- **The engine mattered more than expected.** On the same two 8-GPU nodes, moving from vLLM to SGLang was the largest single gain. A 128K context cap only looked faster because it rejected long conversations. (Part 5) +Upbound, the company behind Crossplane and Modelplane, self-hosted chat, coding +agents and a CVE-fixing build agent on open models with Modelplane, across AWS +and Nebius GPUs. The team published what broke in a five-part series. +Highlights, with numbers as of 2026: + +- **Serving was the smaller problem.** vLLM and scheduling worked. TLS, + identity, timeouts, storage and observability took the time. (Part 2) +- **Quota did not mean capacity.** Approved quota for 768 vCPUs of H100 + instances never produced one, and the planned GPU arrived about half the time. + One model ended up split across two nodes in two zones. (Part 2) +- **Fitting at startup wasn't enough.** A chat model that fit cleanly ran out of + memory nine times in four days. Lowering the memory fraction from 0.92 to + 0.88, enabling expandable segments and capping concurrent sequences at 16 + freed 4.6 GiB. (Part 2) +- **The expensive failures were silent.** A truncated stream returned 200, a + metrics API reported available with no data, and monitoring sat pending for + six days. (Part 2) +- **Timeouts stacked up.** A 15-second route default, then a 60-second load + balancer idle timeout, cut off answers that were still streaming. (Parts 2 and + 3) +- **Agents were the hardest clients.** Hour-long sessions ran past the 60-minute + lifetime of their tokens, and a reasoning model thought for more than a minute + before its first visible token. (Parts 2 and 3) +- **Low scores were often not the model’s fault.** A fake shell in the test + harness scored Claude 0. A 60-second idle timeout failed a model that + completed every run once the limit was raised. (Part 3) +- **Slowness had several causes.** An engineer’s “wow, is it slow” traced back + to hidden maximum-effort reasoning while every dashboard looked fine. One GPU + showed 99% `GPU-Util` with tensor cores busy about 10% of the time. (Part 4) +- **Load depended on the client.** Agent prompts had a median of 59K tokens, + about two-thirds hit the prefix cache, and short-prompt sessions kept more + requests in flight by launching subagents. (Part 5) +- **The engine mattered more than expected.** On the same two 8-GPU nodes, + moving from vLLM to SGLang was the largest single gain. A 128K context cap + only looked faster because it rejected long conversations. (Part 5) ### Read the full series -Each post on the Upbound blog tells these stories in detail, with the configs, queries and numbers behind them: +Each post on the Upbound blog tells these stories in detail, with the configs, +queries and numbers behind them: + + +- [**Part 1: What it takes to own your + intelligence.**](https://www.upbound.io/blog/drinking-our-own-champagne-what-it-takes-to-own-your-intelligence) + Why Upbound self-hosts inference, and how the use cases picked the models and + hardware. -- [**Part 1: What it takes to own your intelligence.**](https://www.upbound.io/blog/drinking-our-own-champagne-what-it-takes-to-own-your-intelligence) Why Upbound self-hosts inference, and how the use cases picked the models and hardware. +- [**Part 2: Finding the + corks.**](https://www.upbound.io/blog/drinking-our-own-champagne-part-2-finding-the-corks) + The inference lifecycle stage by stage, from pick and place to change and + fail. -- [**Part 2: Finding the corks.**](https://www.upbound.io/blog/drinking-our-own-champagne-part-2-finding-the-corks) The inference lifecycle stage by stage, from pick and place to change and fail. +- [**Part 3: The build security + agent.**](https://www.upbound.io/blog/drinking-our-own-champagne-part-3-use-case-1-the-build-security-agent) + How to tell whether an open model can do a real job as well as a hosted one. -- [**Part 3: The build security agent.**](https://www.upbound.io/blog/drinking-our-own-champagne-part-3-use-case-1-the-build-security-agent) How to tell whether an open model can do a real job as well as a hosted one. +- [**Part 4: Candling the + bottles.**](https://www.upbound.io/blog/drinking-our-own-champagne-part-4-candling-the-bottles) + What the team could see at each step, what it missed, and how it plans to + alert. -- [**Part 4: Candling the bottles.**](https://www.upbound.io/blog/drinking-our-own-champagne-part-4-candling-the-bottles) What the team could see at each step, what it missed, and how it plans to alert. +- [**Part 5: From pilot to + team.**](https://www.upbound.io/blog/drinking-our-own-champagne-part-5-what-does-it-take-to-grow-a-self-hosted-coding-model-from-pilot-to-team) + A repeatable way to test serving changes and size capacity for a growing team. -- [**Part 5: From pilot to team.**](https://www.upbound.io/blog/drinking-our-own-champagne-part-5-what-does-it-take-to-grow-a-self-hosted-coding-model-from-pilot-to-team) A repeatable way to test serving changes and size capacity for a growing team. + -## 6. Career advice: making the move +## 6. Making the career move -You do not need to start over. Keep your platform skills, add the inference concepts, and prove both in public. The field is young enough that newcomers can become experts quickly. +You don't need to start over. Add inference concepts to the platform skills you +have, and show the work in public. The field is young, so newcomers can catch up +fast. ### Pick a direction -Pick the direction closest to the work you already do. Most platform engineers start in the first two and grow toward the third. +Pick the direction closest to the work you already do. Most platform engineers +start in the first two and grow toward the third. | Direction | Focus | Good fit if you | |-----------------------|-------------------------------------------------------------------------|--------------------------------------------------| @@ -538,23 +945,35 @@ Pick the direction closest to the work you already do. Most platform engineers s ### Follow a learning path -Vishakha Sadhwani's *AI Inference Engineering Roadmap 2026* is a thorough 13-stage map of the field. Coming from platform engineering, you can move through it faster, because you already own much of the infrastructure half. A shorter path that plays to that head start: +Vishakha Sadhwani's *AI Inference Engineering Roadmap 2026* is a thorough +13-stage map of the field. Coming from platform engineering, you can move +through it faster, because you already own much of the infrastructure half. A +shorter path: -1. **Learn the model, not just the container.** Transformer basics, attention and the KV cache, dense versus MoE models, and how to read a model config. The goal is to estimate the memory any model needs in five minutes. +1. **Learn the model, not just the container.** Transformer basics, attention + and the KV cache, dense versus MoE models, and how to read a model config. + The goal is to estimate the memory any model needs in five minutes. -2. **Learn one engine deeply.** Pick vLLM, learn what each core flag trades away, and measure time to first token and per-token latency yourself. +2. **Learn one engine deeply.** Pick vLLM, learn what each core flag trades + away, and measure time to first token and per-token latency yourself. -3. **Translate your platform skills to GPUs.** Device allocation, gateways that hold long streams, autoscaling on inference signals, and observability for engines. This is mostly translation, not new learning. +3. **Translate your platform skills to GPUs.** Device allocation, gateways that + hold long streams, autoscaling on inference signals, and observability for + engines. -4. **Learn to judge quality and cost.** Build an eval, run a capacity sweep and calculate cost per million tokens. This is where platform engineers most often have a gap, and where the role earns its keep. +4. **Learn to judge quality and cost.** Build an eval, run a capacity sweep and + calculate cost per million tokens. This is where platform engineers most + often have a gap, and where the role adds the most value. ### Build projects that prove it -Each project below fits on a rented GPU or two, and each produces a repo with real numbers. Shut instances down between runs, and use spot capacity for batch experiments. +Each project below fits on a rented GPU or two, and each produces a repo with +real numbers. Shut instances down between runs, and use spot capacity for batch +experiments. | Project | What it proves | What to publish | |---------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------|--------------------------------------------------------------| -| Serve an 8B model with vLLM on one GPU. Sweep 1, 10 and 50 concurrent users. | You can measure TTFT, time per output token and throughput honestly | Configs, the load script, charts and cost per million tokens | +| Serve an 8B model with vLLM on one GPU. Sweep 1, 10 and 50 concurrent users. | You can measure TTFT, time per output token and throughput accurately | Configs, the load script, charts and cost per million tokens | | Predict how many 32K-token sessions fit on your GPU, then test it | You understand weights, KV cache and runtime overhead | Your prediction, the result and why they differ | | Compare FP16 and FP8, and a 4-bit format if you can | You can trade quality against speed with evidence | Eval scores next to latency and throughput | | Run the same prompts with shared content first, then last | You understand prefix caching and prompt structure | Cache hit rate and TTFT for each | @@ -565,25 +984,51 @@ Each project below fits on a rented GPU or two, and each produces a repo with re ### Show your work in public -Many inference roles fill through networks before they reach a careers board. In August 2026, [Zeoli found](https://www.datagravity.dev/p/the-rise-of-inference-engineering) one serving company valued at \$17.5B listing no dedicated inference performance openings. Public work is how people outside those networks get found. +Many inference roles fill through networks before they reach a careers board. In +August 2026, [Zeoli +found](https://www.datagravity.dev/p/the-rise-of-inference-engineering) one +serving company valued at \$17.5B that didn't list any dedicated inference +performance openings. Public work is how people outside those networks get +found. -- **Write READMEs like lab reports.** State the question, the setup (model, GPU, engine version and flags), the method, the results with numbers, what surprised you, and how to reproduce it. +- **Write READMEs like lab reports.** State the question, the setup (model, GPU, + engine version and flags), the method, the results with numbers, what + surprised you, and how to reproduce it. -- **Publish what failed.** The Champagne posts are useful precisely because they show the corks. A clear account of a failure shows more skill than a clean demo. +- **Publish what failed.** A clear account of a failure shows more skill than a + clean demo. -- **Pin versions and show cost.** Results without an engine version and a price are hard to trust and harder to reuse. +- **Pin versions and show cost.** Results without an engine version and a price + are hard to trust and harder to reuse. -- **Write and speak about it.** Turn each project into a short post. Submit talks to meetups, CNCF events and community live streams. +- **Write and speak about it.** Turn each project into a short post. Submit + talks to meetups, CNCF events and community live streams. ### Contribute to open source -Much of what inference projects still need is platform work, which plays to your strengths. Upbound's own upstream list from Project Champagne reads like a platform backlog: configurable load balancers, certificate issuance, Recreate rollouts for exclusive devices, an honest Ready signal, engine metrics scraped out of the box, and auth that suits long-running agents. - -Projects worth watching include [vLLM](https://github.com/vllm-project/vllm), [SGLang](https://github.com/sgl-project/sglang), [llm-d](https://github.com/llm-d/llm-d), [KServe](https://github.com/kserve/kserve), [NVIDIA Dynamo](https://github.com/ai-dynamo/dynamo), [Agent Router](https://theagentrouter.ai) (formerly Envoy AI Gateway), the [Gateway API Inference Extension](https://github.com/kubernetes-sigs/gateway-api-inference-extension), [Modelplane](https://github.com/modelplaneai/modelplane) and [Crossplane](https://github.com/crossplane/crossplane). Good first contributions: - -- **Run it and report what breaks,** with a reproduction and numbers. This is how Champagne contributed most. - -- **Write operator documentation:** deployment, tuning and troubleshooting guides drawn from your own runs. +Much of what inference projects still need is platform work. Upbound's own +upstream list from Project Champagne reads like a platform backlog: configurable +load balancers, certificate issuance, Recreate rollouts for exclusive devices, +an honest Ready signal, engine metrics scraped out of the box, and auth that +suits long-running agents. + +Projects to follow include [vLLM](https://github.com/vllm-project/vllm), +[SGLang](https://github.com/sgl-project/sglang), +[llm-d](https://github.com/llm-d/llm-d), +[KServe](https://github.com/kserve/kserve), [NVIDIA +Dynamo](https://github.com/ai-dynamo/dynamo), [Agent +Router](https://theagentrouter.ai) (formerly Envoy AI Gateway), the [Gateway API +Inference +Extension](https://github.com/kubernetes-sigs/gateway-api-inference-extension), +[Modelplane](https://github.com/modelplaneai/modelplane) and +[Crossplane](https://github.com/crossplane/crossplane). Good first +contributions: + +- **Run it and report what breaks,** with a reproduction and numbers. This is + how Champagne contributed most. + +- **Write operator documentation:** deployment, tuning and troubleshooting + guides drawn from your own runs. - **Add metrics, dashboards and alerts** that make silent failures loud. @@ -591,118 +1036,208 @@ Projects worth watching include [vLLM](https://github.com/vllm-project/vllm), [S - **Contribute benchmark recipes and tests** for the hardware you have. -Read each project's contributing guide, start with a small change, and comment on an issue before taking on a large one. +Read each project's contributing guide, start with a small change, and comment +on an issue before taking on a large one. -### Keep your skills current +### Ongoing learning -This field moves faster than most. Engines add flags every few weeks, and research reaches production within months. +Engines add flags every few weeks, and research reaches production within +months. -- **Read release notes** for the engine you run, every release, and re-run your benchmark when you upgrade. +- **Read release notes** for the engine you run, every release, and re-run your + benchmark when you upgrade. -- **Read the foundational papers:** [Orca](https://www.usenix.org/conference/osdi22/presentation/yu) on continuous batching, [PagedAttention](https://arxiv.org/abs/2309.06180), [DistServe](https://arxiv.org/abs/2401.09670) on disaggregation, the original [speculative decoding](https://arxiv.org/abs/2211.17192) paper, and [FlashAttention](https://arxiv.org/abs/2205.14135). +- **Read the core papers:** + [Orca](https://www.usenix.org/conference/osdi22/presentation/yu) on continuous + batching, [PagedAttention](https://arxiv.org/abs/2309.06180), + [DistServe](https://arxiv.org/abs/2401.09670) on disaggregation, the original + [speculative decoding](https://arxiv.org/abs/2211.17192) paper, and + [FlashAttention](https://arxiv.org/abs/2205.14135). -- **Read Philip Kiely's [*Inference Engineering*](https://www.baseten.co/inference-engineering/).** The e-book is free from Baseten. +- **Read Philip Kiely's [*Inference + Engineering*](https://www.baseten.co/inference-engineering/).** The e-book is + free from Baseten. -- **Follow practitioners.** Newsletters such as [The Pragmatic Engineer](https://newsletter.pragmaticengineer.com), [ByteByteGo](https://blog.bytebytego.com) and [Data Gravity](https://www.datagravity.dev) cover the field well. Join the community channels for the engines you use. +- **Follow practitioners.** Newsletters such as [The Pragmatic + Engineer](https://newsletter.pragmaticengineer.com), + [ByteByteGo](https://blog.bytebytego.com) and [Data + Gravity](https://www.datagravity.dev) cover the field well. Join the community + channels for the engines you use. -- **Learn evals.** Whether an open model can replace a hosted one is decided by evals, and few engineers can build good ones. Start with Anthropic's [*Demystifying evals for AI agents*](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents) and Hamel Husain and Shreya Shankar's [evals FAQ](https://hamel.dev/blog/posts/evals-faq/). +- **Learn evals.** Whether an open model can replace a hosted one is decided by + evals, and few engineers can build good ones. Start with Anthropic's + [*Demystifying evals for AI + agents*](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents) + and Hamel Husain and Shreya Shankar's [evals + FAQ](https://hamel.dev/blog/posts/evals-faq/). -- **Learn the economics.** Be able to explain cost per million tokens and cost per user session to a finance partner. +- **Learn the economics.** Be able to explain cost per million tokens and cost + per user session to a finance partner. ### Find the roles -Don't expect a consistent job title yet. The same work shows up as inference engineer, performance engineer, model serving engineer or ML infrastructure engineer, and sometimes simply as AI platform engineer, so search for all of them. Pay reflects how scarce the skills are: in August 2026, [Zeoli found](https://www.datagravity.dev/p/the-rise-of-inference-engineering) an Anthropic inference performance role posted at \$350,000 to \$850,000, and an OpenAI equivalent at \$295,000 to \$555,000. +Don't expect a consistent job title yet. The same work shows up as inference +engineer, performance engineer, model serving engineer or ML infrastructure +engineer, and sometimes as AI platform engineer, so search for all of them. Pay +reflects how scarce the skills are: in August 2026, [Zeoli +found](https://www.datagravity.dev/p/the-rise-of-inference-engineering) an +Anthropic inference performance role posted at \$350,000 to \$850,000, and an +OpenAI equivalent at \$295,000 to \$555,000. -You may not need to change jobs to get started, though. If your company is planning to self-host its first model, put your hand up. That project will probably land on your team anyway, and running it is the most direct experience you can get. +You may not need to change jobs to get started, though. If your company is +planning to self-host its first model, put your hand up. Your team will probably +end up responsible for that project anyway, and running it is the most direct +experience you can get. ### Prepare for interviews -Practice answering these. Each one points to the section of this guide that covers it. Several are adapted from Lamhot Siagian's Efficient LLM Inference: Interview Pocket Notes. +Practice answering these. Each one points to the section of this guide that +covers it. Several are adapted from Lamhot Siagian's interview pocket notes on +efficient inference (Section 7). -- **Prefill and decode.** Why is decode usually limited by memory bandwidth rather than compute, and what changes at long context? (Section 2) +- **Prefill and decode.** Why is decode usually limited by memory bandwidth + rather than compute, and what changes at long context? (Section 2) -- **Hardware trends.** If compute keeps growing faster than memory bandwidth, will inference soon become compute-bound? (Section 2) +- **Hardware trends.** If compute keeps growing faster than memory bandwidth, + will inference soon become compute-bound? (Section 2) -- **Memory math.** How much memory do a model and its KV cache need at a given context length and concurrency? (Section 2) +- **Memory math.** How much memory do a model and its KV cache need at a given + context length and concurrency? (Section 2) - **Goodput.** What is goodput, and why plan capacity with it? (Section 2) -- **Prefix caching.** Which applications benefit most from prefix caching? (Section 2) +- **Prefix caching.** Which applications benefit most from prefix caching? + (Section 2) -- **Precision.** You need to serve a 70B model on two 80 GB GPUs. What precision would you choose, and why? (Sections 2 and 3.4) +- **Precision.** You need to serve a 70B model on two 80 GB GPUs. What precision + would you choose, and why? (Sections 2 and 3.4) -- **Quantization.** When does weight-only quantization speed things up, and when does it not? (Section 3.4) +- **Quantization.** When does weight-only quantization speed things up, and when + does it not? (Section 3.4) -- **Speculative decoding.** When would you not use speculative decoding? (Section 3.4) +- **Speculative decoding.** When would you not use speculative decoding? + (Section 3.4) -- **Chunked prefill.** How do long prompts slow other users' streams, and what does chunked prefill change? (Section 3.4) +- **Chunked prefill.** How do long prompts slow other users' streams, and what + does chunked prefill change? (Section 3.4) -- **Parallelism.** Which parallelism strategy would you pick for a given model and GPU type, and when does tensor parallelism beat pipeline parallelism? (Sections 3.4 and 3.5) +- **Parallelism.** Which parallelism strategy would you pick for a given model + and GPU type, and when does tensor parallelism beat pipeline parallelism? + (Sections 3.4 and 3.5) -- **Profiling.** How would you tell whether a workload is limited by compute or by memory bandwidth? (Section 3.8) +- **Profiling.** How would you tell whether a workload is limited by compute or + by memory bandwidth? (Section 3.8) -- **Debugging "slow".** P99 time to first token just spiked. How do you tell queued, thinking and genuinely slow apart? (Section 3.8) +- **Debugging "slow."** p99 time to first token just spiked. How do you tell + queueing, hidden reasoning and slow generation apart? (Section 3.8) -- **Cost.** What does a million tokens cost on your GPUs, and how does that compare with a hosted API? (Section 2) +- **Cost.** What does a million tokens cost on your GPUs, and how does that + compare with a hosted API? (Section 2) -- **Capacity planning.** Your user count will grow 10x in six months. How do you plan capacity? (Section 3.10) +- **Capacity planning.** Your user count will grow 10x in six months. How do you + plan capacity? (Section 3.10) -- **Rollouts and SLOs.** How would you roll out a new model version safely, and what SLO would you set for the service? (Sections 3.8 and 3.10) +- **Rollouts and SLOs.** How would you roll out a new model version safely, and + what SLO would you set for the service? (Sections 3.8 and 3.10) -- **Failure stories.** Tell one failure story with real numbers: what broke, how you found it, and what you changed. +- **Failure stories.** Describe one failure with real numbers. Cover what broke + and how you found it, then what you changed. ## 7. Sources and further reading + + ### Upbound and Modelplane -- Upbound, *Drinking Our Own Champagne* series on the Upbound blog: [Part 1: What it takes to own your intelligence](https://www.upbound.io/blog/drinking-our-own-champagne-what-it-takes-to-own-your-intelligence); [Part 2: Finding the corks](https://www.upbound.io/blog/drinking-our-own-champagne-part-2-finding-the-corks); [Part 3: Use case \#1, the build security agent](https://www.upbound.io/blog/drinking-our-own-champagne-part-3-use-case-1-the-build-security-agent); [Part 4: Candling the bottles](https://www.upbound.io/blog/drinking-our-own-champagne-part-4-candling-the-bottles); [Part 5: What does it take to grow a self-hosted coding model from pilot to team?](https://www.upbound.io/blog/drinking-our-own-champagne-part-5-what-does-it-take-to-grow-a-self-hosted-coding-model-from-pilot-to-team) +- Upbound, *Drinking Our Own Champagne* series on the Upbound blog: [Part 1: + What it takes to own your + intelligence](https://www.upbound.io/blog/drinking-our-own-champagne-what-it-takes-to-own-your-intelligence); + [Part 2: Finding the + corks](https://www.upbound.io/blog/drinking-our-own-champagne-part-2-finding-the-corks); + [Part 3: Use case \#1, the build security + agent](https://www.upbound.io/blog/drinking-our-own-champagne-part-3-use-case-1-the-build-security-agent); + [Part 4: Candling the + bottles](https://www.upbound.io/blog/drinking-our-own-champagne-part-4-candling-the-bottles); + [Part 5: What does it take to grow a self-hosted coding model from pilot to + team?](https://www.upbound.io/blog/drinking-our-own-champagne-part-5-what-does-it-take-to-grow-a-self-hosted-coding-model-from-pilot-to-team) -- Viktor Farcic, *LLM Inference Explained: 12 Concepts You Actually Need to Know*. +- Viktor Farcic, *LLM Inference Explained: 12 Concepts You Actually Need to + Know*. -- [Modelplane on GitHub](https://github.com/modelplaneai/modelplane) and [Modelplane documentation](https://docs.modelplane.ai). +- [Modelplane on GitHub](https://github.com/modelplaneai/modelplane) and + [Modelplane documentation](https://docs.modelplane.ai). ### Books and articles -- Philip Kiely, [*Inference Engineering*](https://www.baseten.co/inference-engineering/), Baseten Books, 2026. Free as an e-book. +- Philip Kiely, [*Inference + Engineering*](https://www.baseten.co/inference-engineering/), Baseten Books, + 2026. Free as an e-book. -- Gergely Orosz & Philip Kiely, [*What is inference engineering? Deepdive*](https://newsletter.pragmaticengineer.com/p/what-is-inference-engineering), The Pragmatic Engineer, March 31, 2026. +- Gergely Orosz & Philip Kiely, [*What is inference engineering? + Deepdive*](https://newsletter.pragmaticengineer.com/p/what-is-inference-engineering), + The Pragmatic Engineer, March 31, 2026. -- ByteByteGo, [*A Guide to AI Inference Engineering*](https://blog.bytebytego.com/p/a-guide-to-ai-inference-engineering), June 15, 2026. +- ByteByteGo, [*A Guide to AI Inference + Engineering*](https://blog.bytebytego.com/p/a-guide-to-ai-inference-engineering), + June 15, 2026. -- Chris Zeoli, [*The Rise of Inference Engineering*](https://www.datagravity.dev/p/the-rise-of-inference-engineering), Data Gravity, August 13, 2026. +- Chris Zeoli, [*The Rise of Inference + Engineering*](https://www.datagravity.dev/p/the-rise-of-inference-engineering), + Data Gravity, August 13, 2026. -- Spheron, [*What Is Inference Engineering? The 2026 GPU Cloud Guide*](https://www.spheron.network/blog/inference-engineering-guide-2026/), updated October 19, 2026. +- Spheron, [*What Is Inference Engineering? The 2026 GPU Cloud + Guide*](https://www.spheron.network/blog/inference-engineering-guide-2026/), + updated October 19, 2026. -- Vishakha Sadhwani, [*AI Inference Engineering Roadmap 2026: Complete Learning Path*.](https://www.tech5ense.com/p/the-ai-inference-engineering-roadmap) +- Vishakha Sadhwani, [*AI Inference Engineering Roadmap 2026: Complete Learning + Path*.](https://www.tech5ense.com/p/the-ai-inference-engineering-roadmap) -- Charity Majors, Liz Fong-Jones and George Miranda, [*Observability Engineering*](https://www.oreilly.com/library/view/observability-engineering/9781492076438/), O'Reilly. +- Charity Majors, Liz Fong-Jones and George Miranda, [*Observability + Engineering*](https://www.oreilly.com/library/view/observability-engineering/9781492076438/), + O'Reilly. -- Lamhot Siagian, Efficient LLM Inference: Interview Pocket Notes, second edition, 2026. +- Lamhot Siagian, Efficient LLM Inference: Interview Pocket Notes, second + edition, 2026. ### Evals and benchmarks -- Anthropic, [*Demystifying evals for AI agents*](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents), January 2026. +- Anthropic, [*Demystifying evals for AI + agents*](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents), + January 2026. -- Hamel Husain and Shreya Shankar, [*AI Evals FAQ*](https://hamel.dev/blog/posts/evals-faq/). +- Hamel Husain and Shreya Shankar, [*AI Evals + FAQ*](https://hamel.dev/blog/posts/evals-faq/). - UK AI Security Institute, [Inspect](https://inspect.aisi.org.uk). - Hugging Face, [upskill](https://github.com/huggingface/upskill). -- SemiAnalysis, [AgentX methodology and datasets](https://inferencex.semianalysis.com/agentx/methodology). +- SemiAnalysis, [AgentX methodology and + datasets](https://inferencex.semianalysis.com/agentx/methodology). ### Papers -- Yu et al., [Orca: A Distributed Serving System for Transformer-Based Generative Models](https://www.usenix.org/conference/osdi22/presentation/yu), OSDI 2022. +- Yu et al., [Orca: A Distributed Serving System for Transformer-Based + Generative Models](https://www.usenix.org/conference/osdi22/presentation/yu), + OSDI 2022. + +- Kwon et al., [Efficient Memory Management for Large Language Model Serving + with PagedAttention](https://arxiv.org/abs/2309.06180), SOSP 2023. -- Kwon et al., [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://arxiv.org/abs/2309.06180), SOSP 2023. +- Zhong et al., [DistServe: Disaggregating Prefill and Decoding for + Goodput-optimized Large Language Model + Serving](https://arxiv.org/abs/2401.09670), OSDI 2024. -- Zhong et al., [DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving](https://arxiv.org/abs/2401.09670), OSDI 2024. +- Leviathan et al., [Fast Inference from Transformers via Speculative + Decoding](https://arxiv.org/abs/2211.17192), ICML 2023. -- Leviathan et al., [Fast Inference from Transformers via Speculative Decoding](https://arxiv.org/abs/2211.17192), ICML 2023. +- Dao et al., [FlashAttention](https://arxiv.org/abs/2205.14135), 2022, and Shah + et al., [FlashAttention-3](https://arxiv.org/abs/2407.08608), 2024. -- Dao et al., [FlashAttention](https://arxiv.org/abs/2205.14135), 2022, and Shah et al., [FlashAttention-3](https://arxiv.org/abs/2407.08608), 2024. +- Shoeybi et al., [Megatron-LM: Training Multi-Billion Parameter Language Models + Using Model Parallelism](https://arxiv.org/abs/1909.08053), 2019. -- Shoeybi et al., [Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism](https://arxiv.org/abs/1909.08053), 2019. +- Shazeer et al., [Outrageously Large Neural Networks: The Sparsely-Gated + Mixture-of-Experts Layer](https://arxiv.org/abs/1701.06538), 2017. -- Shazeer et al., [Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer](https://arxiv.org/abs/1701.06538), 2017. \ No newline at end of file + diff --git a/docs/utils/vale/.vale.ini b/docs/utils/vale/.vale.ini index 23f79887b..5d5aff789 100644 --- a/docs/utils/vale/.vale.ini +++ b/docs/utils/vale/.vale.ini @@ -55,5 +55,5 @@ alex.ProfanityMaybe = NO alex.ProfanityUnlikely = NO alex.Suicide = NO -TokenIgnores = (\(#.*\)),(\]\(),(http.*),(\!\[),(\d.*px),(\d\.\d\.\d),(\{\{<[^>]+>\}\}) +TokenIgnores = (\(#.*\)),(\]\(),(https?://[^\s)]+),(\!\[),(\d.*px),(\d\.\d\.\d),(\{\{<[^>]+>\}\}) BlockIgnores = (\{\{<\s*\/?.*>\}\}),(\{\{\<\/\*.*\*\/\>\}\}) diff --git a/docs/utils/vale/styles/Modelplane/Spelling.yml b/docs/utils/vale/styles/Modelplane/Spelling.yml index d49c98b91..435c2c779 100644 --- a/docs/utils/vale/styles/Modelplane/Spelling.yml +++ b/docs/utils/vale/styles/Modelplane/Spelling.yml @@ -12,3 +12,4 @@ filters: - '#\w+' - \d+\.\d+[BbMmKkTt] - \w+'s + - '\w+(-\w+)+' diff --git a/docs/utils/vale/styles/config/vocabularies/Modelplane/accept.txt b/docs/utils/vale/styles/config/vocabularies/Modelplane/accept.txt index 6ea9f82b9..e06475704 100644 --- a/docs/utils/vale/styles/config/vocabularies/Modelplane/accept.txt +++ b/docs/utils/vale/styles/config/vocabularies/Modelplane/accept.txt @@ -200,12 +200,12 @@ TTFT A[0-9]+B IQ4_XS llama\.cpp -multi-GPU +[Mm]ulti-GPU [Vv][Rr][Aa][Mm] -tensor +[Tt]ensor tokenizer [Ii]nference -prefill +[Pp]refill [Cc]loud # Cloud providers @@ -324,8 +324,8 @@ stdin stdout subnets? tolerations -pipeline -parallelism +[Pp]ipeline +[Pp]arallelism [Aa]utoscaling config args @@ -352,7 +352,7 @@ base64-encoded bring-your-own built-in call-outs -cache-aware +[Cc]ache-aware cloud-native distributed-systems performance-sensitive @@ -372,7 +372,7 @@ double-check dry-run dynamic-PVC e\.g\. -end-to-end +[Ee]nd-to-end EKS-based event-driven first-class @@ -402,7 +402,7 @@ multi-pod multi-region multi-tenant multi-tenancy -mixture-of-experts +[Mm]ixture-of-experts namespace-scoped namespaced namespaces? @@ -431,7 +431,7 @@ resource-intensive self-contained self-service self-signed -server-side +[Ss]erver-side serving-stack [Ss]ingle-node single-pod @@ -462,8 +462,8 @@ workload-cluster [Rr]e-converge[sd]? # Style exceptions -several -above +[Ss]everal +[Aa]bove below the cloud disabled? @@ -550,7 +550,7 @@ routable pipeline-parallel Qwen2.5-14B -agent +[Aa]gent requirement PrefillDecode @@ -810,12 +810,12 @@ TTFT A[0-9]+B IQ4_XS llama\.cpp -multi-GPU +[Mm]ulti-GPU [Vv][Rr][Aa][Mm] -tensor +[Tt]ensor tokenizer [Ii]nference -prefill +[Pp]refill [Cc]loud # Cloud providers @@ -934,8 +934,8 @@ stdin stdout subnets? tolerations -pipeline -parallelism +[Pp]ipeline +[Pp]arallelism [Aa]utoscaling config args @@ -962,7 +962,7 @@ base64-encoded bring-your-own built-in call-outs -cache-aware +[Cc]ache-aware cloud-native distributed-systems performance-sensitive @@ -982,7 +982,7 @@ double-check dry-run dynamic-PVC e\.g\. -end-to-end +[Ee]nd-to-end EKS-based event-driven first-class @@ -1012,7 +1012,7 @@ multi-pod multi-region multi-tenant multi-tenancy -mixture-of-experts +[Mm]ixture-of-experts namespace-scoped namespaced namespaces? @@ -1041,7 +1041,7 @@ resource-intensive self-contained self-service self-signed -server-side +[Ss]erver-side serving-stack [Ss]ingle-node single-pod @@ -1072,8 +1072,8 @@ workload-cluster [Rr]e-converge[sd]? # Style exceptions -several -above +[Ss]everal +[Aa]bove below the cloud disabled? @@ -1160,7 +1160,7 @@ routable pipeline-parallel Qwen2.5-14B -agent +[Aa]gent requirement PrefillDecode @@ -1212,3 +1212,62 @@ per-deployment opt-out Framebuffer Mooncake + +# Inference field guide +DeepSeek +R1 +o1 +SRE +SLOs? +[Gg]oodput +FinOps +Ollama +benchmarked +dtypes? +CUDA +p50 +p90 +declaratively +[Ss]harding +agentic +subagents? +GPTQ +[Ee]vals? +[Qq]uantizing +SXM +datacenter +B200 +RTX +safetensors +repos? +ServiceMonitor +teardown +torch\.compile +FlashAttention +PagedAttention +DistServe +Pyrra +GuideLLM +SemiAnalysis +AgentX +upskill +FP4 +TGI +\d+(\.\d+)?x +vCPUs? +[Cc]onfigs +READMEs +runbooks? +Zeoli +Kiely +ByteByteGo +AIPerf +Viktor +Farcic +Vishakha +Sadhwani +Hamel +Husain +Shreya +Lamhot +Siagian From a8526f1d7f4db5c9d7778c9c6608b03653441f61 Mon Sep 17 00:00:00 2001 From: Rae Sharp Date: Wed, 28 Oct 2026 13:17:39 -0400 Subject: [PATCH 3/5] Update image serving and vale error on unrelated doc Signed-off-by: Rae Sharp --- docs/content/guides/inference_guide.md | 2 +- docs/content/{guides => media}/inference-stack.png | Bin docs/content/models/routing.md | 3 +++ 3 files changed, 4 insertions(+), 1 deletion(-) rename docs/content/{guides => media}/inference-stack.png (100%) diff --git a/docs/content/guides/inference_guide.md b/docs/content/guides/inference_guide.md index fde95c7a9..5f6f48d24 100644 --- a/docs/content/guides/inference_guide.md +++ b/docs/content/guides/inference_guide.md @@ -135,7 +135,7 @@ You rarely choose them directly. Above them sits the **engine**. Above that, a **control plane** deploys and manages engines, because sooner or later you run more than one. On top, a **gateway** routes traffic across models and replicas. -![Inference stack: four layers and the clients they serve](inference-stack.png) +{{< img src="/media/inference-stack.png" alt="Inference stack: four layers and the clients they serve" >}} *Inference stack: four layers and the clients they serve* diff --git a/docs/content/guides/inference-stack.png b/docs/content/media/inference-stack.png similarity index 100% rename from docs/content/guides/inference-stack.png rename to docs/content/media/inference-stack.png diff --git a/docs/content/models/routing.md b/docs/content/models/routing.md index 2dd901eb8..0106c6d04 100644 --- a/docs/content/models/routing.md +++ b/docs/content/models/routing.md @@ -175,8 +175,11 @@ The sidecar and the picker must match your engines: | vLLM with `MooncakeConnector` | `mooncake` | Any | | SGLang, any transfer backend | `sglang` | `always-disagg-pd-decider`. An SGLang decode engine rejects a request that skips prefill. | + + The sidecar's `--vllm-port` names the decode engine's port, whatever the engine is. + A decode pod without the sidecar sends every request to decode alone. vLLM serves it, and SGLang rejects it. From feaaec40404a2f0aaf4058b3e4627c38cb52acbc Mon Sep 17 00:00:00 2001 From: Annie Talvasto <46491425+AnnieTalvasto@users.noreply.github.com> Date: Wed, 28 Oct 2026 13:42:20 -0400 Subject: [PATCH 4/5] Fix wording for sidecar's vllm-port description Signed-off-by: Annie Talvasto <46491425+AnnieTalvasto@users.noreply.github.com> --- docs/content/models/routing.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/content/models/routing.md b/docs/content/models/routing.md index 0106c6d04..ae05862ac 100644 --- a/docs/content/models/routing.md +++ b/docs/content/models/routing.md @@ -177,7 +177,7 @@ The sidecar and the picker must match your engines: -The sidecar's `--vllm-port` names the decode engine's port, whatever the engine +The sidecar's `--vllm-port` specifies the decode engine's port, whatever the engine is. From e564715c4e7f393a56cbc5d889ed6db801c8b489 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?rae=20sharp=20=E2=99=AF?= <8883519+tr0njavolta@users.noreply.github.com> Date: Wed, 28 Oct 2026 14:10:25 -0400 Subject: [PATCH 5/5] Update names to specifies MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: rae sharp ♯ <8883519+tr0njavolta@users.noreply.github.com> --- docs/content/models/routing.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/content/models/routing.md b/docs/content/models/routing.md index ae05862ac..869fa0bed 100644 --- a/docs/content/models/routing.md +++ b/docs/content/models/routing.md @@ -222,7 +222,7 @@ and no other container declares it. A `Worker` member doesn't need a container on 8000, because worker pods don't receive requests. Behind the sidecar, the decode port appears in three places: the `--port` flag -on the engine, its `http` port, and the sidecar flag that names the engine's +on the engine, its `http` port, and the sidecar flag that specifies the engine's port. Keep the three the same. Modelplane doesn't read engine args to check them.