From 5a458036a23fd2c3342994bf8aaa7d6525b0a18a Mon Sep 17 00:00:00 2001 From: Ivan Kozlovic Date: Wed, 22 May 2019 18:11:42 -0600 Subject: [PATCH] Major rewrite of NATS Streaming Server concepts section and updates to the developing section. Signed-off-by: Ivan Kozlovic --- SUMMARY.md | 41 +++ developer/resources/ring_buffer.png | Bin 25282 -> 0 bytes developer/resources/start_positions.png | Bin 23794 -> 0 bytes developer/streaming/README.md | 10 +- developer/streaming/acks.md | 2 +- developer/streaming/connecting.md | 15 +- developer/streaming/durables.md | 8 +- developer/streaming/embedding.md | 111 ++++++ developer/streaming/protocol.md | 265 +++++++++++++++ developer/streaming/publishing.md | 10 +- developer/streaming/queues.md | 34 +- developer/streaming/receiving.md | 28 +- nats_streaming/channels/channels.md | 7 + nats_streaming/channels/message-log.md | 5 + .../channels/subscriptions/durable.md | 9 + .../channels/subscriptions/queue-group.md | 11 + .../channels/subscriptions/redelivery.md | 11 + .../channels/subscriptions/regular.md | 3 + .../channels/subscriptions/subscriptions.md | 13 + nats_streaming/client-connections.md | 9 + .../clustering/auto-configuration.md | 17 + nats_streaming/clustering/clustering.md | 9 + nats_streaming/clustering/configuration.md | 95 ++++++ nats_streaming/clustering/containers.md | 9 + nats_streaming/clustering/supported-stores.md | 9 + .../fault-tolerance/active-server.md | 7 + nats_streaming/fault-tolerance/failover.md | 9 + nats_streaming/fault-tolerance/ft.md | 15 + .../fault-tolerance/shared-state.md | 3 + .../fault-tolerance/standby-server.md | 3 + nats_streaming/gettingstarted/configuring.md | 3 + nats_streaming/gettingstarted/install.md | 81 +++++ nats_streaming/gettingstarted/intro.md | 6 + .../gettingstarted/process-signaling.md | 27 ++ nats_streaming/gettingstarted/run.md | 96 ++++++ .../tls.md} | 0 .../gettingstarted/windows-service.md | 28 ++ .../{nats-streaming-intro.md => intro.md} | 16 +- nats_streaming/monitoring/enabling.md | 24 ++ nats_streaming/monitoring/endpoints.md | 319 ++++++++++++++++++ nats_streaming/monitoring/monitoring.md | 3 + nats_streaming/nats-streaming-install.md | 102 ------ nats_streaming/nats-streaming-protocol.md | 190 ----------- nats_streaming/nats-streaming-quickstart.md | 82 ----- nats_streaming/partitioning.md | 119 +++++++ nats_streaming/relation-to-nats.md | 12 + nats_streaming/store-encryption.md | 33 ++ nats_streaming/store-interface.md | 15 + .../{nats-streaming-swarm.md => swarm.md} | 0 49 files changed, 1521 insertions(+), 403 deletions(-) delete mode 100644 developer/resources/ring_buffer.png delete mode 100644 developer/resources/start_positions.png create mode 100644 developer/streaming/embedding.md create mode 100644 developer/streaming/protocol.md create mode 100644 nats_streaming/channels/channels.md create mode 100644 nats_streaming/channels/message-log.md create mode 100644 nats_streaming/channels/subscriptions/durable.md create mode 100644 nats_streaming/channels/subscriptions/queue-group.md create mode 100644 nats_streaming/channels/subscriptions/redelivery.md create mode 100644 nats_streaming/channels/subscriptions/regular.md create mode 100644 nats_streaming/channels/subscriptions/subscriptions.md create mode 100644 nats_streaming/client-connections.md create mode 100644 nats_streaming/clustering/auto-configuration.md create mode 100644 nats_streaming/clustering/clustering.md create mode 100644 nats_streaming/clustering/configuration.md create mode 100644 nats_streaming/clustering/containers.md create mode 100644 nats_streaming/clustering/supported-stores.md create mode 100644 nats_streaming/fault-tolerance/active-server.md create mode 100644 nats_streaming/fault-tolerance/failover.md create mode 100644 nats_streaming/fault-tolerance/ft.md create mode 100644 nats_streaming/fault-tolerance/shared-state.md create mode 100644 nats_streaming/fault-tolerance/standby-server.md create mode 100644 nats_streaming/gettingstarted/configuring.md create mode 100644 nats_streaming/gettingstarted/install.md create mode 100644 nats_streaming/gettingstarted/intro.md create mode 100644 nats_streaming/gettingstarted/process-signaling.md create mode 100644 nats_streaming/gettingstarted/run.md rename nats_streaming/{nats-streaming-tls.md => gettingstarted/tls.md} (100%) create mode 100644 nats_streaming/gettingstarted/windows-service.md rename nats_streaming/{nats-streaming-intro.md => intro.md} (81%) create mode 100644 nats_streaming/monitoring/enabling.md create mode 100644 nats_streaming/monitoring/endpoints.md create mode 100644 nats_streaming/monitoring/monitoring.md delete mode 100644 nats_streaming/nats-streaming-install.md delete mode 100644 nats_streaming/nats-streaming-protocol.md delete mode 100644 nats_streaming/nats-streaming-quickstart.md create mode 100644 nats_streaming/partitioning.md create mode 100644 nats_streaming/relation-to-nats.md create mode 100644 nats_streaming/store-encryption.md create mode 100644 nats_streaming/store-interface.md rename nats_streaming/{nats-streaming-swarm.md => swarm.md} (100%) diff --git a/SUMMARY.md b/SUMMARY.md index eb642bc..3949c59 100644 --- a/SUMMARY.md +++ b/SUMMARY.md @@ -129,6 +129,47 @@ * [Durable Subscriptions](developer/streaming/durables.md) * [Queue Subscriptions](developer/streaming/queues.md) * [Acknowledgements](developer/streaming/acks.md) +* [Embedding NATS Streaming Server](developer/streaming/embedding.md) +* [Writing your own client library](developer/streaming/protocol.md) + +## NATS Streaming Server + +* [Concepts](nats_streaming/intro.md) + * [Relation to NATS](nats_streaming/relation-to-nats.md) + * [Client Connections](nats_streaming/client-connections.md) + * [Channels](nats_streaming/channels/channels.md) + * [Message Log](nats_streaming/channels/message-log.md) + * [Subscriptions](nats_streaming/channels/subscriptions/subscriptions.md) + * [Regular](nats_streaming/channels/subscriptions/regular.md) + * [Durable](nats_streaming/channels/subscriptions/durable.md) + * [Queue Group](nats_streaming/channels/subscriptions/queue-group.md) + * [Redelivery](nats_streaming/channels/subscriptions/redelivery.md) + * [Store Interface](nats_streaming/store-interface.md) + * [Store Encryption](nats_streaming/store-encryption.md) + * [Clustering](nats_streaming/clustering/clustering.md) + * [Supported Stores](nats_streaming/clustering/supported-stores.md) + * [Configuration](nats_streaming/clustering/configuration.md) + * [Auto Configuration](nats_streaming/clustering/auto-configuration.md) + * [Containers](nats_streaming/clustering/containers.md) + * [Fault Tolerance](nats_streaming/fault-tolerance/ft.md) + * [Active Server](nats_streaming/fault-tolerance/active-server.md) + * [Standby Servers](nats_streaming/fault-tolerance/standby-server.md) + * [Shared State](nats_streaming/fault-tolerance/shared-state.md) + * [Failover](nats_streaming/fault-tolerance/failover.md) + * [Partitioning](nats_streaming/partitioning.md) + * [Monitoring](nats_streaming/monitoring/monitoring.md) + * [Enabling](nats_streaming/monitoring/enabling.md) + * [Endpoints](nats_streaming/monitoring/endpoints.md) + +* [Getting started](nats_streaming/gettingstarted/intro.md) + * [Installing](nats_streaming/gettingstarted/install.md) + * [Running](nats_streaming/gettingstarted/run.md) + * [Configuring](nats_streaming/gettingstarted/configuring.md) + * [Securing](nats_streaming/gettingstarted/tls.md) + * [Process Signaling](nats_streaming/gettingstarted/process-signaling.md) + * [Windows Service](nats_streaming/gettingstarted/windows-service.md) + +* [Docker Swarm](nats_streaming/swarm.md) ## NATS Streaming Server diff --git a/developer/resources/ring_buffer.png b/developer/resources/ring_buffer.png deleted file mode 100644 index 44fd040b25ddf13f687e7c25894389d95f03c4c6..0000000000000000000000000000000000000000 GIT binary patch literal 0 HcmV?d00001 literal 25282 zcmc$`WmH_j(l&|=8f8|doE_te|ccRo(WKod`k)fcVQ03*M)S;kYD50RBLlEJCZ*Yly{sMnN zyQ<4dKvj(s9|52BY&CV=bd{9^%$*!qOf8(uELl7qAiyXnC?QV);G=`3n<=@cgT14x zfTu9!zYzk!=hv@UDarp0akCSq)Kyj^mvnNmBS_t`R{b# zU&53&Zf+0(R@RRnKeBw}WN~t_W@YE+=VxW(VCCRo21YQudO5n8dNMn@QvHX>|Im@L zbTxOeg}B)|Ig-E9H8pc`cN3;KXQ zrV71&E1>FPYYD9UN?(Ls=-e~ir5;*6 zC@3)~c`0#CPw2CbH>t#PUKj75vU8^nO);$AqOx`2BfS;FU&5{B`%u-@gTNPTBAo!9 zGTC~AwvQOx5eP@z-Hilc14}K6K`DabNOAFrPfaMB30W2?@hrtDsUyh?qmsTkSAv4z zJ{=uAy~t`6ew|yJ5uQo#{(G;#p*dG&=o7HvbU*w2;li*DMN%9jCHDXQDVHMGEcQ+N zc8%DVKg~3hLH?AOKrSu^uda|8B~?LT%!~D6PLDT7F=mkZ+mn>Un*3Zo66@A_tzKdP zC+Uf=gdB#F9440v<1Sm8gN_VZfc!H#b7RV5%?sf!$9sd?wRe0=Xtp74lG%hd=ET#e zwGp*^kn(4@UyX}tW8cY!uuAB_7|^c|L-xo{1a3S<` zk)qx0+p`y=zTfY?X^-z9Mrk?HCXFV&X+o9NlcWEW0_$FTv(4Sox)I8!SRGRnz-%G7 z>oBA)S4I&ZIy=&8#e1hCiLWhIhk>>xguZ0v=r!8Kd|SW0QZv!zIsfLwnm5#T@w)Jz z+yK#WQ%}ZDiv+f#>%rF&YVc@kn>P|a{s^h*j*-M)gq(#f?kta*50HemLwWVan(x^0 zQ)y=@Nm>UT`J->SD^h4@M&!H=J;=0r6S}~*_5l%=B(8~K{&w{j;n%%8Q$$Ik&?67Y z%IifE>}5z_J3Z3@+>rKl=7Rp8nKBbnZNl_^zk#t}-5|82{>a#@3&){0jIY!Y>M$pv z3*+oqoC$OLA}8TqoN-#XC&{ix%jB`v-bNDwMc>7(wDp%|1(_W!s{RinT>}wu1VoI; z_hcb8c-_c~wY{#+j6JJOgAEk%gZ7t;JEmc9i>7}SID&E#FaCW325E+bkb)9c@alJ<)^J|?6H{2J0ygvGYe3fow zf4V}hZQ{NB^cBa{ac7Z3Re+RN+*6nYLjZ9P)J8HJ&~Ucxy#H)quZQckVROC_eJ13Z zcZD(`QL0L82FKx#C$XYOh726R>!YRtWAD3|_$|;ryheNlxwc3Bl#RtvI`i?anWyjS zK}62c6BfMUkThYQXUn|Rf`h>`i<17m2cDakNg9-c;-eb}yKl)oeCOKo*aohp=EWRz zz}SB$_tNBGn^~2(NYdP#6|W*!+~a6bFQ97W201hlLZMJS9;~mAcuY4!sQrPIOjIQU znpyI3cbPfVeYNKg3SA$tKj@btSI|R?Fb5F1knT!r)DYN=R&}SV4*TG<_n5pe*LJth zq_Uc>z;CJbkYIdmBT;7((%apAraVv(wp`%8|CCBF72?p;lGj!{y}09ZhM=`98tB_3 zjw^APcq_I{t|RVEo{lz~;GRgtM9Y%*++yp_^iLqBJpcL(86d*?=r0m=c~+Ek3s~i9e-VXy1Wbn=C3eGZ zm{=w=C~j{p5pUFdpNuy_5B{Fs&Hgl~1KL?5llAO-ysdfFGaf}eMc%U$d2Z|SEo~cI zA`yi)`QRSLv2@PPS6wA%U2PlLW4_QoGydlZ5sSSF#5F?51jJnTI_7q8=huIcBP|6T z3Y0G-Dh0Z$B#=P+l&E^^Ak&-_AAXbtswYQ{$`^5B0T#$%s`vi!p~aJzaTFdn_rPyt zd*Ve0a3Dd1Sj{+lmTw}T@{-Y@h`JgoZ(?8eyxBfvlvl-*iL`4#N!ZEJBNV_SYX>%xj zHkH$C8#9x%JD9#x7y3I%xLp4s*|u7qwr%2r79jde(L(Iip8rrBGXCpjv^=3v+&bI6 zYXdVtXC5Q3nB?VScKOSP!}MzZO1|8Kmtz{vHDr4nVwrs6>UGm{#T-l|p=h@iaxw{) z;7eO;^j(6X_ZW<)gX1k2>s&`_m#eJg8WetsI^?1eY<~qquY5U_#EnASvK|-lnOqS< zEw5<0HuTU#Z9T**yG}$9jslf9sqRfzP=9B6`LZ@ve}23}b@yJvb*8?fU5P(yn|%tT zkDZvy`z?x8z>FeDQ6gE2=U|aq$7)}|Nq#}B&4D3LEzK>g#dl6B-_aK7tDkzI5R08X-Imd}fBfn%>jtdfie&ztDywD~Xc!+_ad!a}4$R>u2w!%>O5& z0x#x6(B!(AQYMS>q0}z+&{(1mxgf4yVE90`xWQK z1FwB8rkYN@gx|mDJaWdP0udtv5g#(U6c)#6?Ra{$ZFe@Fbk5l0fU8IgW5u;$7vgae@`e1841uE4Cfk*H@9 z-n4fFelEKA+*CI7cZUh7)4;sIz`2+Sf9y^RPGSUwc5l>L5Js=@ai1Gg%#pSvngl*x z1lwqct|eQpL`k@SYk)v#x57Qm-Pr2oC$(-uDuPGp?lwdrsV~6sKMtMkZIE5k_D2jJ zI&7XhbjefI&i9aZA~W;~XOo2YK|4&ljy{y9oU8*A>^olT3cTo##EvHDVNb1=R)oh^ z=p)Wub|p(xV|+aRyi40l`TOB~l%B#(Ev9gkp68v}SoQ`4s!xKWe#^5~Sc)5M7}fyY zF1sc%KDTZ$+KF)w^141z=OT0e)%|cDsmoRGl_(ER$1vvjHfO|{XVM;ln&7>_&gjUH z#Fj*!&-UFS)#jbr>q;X}JQ|*;)qc?bdsft7L@FE~^u_!-Xj5}n#y({GK#%tb`yTEp ztH{LG6!xj7LGkcAP8?MHyLEH1=0aidlXDo1La`(R?@_AW*JQYHN;p7}M=zScPMY0gUj*Egw1Dy;M|2S*!8(8RZzoFTly8DO(@ zo!4*zbV9~*hlzSBdIemglHG6B%E$r(qhGCL0kIu9#6zqT$!jE)wr1@OKiMzqL>Jfy zYL6wBZT8+ENunLaMI(fxFr~0bXFWtOP!G~@wPa;|`~33Yd6$J&?gXCtZ^{U8#!8AA zi3WN_Eb8J$7VSl@kaKKZ6(gk8aVqS_41k*F^!ZDoRL?fo2E2{fVU(cVNbI058bI9c zr0E(67e;v`7iE{fly8FjmF+b;=8w2zCBEp%9LN;JwjTXC-Y8&n!3^!oBE^*Yk@s z^hRQte0PE`{bzj}xy3!#t|elxg&eVZRa~>we_(!>6R5Fwb3?L@zYb^6O{lU9UKFXe zM9_xxN8A}vqpNm%+8u1R`HnT**QW8MrU@z$E9#y*eK|)Xr5|`bGI|+V0oKuI>JqjEOwayb%k=|LkQp zN~}g@uUr>gDj1|h!^uL7CaU#Oy6C?#E07pWrqj8hd~tU6#L0&E#d??K$M?jie+ZWq z#zpE_+GuY+M+`{S8SqhzlE$p6FAfhg^*)CEmq6kx0qs_$=TtErorPR6atLa;?fgd% zk6)Zy|MjfgZ^>ng!fUirIgu*gH=WZ?p0+$cKM!V?71U$CCUO^;iB#Kbt!b8)7&bOG z&2T?g@R;@8+zB%1%l?V83lRw<4`s<= z_(AGrv%HgRMv)~qUi>9)bU-RV3H=wnVKc5?n3_h1gA5B`XJvb&ZX2!2V9Qlwj zRgBT)oAj7;-qja{rIVVxo?v_VEjML7;N^)(^!IMc29s`$Nty`d;h?d&|7Uj5BCU8i zf1YR(C9(J+6H)=pAaM|z@ved!>qoOltMPm+0X`*@llgCP0oS|%GY{)N7lu*c2OFD- ze{Yki6_?Rz@x2dp-!AdGaTG(ap#S3X>1hd5I+oW995KiiscdRh3L=_x%CVAwS2Y(* za=_!3YD1DxPM!EGjSfV&tF9B=olkcw8C&1JGyx=0Y5uBAr1+{~FAj7)bmhyDcm|2R ziGBEnhmwRkM-3=Zt8ZEx0zhzRgw33paAb~@t)!f$3!44;DDS!Kcb0W7LD-r74P8g@ zdQk?&RXJ|uXpUSyj>0Y9JNBbpCP16BwaBIi3P5+~Yd(T)pRPiGtQIf`D137haLBi| zK=Do;XuldDvhEdM0$J~pA-7|GYXQZVcSO_AEU&CoEPP}PpJ^lb3BGb%8?HmfJUl;F zDUc1VzFF5EhDgH2c(`kCTAG{>Et0)F4DJN{9x=}Kyxi=*FpspOwJ5vg!JH1I@T=m{ z97d-!K{G!;co_r4ixDIQ1gG7gLOSq}ER-adunpUfFeJ~76yB|gr73SMs3Xq*;s}LS3M$(qg@MTM zY};P^Rimp%);B&P9n1Mb-+GeJ^PWdm`4bmCr`>#InoT`NB6?Q?mi}Glr}Kyz>u<~) z7!giTB=Nm10oDTTBJI{^%MER#cE_DtCnH-n545OB;Vr<#+MDV&BX_kxCyAfnGFVSu zTzc8QxCWY>69p(NHYYLIW*pUP5Qd+vTh5SOPIXVDH!hau4sSl0DT zO++(al@-mOl8oaCxsS1SuQ{?~tZN+1bLq^Pcy^i-9(e3xo`U#Ng)t&p=o}y>4>uiH zb#JcR6g()`1Xs{YmIwYEJVM921v(ySWaJv)zxjg36!e{PPRn$P0~1_qoq^~dqNkad29KG8;u?+sy3@`tz3o=sYF4BN=Ar`oQ{ zeh{%oJ&ZXEJeRvivtR8m{+lrB@W5`dkaxJj`DeuM3))}5QxhIrVq-g5n}=ncI|U^r zQe6;+aHQK1wp2bH^m7UW;K@?up@b=%3ZOesRd-L=kA4w1sSHboOohYy@DQH!H9GsU zrwOCMqVmRnB=?7HypaAfm4p9*&rE_IKUJ!#p<#{>&=mB-!Oa}Gx4%%i)K={A8Vj=n znn*Gl?2a2?@!rA1jnqm!(@Gv@`=YLR98RIFeB5mQ3^TZkN9C?mlRo_Gu+J^v;c&Dx zd`BHMs-xgBe>5!w``1tCY~j1e_B||D%Y8(oAus`jnBloGjOVf+HtJOv7LAfgh$}3S zaLL4*O=9?)?NJ#?ux$0P`axH9PuSyVECa2ud^m# zG?Sg|OOi1fLd3P`FZe@nh-B)pPn|-N6^T2Q&u~tzW+$mms^u6Swcu*&-7z2dY?k>!51&f?^-OJpe>7~MTbYJ!wElRp-f<9$rGg*# zOUKNyKjf6UZakw-n$+h?|5DH9ui+JFj1+@ufCul6CDbniUSjF8H0!B$=%G7)DDW&R zu=eF34k1o+6TN(mNDKKyzo@%CQ=8BJ{gt3l74BXc$~~4^LgeF#8t!Zslw0EI9+Th!9~iPUqeLv=yQc!br1qsNTMA+flS4sIuGZU_MM{ zs^)q1VypMxBaW=;F0F0c@Iar`19BFS3S_z~r?|M6AQ0JIF!7j0>H=)I1=qW|lQaX< z$K6Ox{L;E>T4+8h1HKc*OjZ{`!UNHuUWgBM1P$8-Rs7!!v+=>=#ZEM9YvyoKmI@1k zp{7$P1e?kB$U08QZ@4A3Kj&X()Thj>q~aCfMSo{pmR0t8^T+nzRP?l%Ux{u;uYY4$ ztYV4~2QUI#{t+gaP73Peoe*w(IBrRupKD^c>R2z@{^UqQbJ<7A6hDZ_Jo(G})mb04 z!YlOD%<*?Wf$}6#EK*Xs=DUuX>GCO0Z?l{30DDhFq!JwE)1Z`xoW>p$eoE5P5&bM6 z%4-9dgcWg9YwmLmotKY^!9D}N5R+`ElU}~hqrSlg60HSlxN&Ez^UgqdyVXgkg1FN( zs>84IAwzwsGACyBpdg|Z!Nl3-M&WAmm&8d2wzz5&n|3R+OZV9D0t#&@1~aRlf=DAi1B4zftivAq^IuC z-E+nl62!>Uy+Ci$rTI_QKXf@!;eQzS&Hh98AZ#!bz5cyv%d@KgPMq}{XK(?*ZDi`7 zFKVsOFqVIbqB0R9#VBBWKU3o)8;Bmq&R!|aOy+WNkim~%v|E|JFR;7R6ngWEDLWAl za(x~S>q1GxxlT`8gu!mz_x)ga2qdyGsqO6)S?sk#5M>^l?+knV2lvji;J`%Gr)v?* z>aI>#eyzQjs!&U7%^-w(-cXp58@_70T|Q9JHzHnvtPgWz$A%)~!Le)j)6WiZf4w_{ zwz=$G#6fd7Vhom$Ct)synB%9r7(>xBbn$%lD;qa;njRh35!6HGH(`UtGg0W6I_A^t z838{?R8F7ptMp_du;Y(TBHPF$U8>VQH-$_$nW#6=R0xzxWLHJwa=Ch=%8L$SZNlha z(JG&kJZ+$nIv5*ren!D3bBr%G-@2S7dxo3|Lc~~t*_*n5=YMxmr60)?T=yhnXtud3 zRE@2i;**%!2>$MzE{uLx-c6y09I?Cd5O*r>^pp9)gp>qp0rbqeNtGzS__eX0yFoiZ z`qsQGfJ6>91FK8pbo#P+)bES+JxPk4!bVli)CSgi(r>=zHwRzygIWj}?VsQB=EO^P zQJV1V8&titCFV&wiuaDYe;_8?v~Swt?S%W2d+2FdoNtb36CY__E)m_FF2?4gW6v<~ zd>y(dZ`&&eR|3~N@OTK#Z>S@AoEoQu z7uiJt(`~caIq>r*d+}wYiULuPjmI{Q4%2h5`ZQx4;x1L!7U3vot1vdQbMw9g%7T4_ zcTv_U%B%SZ-w6yMNBO^TfDH)t7(BVeEG3 zw<7^J&1K$vV}y2%i->?pI(-p7@97k4sj&O{%xOdHDcs0P_<>KQqHMl_L0E^+Tgs}3 zKo7k)x3^+@qipcK{$bYdz&wUYYh+*8(V{jRRk>xoxj6o%m`LFkE3p;h;De9Jj z6c$0$opG1%j%1jSUX)pDlbCO5yR>w-K#&IQXO$*2U#ybw56+gl*mx;e>XL_~^bDt9 zL5pxi%`~XC_Nl$Ue63wS?B2o#tlz8j<)d)Ma>UxxRvGe-p15Zr?oyiwKxY(KZ-2Np z_uklE;qk)K;lnh}nK1O_>`X03#2*ys}V7*VN@JsytfX$!*Y00VsS_i z3GVO@Jl$UB19m@nFXKLcC}+DA5ER*ul&?R~`cAxeCJaOP>B=Z)xDXl47Gc&%*~a7H zHG))zp=+su1(P}S2iIGAJQz04Qnu$k9+NKHw@+~9s7dle#v^SySaq`3=~UZO(P?^A zR9yx5Lh_x^dNH)^BV6ag9J#HnBA~@LFbIDXQlEwSkl*uJXfP{pUe=R4@l(#hoNoE< zMS3>7SnB@<@nMI2zExP&Bn$M#QB6Zn&x@(TfVKt)az4Gq6&FO%Hw@Xh_htfb7}%n2 z)z5&Z2r1LxPeT6gMn2>xTbkK|9-h!T9RgP^|hl|Itv&wKo(|C7(Z)aoGOz!7vI5gNe;+BUc)^xMi>{SIDKEg(P zT{z)9F)TeNzotBmFD-H1lw^ zvvF~B^fUEy!t`TCATHOn)7s;*ix4!+;Mo02AOt!)F{d+}V#iN45UYW`@)wMl8~TAD z_o3ct2-bAq$x&c?g3$8G)iE|53=X-NHApg@m%O56D;A>nK!GRH zJ2}d_73y2Mo`|d&i2fZY{PZf^r~Gu27?t;2fw&qL)!K>9b=s34LI3^nmw-6-f+Ar| zlAj79@rnK5STwQG>8<6le!y$NXu|tGY%Uus_!2wzj6Ltt*R&}XJ4gY%xhw#YIS>4D zsf9e?^hr+~hmL11;xOu_nbwggu=fNqwOQafZlBOCTPHY5{!-Bh&N4~!oTNW1{? zsQm;HQp-W-N59LVtmJNZD{mTs$ri!u!+!NhU$GFfVcS!ldB9a)3}5dk#Oa(0W^;5d zMN%Ir0AdSYy;;%xC1O79t_#ZG>ZrQ}WM`GP-M}pkKu;nt`xPBKG%DH6(FY+@UA=J{ zSnO1pBy!_MNhVw~u@OODzpHezfzU%P`<4u+za<$ecYI&$DuVpLHU~pTyCBtr54zJm z9HK{)k`@2bGzSq=iK{0)My=I!65yN*eR{DZv15dh1Se8)U{mCLgbRv~kI%})s>2-~ zmsO4IB-x49Pf2!s+qI=K+R@5{!GX)0lZ1~mWGhXHrJl#$leM!|+P7wt=T;>gl%1z` zhNBim$)*mI-HVFp5EF=qO^XpF)rlQz`S=BE#tVx_MOo(31md%v z5{(^14AjG10uLt*L8A?*4|9_%K-uw<^P?@RgLIc4i0>UY(NN`wPr3Mpa*v13Wwk*y z?69uVU-1;ac!FyHzf&t`b0Pra8<$v(C2{XQ$?QM0yq&7OF1l)n3_R z&Vq^EJR*)E;`Zqkh#5!baM2C?7&)835OrXK4YpL5A?P7AOz-hMJHa_j<-ZcJvVA|?FBC?W zxKU^=uu^|0t3)zZs429iiWwVk{F;2&9V`~0;NryiNjSBmA@dfqb*|V-@r)27kxCtQ zh@sF0WPdF|`puwfSRN#mj}P6MgNp)JP2?_f{9tzUD<>u=nb;Bpt5YXSz2Vg#1^9y} zcmqkXYY77lE%jM*(51(+a#B>;Q}?^DjZ}IIj(H;X9_BI=%Bhd$1zgehyg@ZT#xibI zsA}Aff>W@fHa~@jhrgMx)SIzBvL1iq6?rXDV>ATKX!IVqy6_3(!P`vLM#&6a$B9ha z21P2Cx>-FY8}&_I6KgWn_Sqlz9QPw#4`p)8(alp|x)*ZPO&Enwc-yVzsXe5WfjYZ` z>2_$|Y%7XnOx{16;$AdRKTEB^=zugLF!YZ#^z@4>6Vin z2b+KSS8dw(tNc|*nX~Cg6j(@z#tU*m+T7a8GGTkb+|Lx!K5tOCAOke{uf{@r<4rUp zuK!Ot_z8O(!5jIj0ZuG-I$uF=8PbT+LEogOcpkA~pjW?3jS(l~)D=blz(*bN9!F{u z5ttt@Rv77xF(ehs4U|VD;I+*yOsWy*$uk45_VA|q+rE(slqmtiOTyYLx>lp(BIEb_ z*c}WvBa{2Z<&qH4YNLvGc*`?WwGgyf5&miKQriqyLTqrY2aD=J&dkISnniId8yw4uWgFK<^Tv4o_L2GTMLDWw5(yTl!1p7bmrWm(m7nw!;*m*YNSxCjP4;H)W4Hdou~ z*VUR(UQ1O9b0c+7es39lRI_zA0!ei(fgTt{yrms`C9FR+AD@<&XL-0teF)cwTS78H zx_6R^ALa@bNeOU_6lOL)ip$MBRV`Nk#K?gdizd~T1DgN#elV&6-dj4oWd_F9P#!p0 z26+vyG$|94LVdQjLgx==n(}z$?{?_veZz$=BmNS5l`YWa06nBHNRaL0kKE34Q|_8r zd%Kadpzp}FSY1*LL_LFFVF1s7%lbW!u|A=HEEaHw{{HdpYJ~l-#}fgC#}G>Oi_2T~ z!XsgRx}Fe-E}9|bznJ6#g$Xk+YTJ+MXETGNkh3F|GTMZXXlAtHyf(^$+0y1FL6k{D%!f-u@Ul4m8ei2|9x!r)a#J;5UCn>bh7@MM zZ|-x!6~l?$%K4PhC+%ple02hPGB90OS7CpQ2uIz(q53qg5*Ej^`_uAm%A4E;B%o}j z1MWh&G79UlyJ}dMH7ssU#R2Y7@%O@&&(4&toS>7b;dJ{ms( z%zNDXI&>bNs1sIEWMc=(+#(b)va1GZ(|x@OBD%<9+5_^7uv)}k^C=((-_Pe%a9cYe zqIRJk(pK?E4FeQ4qr09UGL9P6a>bBVS~7mh^D;(N+i0TkTeictTn>eSSLE;+%IM&1 zAiBawUtgbv%!8(NG7S)AbKUalH?IA#zOBgm?eCk%sLmg?6nwy?bkKa1_hJvYalr{d zG`f0cDcG;MYNIXIIWiF}C{24r0rcepVDQWRlJs>Yn{F562xo%XO%Q^~h&5k^J%Y>dlTDq5V1X4S9OMwd5K?GYm`6{v zEdfcU3v&oUfl7J>}Es{1ng2!Rmt?OO1JOMCP7C=7`>raxc z@w4v~w88;)has9Q(ui&z9s`@9zRcLjWxPd*v^hZA^&u510>Ym4R=h7A|_<0S#5XOWVr3b{POEE_|3Zz1y5w-`NTs?Rl%kd~jaV znj}EE2@m5N+1)#e$$Q8^5&=NUq08o@a!jO6nw|;o7tIt+MbK9ZfNBV$$uvWTU*7!G z)Re#>Q0I^YXj+#~XN6zN!`BUDEz2_kte0T1(S8$&dqgDrfGs2kj6h(Vqo*;|wL*5Y zJvuw{>!%lxM+e5;g9EX8;KsRxb~?)$<<0pisZG@jw=gzqP%J_^899l)TV((zL>VYw zQQZxWvQ-AhL^d@7!qLi$uY>1}A9oXYD*BR|%mHl7k-7%=KDF3U%g7s#2C!TTep*pZ27@r&iEu!V?P!G_|#l>_2P~Y~qKW?$y`!%D4gLH4D0TCN0 z{|gS2RMVW!+kO16Mnm>mYz508k1jE5zr%1GwOGM9Y8Qq?C^J_7y;EB%g&D)4iE z@sMfnkjvEE>zT1-2Da=zD(e2qK^H_Or68B>qry32+JnOmkntH3U!f-xcwL?B^t zyyIqFq$ENE7$`KBkVeXk$-H$~tSKldBw*xxWfTf}RM>~$bF+zO1OQ+zW{WkLvf@h8 z982h^gn>=BOLt}a%vbb}wQE!Z0@cV+seKz?HS@ne%XYO7j^5>3KG(~|M~a?we9|_~ zR49IEQ2KXayA&C?yf71&yWpwCq4SlS7LfP;P{;fHC9UO(1&@yRgLO)vY%!?3_)Uw4 zW-PZPG%8?-#V$%(ZwbiCPIq4ee;F{RQZ(pmQ)^YAWi)!6nr~nyFBUJyZ0pm`lthyH@@^a}tGN0~pgwq5O)*}MpeD7R4f=abeH#Qh z2Mic?&@lMnVJE>n2*Ts`_iAvEF{8qz903TgxB2JY981#WQRSqSx>;{w(5M41##YTN z;&fXnoD(FqQuV1{kpm?iHKS73DPfMKM;fk5Dd+21szJl#YMq()oJ0nQvwoG+3e6ftsNv4guA<`)cbDrBjYrg8JIZ!c5-7QK(d}X2bnB~y>PHN zxM2U(m|sJ~Rtp)}QL&o(r$SJ0_ol_$MJ=u*m47mTL(J!Wi;GCW%axN(*n3L2oFell z|4Nk2w6g(bi7`gt75O3@8ZlzJN-mqus8lae`v6LiElZvx=gaBqCV)?zMxe%m-z$}L zr-}Gon$$J7ak>;6@Iwj(MO@NICWXgbZvhgi$*;@BrgnAt)|+@p@7Vr)ch_!4TM}!} z!DbDl2flRb5CUM2EER0DT z579@-(Io3T^xK}00;{cR=)})qOLA(|)VJ+N@->DRWg>cph|$>m^*Wuk&=9dRqb)|I z!g?tDeKqDQkA0v9%pulicYHVE>%h|B|MbVtaZ1k{*dhSa{#7L}OHZ4DzEKB71T&IV z+}E-wI&axoMi=~OK@T99eaX$P+`~p019tP1`|X-k7c6MHdvtg)Ra25(gYsJ@Bj%`* zQ8UD}T~{E-6&K)Zyd0hP11F9or{HR7vl}iDtr389G0Zji-52WVWQKAG6tn(Jy@fd) zp=XheM2;IaFekno?Vi3@=Oe{2P_CSlHwQz|gWz`xfY!*AI(EEDk3mxS=-W}8EFA^! z;xbWS!fiFIj$Ie8l}Vt{n!7u*f)^OV5>%{+mxp?$vR|WER&aD<@f|$;x43ikJwxrE zuPIdO#?MA{ehLCP{u6J)hu{ILIRMK^&jf_QjZhMM&~r%#M@D93V2 z3)YHJMQtTB4m7SPkn&2(ju31(m4I-8zzka+tcoZ2X^t+G> zubA_!bn&2O@rN`M9JU?<*`~VRKrS=1(@1OtXJOc=(EDBvCD5^=$y%#>NGH995)o5X zGu-_bL0p3 zMF<`Bw5iTf$pWFkEMj%AqbBUjgJ~pihokma(f5BBnnGVMH2wRfgJR4RhJRCLt@!nq zCd5sndNPjj_u9XE3Hd3XOApF*vwm8zQLRl{%#}}SeFe;U4@2zY)PFSvsG!4+QP}Q9 z@)c1Qg};&oo`)pBIg8GFhuO>U+A!uyaOpKj2&KgVc_m$Bz5Xu6YBJkI7_QX{w69HPq-lbzgu+hoP+$E&1S$yj5NzuaNUC0- z1enP%=<99>V(j298$V@8;vw+LQ9dbXlIT|+T?W=6j=?K57-3L4e)T4=PsS^ZQ+JqO zPv`MnC@8t>FPD0yimyaYjCLVj9Z6GxHn1H{#ZOsPN<&ZopLZ}~#gL(0r2rQ2-$vwW z6!X`cWmnGkguPh_7B_d`Ik8|a{tQsgLkyKDz}~McQ+72i1`_$OG;OCe*tmg^C9%yn zxAT&F078unlqSznnL(Ors%9c*Ssn&Ezq8~QpOcQkRYtOcF}BTJ%E%YF@-$E-S1H9m z0(>nw3=422sjnDEhqVFr-3~(S&!4)xnM$q@O&)I3p3uG9Y3aRxti_2D-TBxLF1sVW zx~Vcc+R~^N3Qm}CXBEkng7pa??^7dB?3hPtl}C8x7X$j^Yh`ZSyCP;DXnbr9ltpEN zg}4;Cmy^-(Q3nqDWAG*_Vk8(eUlmOWMt9GP$)lmG@J#^NMan?e_EwFcgTk4QyhKJH zVFJ$XQ@Z^);S$d6q`c+7%}7JpMaG5z)tkU(8%Eha=`<}>t&puQDrM`pGH@>b>{Ie( znc$zfY@4OC8scfp|D?RB0K-2P&FGLgtwqySkRir2gPD{0BH5-~LTwh5SYVXIun3^#6BtAL*D;@zNpQG+#y*RIM1P^!o zH_ej$b$WUlO8~vI{?3{v{BOO-X`dz)aq^O4I+N{vYKK3#pwf<0H)TTXs0 zt)L-e%Du(ZiAYeemdN`l%4;zh=2_gd1+;ckl-Ky_j1Yj8DAizL{%(?3T-y0@)9FaZ zW&;M1#Cck02at0UAJ&2sj{&AO;K@Cwu!f`pLm0M=-mJ) z50FqaL8hsAc6E~c&U$PS&wl~7pa@juGTV_t_E?dwjcg!bVhvnH5uC*dn zTOi=14#<}wV;mkIFNbK1>4Ltsh+#^<`U$Z)L=zkvWzBDdYs(f)vPC@E9Y(_}H?gHU zeJ2e}k5XMhAT~XJj@fA`ErfaHGKQaE`n1$(9I7sPGw-k_e2Sb`7>W!%`tEah>_<&% zS4(}YR~-pnS_y+NgN4#At@8hQmj;^rf-HTkYKRw5!a$i#SlXId5xmsw`5g4`p7HFb zE^Hx`4xJMTEu+2HHW0C*=t1A-!2QUF-+1IJ()s?N z1Lb^@>ZyJyz$}3G#iUKHw)00r;VYv*Qsm$t>B z4e0bb$jV&zYuma)>EEQ+x;2M?>+;(`e#*6GifmT$fA+!%lbPosQq-osSC)PJYS1KA z{-d`zd{oY≥$dv-+S)PMQ#Lb%~YGZ^k;`>6WJZpYqJipx2hwrGO*yOnpyzLI8-v z5cEOLWV%S6Q*D)H6n)9K*Gx+j=aokoO;Iab72FlvC@C)0z64RIB_s3aae4iR*8!;IB4b#RoU9|@E8D#h zdvGZQv};*lmaf2N8X;~)VYw8L6{L0jVPEO+xknqtViz4blLgi30b+m$)o6Ap3EiBs zWuTRcM==zB0yIT4rE22WIdB=Ydv^gd3N+}5^Un?FgX;a|8iwX6i~;)$Jl~^a7y9Bw zX;fvhL<2KU4x#Xtx>>H`JT~jo2ku(2OIVuFLdk}jH#+-d-vPps%2x+9OF?BIZze;h z4Qu?K%>cCj&#!xfqaV+=)lJ=W{vWNf4`Nfq0ol9Lny`Rtls7WUe6%!`0Q(vvIzYf^Cr~43;}x79%ZUk7x`$rO6*{LZMvx{gu%!!5RPYS5081=`Bc&vo;9&+1W`bsg4^j0r z+LE=@nO_eT&N3jq7I$2(L{vEguuqsY%C4x3rNG)sFDqjV!w2il&R-J6zqCchfvW_4 z1h<>~%$Fb5R83}Mfv$=61OaWxt0v2kCmpu2b5J;hCXC|)K}Lox`AK^WSx1~3$Y_gb zGM{uJe}7n;$<%G^3YZ@;19qfieK}Vz%P)&nen)vBex3ET^ znJjw4QP*mG{{mO>OCD1KD*=~?&k_Wr|2!~3D@3Y_1E*eG+1Y4BdOHD3dUZu>q}6`+ zuQ7kU)irECdn-N{j5^U=*jQAA$QRKSI&Us0xieimxhUB|k!}@)=q>&AEtM@*!!Ur1 z*8ufvZhI!K?g+OgHkMXb0};!UW1y^(e9n&D*QnYma>GBE&~ma95jc-Yq1x^ga7`P~ z)v%z&J~wQJY!U0wvsePUb}VD!e6anyfX3Y3;_v2qVlOh>cbS=43ofEj{})KMhk+sh z)$A(9g`>?qt(fcQinQ)Ev$$}AS}_svkJ8vteqd`*x)^X^B3i2SN~-)zT|{1^L^aid z-Ov^RA5{igd-Q>~oyDvBJ{sE zMz2=qH|+XQCLKb4aM!(B!zU2sR@c4xSs8@hQtJIol9;3i<$m;`D`qj2s|^5z(Z1oM zm%EF4%T(Rx8*;ohStLNXtndH!+t2f{PwThy_8F~~a;BKaiS7HqRzEJQ+H7#Y3h!(A z>bKICuW|~iEaz3SwyZF7Z!u7c$z+}6B70ei!-r~*Y#^!N&upf_-UfWCLtd5k7lgJ@ad0<|_3Xr4V(g4xceg}7oO+khg{ zs+O@Xx&s~*6vdm@{|lgM-UCP{{$k`7=$%(p$GFGFY2E0H^x;tF-cCxN^ZQ9)wQ*pV5}Zi>-9r3&Z1V}_;3(S;s^OG+TNxqti2CA zO)Hjsy&+`N@2U3<=sD4vZYcTD_KY+ztw`nj!T?375$8GA&pZyt%r74dx3qysx2o^T zGVe3?iQDfGrv<@ja`pJXl;MzSctQ7JPGJ%hx8a(J6-0GFH!58JSAP7I&CR<^9OF6* z`8I^H6nS9BmY60^h@>k1tsZB~hL%Ro%i>d5zu@4Nwr zg$Q{j6%POUR9>rlD!%H=k}DeEAG2VeA`DP2#$sD|@igq9 zDCWFY2auIfxpKx*uu;;J+V3m5w>|FC*nqq7l?!y0oD}rtBvm-SDfrDn0BF*owVDo$ zXUM+MYADOu>nk!k(wfbHzzh*@?(eU~cT04=et&DF%a=G&0m92#Mb`#$jSfgvVq@Hb zqvGaPDc}XyqT>Z|(-oc5 zF);yommIr*fDR|j>nk)tJ@XRhReA*sK;oLu0($5QYB4pKy(DsfCl)3EwAsP3S`k0e zy7}0x$BZkZDxRN7hP;h2Hem<#uc7qRq@s}n{R1f_v&)b zs{oBmM9tv9-}%l3zc=4SF{MR}yef+qoVT){>-O=pFP08f(&RT8w#$C0LRI7QTmAkK zQh@kp=S=iF{BVoTZ%E~xrN5G6fy>;OG}PtnK__bK`B4?HF{E&@mSdcc!7_B0qTFQ- z1Kfq%Qu#0-pk?pLTz9lqW7<1_Y*E4_Bmp=sS%zsvt4iY^{Iu zg4^I0Bm@Q$G{GgfTX1)m5F~?z1PQ?j?j8sZ!CfcVpuxF^=Y5|a-}nFixK*c4)zqo# zBi+4v_u6aiNta$hW_vtu2P`oRU#J<{y~U{ThWkj6j*nm8UiZSMh1sF(X69F*-iS|T zm}7!2$`7nLW-+H0Fhp>Oq`9}ezf)J?c9qvrDt)%fAps=9Y8*=l3&#gnP+t;n_Kj0O z!Tl9-3_+Ad6mI;?wi^cfIx%~iBeqC@UmQ&@P1D6*Q=jGf#;hC$SN!-eyPA!o)4)A8VL|xH2tFu~BJ^Y&QncCU>CqfFrXK zOs1zXa;B27cJkj};X~olJa9iws1RMH7(`R#ffd_${v_+geOQ-jqBqlu9%xICk2cv$ z8W!mofFVaymX`q`W|J|%Mn~hIQIr3xm`x-mA|Xn|XJJ;ngYbfJC|5Hqwv z-4uHxx{-tV5v?lFYxyO8%L9*MPXTJo{0miB|IyD%1*j{{R&saD*9nn-A_^X(DT5Fa zN)sguqkU!V7Y;!746vbWp=D7%GeD+FPeO$Ng3QJqrER5mg4CpW%O_wgR1_c%)JZ{^ zl)yrr&Hg+-vhdr4Snu=Bpu@g7jn&uBY@l2cTtuo2F=Z=WU%%+k05|$WJldT2r}E5Z z7+|g6R&&+b9#!=i{j!K7WiIXUk=uJBl$B~TKr`!eDPWN>-3mwiFAH3tnAEror;8vJs{MTg{_I2$;VS%bTdi0O}W@D3fOwb>B zrKd|oFGmWhNfd!iF3gva+oH#s19%C)hnMi{HRO_ckaA6j)qd?nFeuC?%lyNFb*=-7 znM{6$$0SiDgDu(!V1Wu(1M~#pBQPDB@ZO0t4ktWR{~fMsDS;Npfc_ny!X3S)9yduk z-1hfOLBA>36kcpUETKHy<|3`Q?rc?Jbq~;)^8lUIIPX6<=%Sfpd<<-*Zya*4vVwX5 zcz!)NMrw4XA9c$AV6QR+9D|o;qMKv|H2|tcV6Oxmy7j5z7x_4J%pC$A2(FOy#nE~c@ zdGw?E>t|)6m@yyspB>tKr3g~PwuZ{>aY^GLvq@Rg;&H!-@25sR;<|A zK#_7={=4@KMs*I0q|Ae=Ix+2q=mKxfG$`guP*KZT(|?*T1y&>N7k)K<(_-1q%cX^V z`BZ+eZu3gJ)gRI>54yg-7VcWIzX)jT;Px0Co+DbhXb#4j3`J>jUkz-=^8&PVW>e4H zuFv;TG^oBDj7cT-wUgu80{(X?FwspjwxWnc_DMgC(_G;xGqT4Yv4O7~z} zSrx1)rJ8V|PGGOX6C2%jGdZLTL%f)cdPE;fG+ZHob#Fxpw1>UAi*(v5bR~hZhA?C( z^D^bA?Z3#bz(2Zcld<(#+rT|&jJ z&oAmrtroXe4NNQMpuLWbE;P4|`~Bv&+xfPA*Q(a>p<8qT#qL}89bH`H^E0w=Ia69a zkIt&yog^;|HP0-RTI?=CS2|o+bd+=!mJ=7bWDAl#dU@-^{k&+NdZi#sE`1VB=efo! zU5?O~rD`t~rA+FPf%bkm_)c?g9 zcL6J*J3gxJWrS_y+=|_ff`;#WF2ix6bwRA5)qD3P13ma+K4ly(TN#!Pp0JHAw%8~?F-;jLOP3!u)~vycgj)N(I8Pe^RcQn8?Xrr@ z%3!pleV7;7@;9|hOG{ulPugQ0MY z7!x5bqCi#EE~R*rvpEQ+c@n&Y*U-%g!zzhv>~wBK@m5>kjrN`Bv(#dBHGnaSin_gL zG=_hOr@QeNDM%+IAY9SF+8)4Wi-3=7X99dWyXs`br3uu6J{(FPjG$Zzubqg)vMZ-a zh`76vjQbe+-c0^VWjkmou@~7EcjxFA*LELZ_S5)@+uYm_+xO&M?aQJR>`mmue`QZG z4EZ##OuZ2BhX4m?S!S%y%gL7-*3T?m(O(KYVS*f=uK{sWx!h9s#2Q{N;Kfb>;@d9( z$tYJOV%9yElBB|NX2t7=sL$O#d<2D5SE$c=xyVM3jW@g{2B`eEN$18QTP$^|JVT^L z^_#oV>D^a+Ps;Cb?XGxhv0GlPcQBMGs(jWU#tWmj%}$EMaAGA4cw&3nLYWQSpPXpA zw9qZfa4Or*#oTZ_T0zoo$r;2=|B)I{%{L=1!bHU%nha^g90&#M)D})>J-cJ1$FZ0y zH0B_OIeSl=FDX9?^YT^)Tv$;F22?Pa)E|N9jEw3ZKU5$Bg zlmg8@Ted6@{ z7r(Y(&DY_0+E0?7b-=?)a`)w30X%1u#rMH^>Dz*28rD|><>Tn_7sNV^D4W}KAU9bg zT{M7N^lt!A4OyzRvwCSKCn(`r)l5=;`zK{b&P*ekooMCW1l@5Uwq?cGj4^pguk$c~ z`WkxHWK09=XRvwtgqulTG_rlQQXUh*qD8BOBHU&Hg?w5Gy_s|bh6bAO34ODcE5Y7! zMBy*tp~sbUx*D&4?xKGeeupChsR@|OhtJ>Cd&c}q*n@0y%EzuZ%j1Pd5czA`*8 z%O}F9qUE!i#xi>wnlJv^6D$hAlP^Y$d5X#rSJ#E>p1UuvWW<(m*mTK3x)_oNZw+?PDRUDD_iytI2bw4K4>2?NrN`e!*wS?^2I1;uvK6oNOai zdC2PhC#j3qGwkJj#`}G#plv6aVfK$H^35e#c;HJ!xTNsK30|)=GhZ`{4D!NKENE09 z;EXC3qRqtxtBFe(bP<|%U?bcEmCH0c^!+@@KQKgOC-08hb|xUGH^8Fo?h`w2Zb_eY z5gf!){cP!&{w;Zx?^+546u?Y8NIjf)_p?gX0cOuvIZqd|1V7WdRbpud0X!tTw7c!c z^yLZ^wG+a2Vv&eli~1=7Rs?=uKAQL;>U55&j}If&jL20Row%rgkgM)cP#svPZ@kjU z^ABM_{drawYt6bXXiwrL3rXGDdslLQslPW{!fltW;V=AdQ;^?}n9nLCaC;rYSBrOD z7WP^(n??YaO2b^AlPYDnElk*Tft)tWp+XE2yFHIzsgay_r|x2~sjLlic-+1!ej~-T ze|Q)GhAD}8vyQAW1-r;Dyv-l?c?w_>b#x-(@OptTu7xC_;)eJFa_Kner#r9ntG><_ zKr&JHyfyP6fta~+LB~0JQHXx}E{OYCR$(!#KUjc#x^v|`5ix0N_2ai?$652)x5@64x` zg~S!KM5loZc#DR8@Xwc5Ms9C8U*!B(4@xishV_#b^=BuB)gQHNIk_*E8uIu!u1`+Z zn<_kitARnA^kBUjM*NvWx>VMP0`TA@-Fu|v0Adu;3~ z-TH{xKUxhPu5-W&(zk6Uqx^4`g0-`!T=D2&;dSgi0w`p}b8*Rgo-!8Wz!nat(uk%X zdtse_n6-BlBSs;Wqr%dbr&IJYZ!2B^ZJDxj{B8#o(S5%(;UqnrgtivnnDTMzaqiD= zQ=7Z>3qzk)DHgC8f=G{Z*U3)PWz)fY5=28croVIugi;%Eo*zre_V&$xW?28R#PY`) z?FqMJ(2JOP@$dq3R}BAZ{MBu6C;oQXo9cq{4P3h9#5UB)RldnhlA;<6$BqRK%n3*; zvmmVuKu^aPh0i?=0(KATh+}1@uCnP8QjBG}N=mNmOr%=o&cDhZcNTKmOThhvPdgfcTXFD?2iX{WXqt z>;drLo?3vQnI72B3)1!(qMYm(eC=OB ztX~w|v*W)UBxEJB&tvC#-!K|p6uFtLy^nHSbNV`)-V)w>AlhurOX!@HZi^(M{ZRO- z+`!PlktykR&$t#$z!B44rK(3u-Z-@_vJ} zW!}b6Md2#dC~*3E+UC{jzd8ADbDnSY;v9g2B#^XzY_`v3mU78-d1fE!wf65Xz7 z=WsAgGyOSDxkOL<>CM>9*Skb5W`PBGk$sOML@oTgnTT<~m1T)cK>e|PgYPR-$_N%{5jLx_*a zjwjqvnc4A+_lX@eck*`J*8_taISa-!E2r7!B&GQ+YI7_Yl`X8jaT!_|YHNgCda zEepP@%4-+d&DQ;UkN@=p!bwHGkjF($&f#+M5nPgFMfMZ!F%N+J3oCl)ytn7V^X#%d(Fr1*dBj3x@NtP-%@<4ATwF;r;7`f6Z> zut}~Zs$NOb_D2a;iOAheL8UrOIWU{RIh;{qG$q*8ry{4f;T9^)bCV)=Dx+vMzA<7* z-jCtWvvvQ6!{2cLJ@z($a**a{`|b`TBd}SIaWVup&Lv9cCIraLVTE|YGTaX}EydBJ z@2j^Pq+boQ1ct9USe^4jA>|?(a@L#jpjyKZq$_r#)W%jl{!}nUemFtv53_bQtDYx# zt4>&9dG;%Zg#oWSw!V*DG?+tyeZs^H$OrSiLTvZa=x}>q(#z&54`oyha!yb-AeHT& znBrJ}-Y$Xu@ea``QheLH#k+1)v=df=@Gm0!BYq4d6hz!bFpy|+l@KrsRlJbH%&p**V=sDadgUn zA3#86)b{R9BqJm!g=Q|CKDmA z33g=-0PW#F*ZTgcs1-e-Mag8h?? z9!3{czw=NcIp#8dY%cg^GS&3b`uICrQ>D9&Rfy>%zmB5tjv8z9*|FpXBtkNn0U(Yg4K zjOz_u8{MeT$zoDdzt354qd;+#yhRD3lI;HEGdVG9N_UQ4shqoiE9e2aJvC#F=owt= zI`STcgg7IrPK-FUH8RnscqNPt&aum&Rb;&T@SMv`;Um+qlFAhBMGKCghIfQzUaIMs+dlV-lpq_Ia!~v7&qVlVE}%+I>ETm?f1M@1 zOQhV2bQFsLHrtj0x`&B8Ubiq2Ygp-Qfrs@R4J?N#6ve6Aqx0 zU^3OZNQ(L(OkMcTPicQ=to4lTLr6^&!7{sR%ur%bEzUS;+|HVfLD77! z1Q{yD-*|(J;I~hU&6L&t1Y6v^w%a!mCa}WHDqck&#BOeE@A>s99t&-7Sau89 zj;F1^AifmLIkE9aZ7J3Wx`s`=UWm~alQKRH8zWDQot`cL7@Qzx~K6-J~PH|ATzkT2VVG{Op$jpmB zLEL}zdIhe9Uw2d>N4Fx|>=nCdtIRC3FJ6D{j}q#EB*^aN(P5S*;-4$+6}ranN5~;Y zZXVK(z@e|qxFg%b5Ip>T_Vl3ZZg;D4%IZzwe*AAY4nE7l@Yvxna!-7){nnsDY770I z#LArxgjH5SE;~dYt#V-Fk~)!%$j7P|)knU*R3Z1yW8UR#IsAmJzaddNUZwpgD^WQd zt>2C_6c0i5Hi$MSSUt@JF8|w}^GwnC-!iG{=_pO>Y=^Sia)yCUp5bz2`$Qbgadf@B z_z|G*yHtH5G<$PWR=uuG*qt@a@zy%hR6ylf3+fTM^sP5o zAkC5lc_~$NrajJ0^JM3zx~ndG&A6)#gpAEg%8GhBlXhVn6FHasw>S@l$v1(Dm^bOe z>}fF2dV2h&OHS1z)7i+yWKb4=4D;E$rH@0}UNvLUW+K&$6O87WVaO|OCGZ!r+a#6$ zmcq10AFwxu8!u0&o3d`m_LE@wa2WRb#x)m5iX9lg(K!W85rP>~3bprD{or6fg_prD|4fd4TF@W35zun`{c1KLSR z;v-b~DA7LfS<_NQ!&&2#ET6I6XJ$haJ0nwOx6k&#C@3g?H$LFwXH#cGu-j)FTPHp@ z0rG!F@ByD+Z?lkt{~6+JEkLgENdYWk=V%J%V&-IKB^N{jgTefcCT4s}qT>H~9QaLu z+``$}o{xp)%aIh>-3*(0UKm_{f32&nU&>#<^~?+ zf4$2m?`UZXto%B^ARGTbBmb{^|5=Bh<#qA@gP4DJ`p;cptAa@UEdRqcK_uBu8cirD z2$Ynlu!aXNAu_DuS2=}D=bkXtA%YR$Z~1O&pG5`%sB{JPO=qK=jaua zFrQJy)g37+E$2(J*a&bv+QD$<%8g{(n=I!^3R;MUyrXzaEVfTcndASbb?oVBiN4vZ z!Rw**U`1KA?egxisYEWl@5NJo^?uHMnm0vU0z?V<|Mkjx3thoiqJV{IPw7V;Hwbgr z`6J&rThbR+S`NbE{!MmV4%T0x4x4>S;a-)fEBZZ|KrkuV9hQbKt-0{WXz_l_Xl4Xr z#AvA33q=wVaX}bu0^eZRuF=ef=eLf49)vK5_T3mkf)5i6joBaO>s&d}5DP*6CP3(j z+>KTo{09+(M*PF_r$7mWNd)}rxX0G5_#-7uvG(q-9ECc>mW`m)>U?61I+d;bP!=N!Ra;cVEH@ert zR#BrBz9l`YcgX+A1Z$HOm&r(pL|n*)?kB8^PFiyc*xLl<-6!8O`ufm{$-*!!4{v-# z52I;JW)8$UkLBzc2oKf~BFCjG$JkXe<4D`PBQs&9j3Yl?dxR40{f_Jq)4D}UPszRs zh(moWeaL?GMC`Z@0ulYcN^-LNn`|+W3hs)K6p%RU4DkMU2p= zuGdTm^E$5j&uQ1UUz8??wtBmqLd!s3d~J0|;eOPAk6sUM{$@=59wWof zv{6{-b+@!!&;&b5SheyVEFv>r(Hs%EcprXTl6^;aSRyMFISD-S&1OPPvXu>8)LuJ?{0bT9E_OSxQUSwg&hpQkkjN9w9PB-A@*Os}6QTDJT{ zEX~uagta4q^cXNrQ|iEA@x8-n{AhxHMD+6qoUPsHMDS$GSGp~pemihmK!mU4d^mWC zY%+-Hc{NA}kr(;mck~O=Ii!0{02X5o_Aa%xnnimtX4S_}l7}s>LPRj&QKJ6``aty( z53T^!*fqMt+{f;A>qlwX_4n0G!kWfO1kTHle0wq17Ai*D>2{r;G zGkp9a`~ty+BD0Xc!j3NNuFOOqqr=c0a!Q*KLd%{1V}I-spz*7{-Z(;$Z)O$z>}kMb zn@SIhT}Hobb^n7X)L`C7$Ky_LK_VJ)-4d-YOqsS%x%>&81C>|WjrSV~1=&WqZ-2d= z6%TLX6=kN=tR@z49v@E*pI$3(~^%f5bQtgKI!Vk_yL zzLYg}9?pfeBpPKb$2CE4YUam0msT3(cA}?ld(zK=PcV^7V7ARg{{C=3JnxC)F?7Ug zwEt1YI%=@E3%96N{UkeAlaqt=f0dL{!4@<|CbHl73EMJk^2fh+>!YvkLIBjpu|Gm6 zh9DhPf$Wh7PRxo+8fIYO|I2c-qC>lyqKO7SzYdB{_w_U!H-OzAC5ihY^{+Ydh5bos z&w=?uWF%+riMD$|^-=KTBkYQ0|D-25JL7+>W^}tc>@b||ifuK6xh6&Vdvw!Exf&v+NYLiXIf9BKll0&iS2h?Of)V?@avs`(qXPZo}aqQ9^nupR4 zg(Rks5kGYRZhZmahjFFcg=HX0L0g<6&LjS?RFbFK4k4q!3eBs1z_f>gW_!y5>2BxV z*31y6$Y6|m2yT=jgO>w3e73GVuxF@^YcHS9BdoCx3f*nR`Co=gp(A5Fe{H?UNZTrl*@ zg^o2Flr!>=yTY@=lmg1K0St3hq7^)BJ4&tC(WdB9 z`sH6fUbx7huM_uAp-ahJ$rDHr-KXtQdB;5KMsK4}=spUvl{<`6GI9KA;qig0A|!82 zhsmb+mo;?IY{5~GS>bMl(Di}e+O-p*4C3R?lSm0y3HT(+5!P%GlaAc!czSH&Mn|&0 zv1krQkT;VCQ|gyQePSa7m(Pr|MDD%R})GU!~Ekffg|FX|As}85M?cDz{WqMM4tap zOJKND5XdI~eq&4E64aL-eN8v#m@**$7#ppU%j{ijOU+FA8PWV3o*-MoU1NSIm<8I5 zZ^yw)`r`%WFZ%68{!d(M9BXtA73_8eM5kOQ1IV#u>3y?{%I>ccp%Bon989p2WJg!K zXihnslu%TImyg}_encp9NF$+r%Kj9kU#0&p`jBkW$Ok@Vv8vZpji&cXxF4rqGujwj z4Hk%2`HhS_`5}6d0lY24Rbxhkn0V67g2LvWzq-rykI4IqApVJPlsc5viu;6aIZ*B^ zDN_y?Q;JNURYOU&;sl&jgi>Q>-~;+agM)l$5O`86pwOJ3h4gZ8(M`E*q_phLZ~!tn4RaULZ&) zxEJuZb};f?3CLRs-?)D(S<-}ryKhLDc3Z{;B9TNk4t^Q0XL)0Zk$OJ-u%n1#BPgfG zxZ<01K9ZZQeD{^#$M{0S%`Yg=kq$fsLH= z9zFy9jcz(K=k~M0Bgy1;sjofo+F!V#LoQ9wT{aWKm48Kua#VnHE^O z#ni77V?qh(O2a@F_$&=EGabak{wLm?Pk=(L{#+XQtq@C+I<_q+)ALm7A0uB!1?jR` zD#_j0hxPBs@;u~Wdxzw3BFaak7Yl_rAN=}M$+vHB#j{#kbaBl0NVu%t^(DUv=y?FF z_#kGMp<10)k%k{^-Fj>>B3!R2=c`&2@c_}qH+5)iS}N!W4w)~1Ba{jTYF;gKRvN5D z(vVcDyQUmenQ3zJNV(0B6bB6OP$|MVtF~CezFHuZ*6=C0u{XOa#{F>If9yFsdi!Kr z=2)iCD#lWqDJ@A0Fd31YFOyk_`Y)OA-va0eXJ5?{)da&ofm~^T4c!prJtrgBNKs-| znQ{NifJk5~$0ZR+XyQ*PmUC0tx z?Wd;HTTBwTlhi?*f8Dr?X-ksdLu;m7UNsx@fx8rk_5IJk2|lDr{=Kc^A~{ z0wt!7V@Tat#o}r@fX%?EJ^W1-VTKMBn%}@XJWhv~V`=x2_+dSSRPUw-SKqVQpu>Ng z$eLy3B`#LE<7MF8quF1)OO7-h745k1!3#JuqEweu=qT`}iN#^r3jOgd9|_;v{(Ro` zdEPy^S@n6rtjTyGMqBaGYIbu_CLxj_`zf1~>jVuw+;@L15-`;GYr~4b^Dh zC0YJ;x=68{*t~bx>htB!NbQ)=i|6%+(`F}Vvt?U9zh`q#crGv2X*bC?6aI_uHF#FO zp!nez*+sH2D5Y}_e^Y^-`4!mCZ3DyG5fk@0juxfrZ|dq}!O4PoW?7Ge93#$Gul3HL z3gM63^zf>+(NGap!~M_sL7J@l{D9Q$LWc_; ze@f*}hYe`K>AhPW_s>Wqn}e@O`O1DfHDm}Ikv>oT_v)%YP>K%@q2BTF6ZDBnHVW*x z)(U9zk6Q3gn|Qb+3%su+@Dsh?33mnd^J2Y2%h*qa$xaqqzJ4)RS64?d+*$y4K|8;t z0dEe`ZkQE+3BjG=?#VT*`=qKk;fdH@-yF^p5LvjBS{~c!bJp=DBy4(a?qxepS*`)D zk2DL;iv=e&+8l;-TCU^>deA=u!@G?iDplJ3Fn%!UE(XGjkVSRN+p8jx`!LIem8`Hs zDfpER;dH*z7^~WRthZWA9wI1A-dVOJX~UxDI&7Ev@Ho!kFphTP&9Z*hFk2oO8A&Xj z_KkGQiA}%#rkdVoCRxmLPWRkVBOfZ5Lk}V#*y^}Fn5h~6W@;n~N_+V3+{oeqW1a*7 zcI&H|#3)a^eAd|p8#LN1b&X@Hfh=-LP~?ehS#hycD!ZvP(W=mk_2fnahI{b3( zNT^%qZ;F?E^u~9fqRiw8i>r%wEE#G^fYt^v_%3M686cy~a&8%EcQH0SuXXTZY)9ko z7`demQ$!e`k^Ee8s2ICle-y)OpK=hQ2|3d;HqPbty1&xvUJ6W18e;qU^#jV`l%&Df z!AxnB!@GeY;oUVqm^Uc~!(zMCc}%6!y1|br(RKV^?yq_jQU1OS{EBP(NAVy7)o_k* zzopsMY`*$5v191R4e_=&TR*Z+GR#mo@pH&uzpwJi)n=}mfb5z3b!q>lffD(JDn6vf z;G1n-iQzHk*MYQ-cW`_hk5g>v_PfJ}>v1&%W7*cTV!0y6!1G=g&3R&cJj-xn4!VYR z_4?r{u?xXY4sprX(>v2KeXck=arhhV{_Zd?YMu}YJDh0)LLc?sL}1mB-l8r!>$Dw? zbFXs`MbYv;KhR zN2PpoF+O(qZ{=IO7F@wur~Bi--`-u}(-55sBg;1i%Irk)Ur3^2wfr@7sUg7=Ve8kW zlHm^tjBwD~@7S9QCo8kTQ?k??>u4d&UP5U$=^wsVcX;Bv@?w|@?}ungd;G9CU7eAQ zM2%-v)eraRY`fw9T~qUCabcmsI?K)WqVHu%_Hp&Iy>jxgf*OI@hiOa~#!%mhPd7;9 zU(bEzJJ-C<&-+NxO0Ud;V5|Fmw-XA5C*M_Rh;vvw!V`M%MYJ0A3HGsg1YUN!=f%L+ zkhjE*q~dmTD6J8s=Fa032zr$f@0l(*BW!VFa2u96r&)=dIu4kxh-8HYY=dnbeH27c zd)03vLumwR5n=oA+*ym^QjJ4`S@JnPrL<2^zO=F946tbefkXklX0$Bq zxR@s!_3uahKj9tol7|g8a;`^yWH!RV;biPmhxq+Bg7 zUFmqS&klmR9i}kx#^Swh)zgOh0x7`+W&2JYE?9baXu6A}k zxRh4*yRm73kGcDN37wSD| zKjr+HpL*|9Cq|Av##x)5gxcY@C58+TTd$vLYgEc;ZC5FCCAU7}8Cc?vi6Wo46IC$X z*KWnp@c?w3_ zWy4MSoopb}FLQbCS=>fo0Wi^9Sc!<0bs z4=!M&0d7~?;Yr=RVr zln>eaE`*YddGh_kvaeMvbDyp&e|i{Fv6gvQne_YVs|!TZqxk{m-Vd1g{CsueLUaW z-tGz}my~VEG3_!YG;gNMc}I_DRa6IoM^{J8-jvLb@_mz_-u0>2nyFew#?Et7G3$s2 zHU=#=l~PXFtyIHLa|tMSrF!}s%BSw1eLl1aX2$h^b6qMySlQg2;M1P^V@ftHIqt!` zGex?evOE5QeRqUazY~;Q%9C*>I0f9HgDFol%AxjVvy%06Xzq9$rmX!JP+x0GN)D`ed!b65a;~w~ ziD!9qQ27gHzj&5iopa5XoADQP66R7Ns?Ns03)UT?S!#r<&lamYC?TQ}rBWmsbHXN3 zi%rMR$ERd)&P5?tHk4lHzeBtpeD%A29(w8=jb+^Kz7QOB)J1=KxH%<_99uSd+Yuj) z%f@>^R_LWf)rNe9?&+U6(nT+z$d>8hsH{G(-`R_W}PvFhD)xiNLTbPc&_##YWhum!G%mg~!^#WDLZGlCFG`0A2W%kU3W-$84} zI1JCT35c}1tes8IWX34!J~->Ll3d+U3*DLYHL>>Zf^)SEB6{3b(}x`6#prtb6VNrL zoA;2JidAuwZ*lNev6QD+wGzu%$LBVEOmMZv-y2d_m(hGOOfsXRX5n7vVDqMr*IGbI z2Q07(zDm9`?D(MNHLCod4<>TAp;szhA3U|}e%omqwVA&;`P2<_BkrvJ%qT!qA=OR^ zX7R|Ch)UR2Jq>&$a>R;oQ_NgRwoH5oUG3F+921Iqakte|w7uosN$lecBT_bD#Z>U~ zrE!6xy{2j5S``^zA z*>wlWHL*cr-pTv$Pt)LboN9t=2$QrV6xy(R#-iK9c*A;Y<+iMpgy5Owo4SYJUj!Sp zNNSuZqj8~Y_-M#G9Ya0!t_7MRgp3;|;hl|NknYVqEk}?+FxuikLNqkazWxp4jPkTN zgNSgPHe9dwcloBa7S3o{QpWIn9jw88oychWe6{oE3jb{H#kZ@FF500^^2nravn8Jl zqIZRcqtKWE^C6Jcqk|F?TN!q_$Xl*9>QcXn9}r(1=mI{2r=qj5u~mrzKlrstl@<#?u*LMov}UIdc%g4v(Y9MKb zqsMcTW&M=_rN$;wQHh47PSRc+jK20wpNh&_*C_4Llvuk!hZ5Rda^3G5v~oo%!2pC0 zvBu7r3Mo)K@{G}j1hnqZT2oYnXbFhsx5NV9ISc7geFupt+i@<1-|;JkL6InNWR3CR zZ8urQsg~d$7-2)-1tP&alTGSd4;Ru(sU^WZf1>QRJ&GA;lqcsL-d-Qpxd21u5`7$6 zhZAjp7rzVR&ta!UoKJy@xU^;+-4!MYcFzeu0nKtN!G zMSKT=YD*+-Ut0+&I)vvKwO|{<%xtSDFJDoFs6{9VgE)t=15bN@n#$szSjM$1SwPog zzzoTP;IN|Y9t_j$0@l6Aw13y>l?mkSFWyR0l0QJE7D54_>^uDsV#(~>oKWzaZ)nSN zi9Ff-j4+5Q{=RtTnk6_a4fr&HQr3&H<88X{tH1b=C*MAR%$(J8G=`_Zkgj2$pel^^ zt_|kq{r86CNKF+(;_3Lj@xGNR$EECyadnf41#fCd;b=qyo+&mdJh>L3Wvgr)=-<#J zgawrj3nWS2)hy5_xa?FqPX85xu$oev!^jLiu$t`TZq1bRib2U_ODM*>KXflA1ik zC(px?vdBHgODs{XnkKraYv`Q6u-abw&LSyYDfU|ejNPeIc8ZPZ7lJlrwVIfsgfhf_ zbyR!Ywd$h%+maXJcjTbpAguTTkq zRO3tDn$sG4YST4s{PBM>)pryx)3$C} z+^hJHP_k+Dn-gB#!6RL<0t9oevT+Uh7s)~dLoVJn04^xL!k;IB4UwD)+wQxZd0MFmdTcZ7JSrq)E=Izw4QL> z_oz4tHw+TK!3~Z_#EBf#35@Oa7yC(M9u1pFeV`w{Ie-d&S_R;ouNpg*+ul~2SS2OK zIT;${z4(hy?L{trr{nvQS#F>U_(;ucLbir2l%JB6OCu9NMN&f+s%(4xhlNRBG_O9< z3Gw~*m!SaDoZ1wLHc_oV$tP14RltET46XicaMD@$tJ)8mkArIUuI~_{ zuA5W-a3*`?_bvD|olD)A*a@XMO;ajtgqf|Dn5*{X@5E(M-5p>&7}9BQ`wpUEo+F5P!{9y2uWm z8&v5=S#m_6&XLCW5BS3qsk_QzSA)5MYz;GHyOi6bp5UCkB`lWnl1hE$>}Zgg#cpYP zcCpdIv1;x^>CV=Qu7=*F2V1`q0;7dJ%i9lU_G|T3$5!!3P4Q43b&ZOD8B7~=A{6Hu z;sXx03k(5ALH|>qG5VH9_T4%B!R9Bt#6<=M6#S*W^r-UB-W59Q_lAEdqgkMvWrYsU z-Sm3!u8r?{O>ZxNB$XM>Tzkjyya%~0TTEa|iCYIP2mCu{`;o^)us8K)vC2Rt1y)B6 zjE&*!bI&Y{u1rr=iXDAg=fzWZx`zwF85uVz%Su7nnSheo)Zh1jxbnIDm1hYO3@(TVk z?Ub^aZ`qcE3Py!sP^qtK(ba!tZs2nD60Ms_sNaQ&}mZeLY}V z0Rw4K8_lw0HI0f2`}98_8YNgq23<~3~V1@WkWGWS>XXnE2Jt5!JV zD0}iGkSb8wfKV0XfD-;ZX84Ha(9faH!SoqO%xOdzkCdo@ufcxg2nIuH$p{76GtGIS zCU~{5fbh#eO(05LeR@2jOjMPRoZivOIM_6rca?jIv=K_QCJF{>V+tw0van6DSRM4* z5}d>F+ZD1WNw*GVb}F)CU%RezjasUn(%9j$$$=^p;H-iI43aX1PZxAw*(-(x$PGdE z6IL`1OK*7MEDDT_fOBMAc>wBIJaHDoY4{k%GQSPOTAR`}`4YYXeOT(CTTd(G zj5n#KXn%D!S#>M{XgD!AmqO{sY}Gqhq5e$X90=g;qal^BCm{~4L1jBeY&*D_!Ms9x zk#E0~u5r<>rAIPTA_En@Xbxyk!I>cJ4%eBvIsGIjN_fT;Aa(5n4#Yw66b7;g#B?Hh@l%}wynweZEaym=wqP3T_Wmg}iszg0+7|)^VxNXv zCpj{`wF6EW#w5NPVtvQ=IDgX1?r9i*oo5RSil+jL=&}pPSzBo9qX9OLLybG+)yD`o zROd`stP1`dq?hgiXrh;c1ULS=9q)4cu}gA)oKHU)r=Xe$DWz5ZYne(yp9Rd2JF?2|=T7D>b^$#?JI zgvz;_@9K83LJVKKIG~mwC(uF6N1Ls_oIl9`yM(HquiT?%OJNviZaj&bxT?}vP(*}S zBn8nsICpH*_;h^qD#TBRtf!c35>!j|E&WpZ?zzUWfE=Q?Q?hCnI7!RM$iVo;WdH}h zpG=>b)4cz2F>!Zh2Aftca-%jkSzFWyf7#+UBB?jFbXQeYCF>q_LV8aX3%WpW-!9T_g*$=ix{5XbrMU{Cbqd z;z9|OF^~;7Y34!-6`1DPD(5B0%R>=uqNzzV#_!W4hiWZ zzN!RN&ZiiJ?540)=B2!Fetr8)5;&=M|stQjgzdUIv=PeM>JF@n?0@%PB5tk zOaQ@?TmwT7>$;=jW(Sb)(|rR@L-E!I%l++&S(8mJ6E+n<3pTXzi8LB6A>F51fQiAm z2gQNoGt>EvF|SJ3-U=IGK1}qG03jZ+OqwZ1X5ivwzzH9)W&Nk|zyQ%tlzbv6qam6q ztcg*P04RdjPt;C#jpY&PO)!!msw6=r+{X{(XCsxahaOob5i5SzL}m+Fp4YrP13Cl% z@B)wJrRkT%CG~uk$$la1rj(p@a~MqT%bx zax28s19CK>4MY-4& zR^_`h$%n)ecB(8xAmto#BIdH17B!h%URi0KyBJP_2884g@#^niJ;I`>KF<)Ens!CD zsNc+YSaTwD=)e*At*J!qoV!s&`iM&orCz%ikK}>KGV5ffyo^_nTSZ!V=H1N3o0_%` z*lGBO#PUaAA!eb=s;$tYEI@?Ru|h@3S1*Z4ZrrS}yomUPQ8?fCG~<2t z9-c8kV&wmKo!~HiH}=CC&~65*!!Jox0s=lsHoWj94a2c0;=5mKY&G(q3~ly z1M&*@_9|rsyd7$1^EPXm^kN2JD^Ca!uP*=eSN-aTqoLUW~MCzjjQdDt}_9t*9kH76R2Xp6G*X zG@Qc$9971lVS9sS)(eqb66sRy;;nNd-2)5Qcw$<#My4pNf0FE+*y|NZWRvg%mL38=Z#X?%AB)kt)%D44F#gcrO%?j6kd4SAK!RG`WrbH=DbdN}WQY+m z!w@b4DnZw1a5Qb>s;u4*>WQ4)ba#GuEo@f>ZMHna&c`ojzNvV%U9Vj_Ij+p8^DOzKfpD~bjINTNsoezk44fYZ139ZlE7s^1$bJ9eKqiBXF{$uruf)4`t}S>z!F{2fUy*}lc!e` z)dMiH3`ruQpH7AX1#Pzf?P>+01T00eG?O%QE}+K)aTvbi@GPNm4&Fma3?OM2Yd!?W zDg82Ba}jfrH8G(OsrY-h2{M5hwv|6$wC@mNR_&uXI2y6o`MM!n1m#rqF8(LroH5o}Hu>KSJUjPeF0 z@cMu*$}`Ag>OMJ<>2v9CD*2$JSs*?z-=0;k<&#mjg51C zR#V+fY!RDLCfC!I*HcB*O``c{^N@33^Wo+i6Yuz;pMD$+{K(q5bDUMr5-hh?U5wt@ z*>T5KjYtL>$Lcen)X0e75p>RI$6jCwcKH2@-w~B52Fc9Ha(VWGMPS5|Vks5TVRA^YInO;|03&-^@`Y~QM?cdj3Vwknd zsMlOSVAhk@2s|(fRtTxBy5Y^G-DZG8b-L3?q%4%AbPTc+SMV4~Yg?yeL z?}6lj=#@cnp%57E%G)AQ%LZzE0y9EY7ak(L17iH;Q{t2d9HmCmvYp%wJ)hc}wuPc1 zmH1bA(UI{DUZ7TGVbY75b8L zW(j7o(1Yds(Z$>EM~2BuNj9U+;FmCLy|$Y}m-R^;rYDgGu|wPlJ&gdW=jk0Lgcx{0Nj)Oadnq zOpG^&U(uNHPe|1O`>Q(cXC{!DL$IemhfByB)WfIy_roaJbl zgzQ`3@ouw^RnbMDm=ussC&V>g$}wE0X*osDPahsT1*anD}pH7y;ZTw zS7Vca1K^iR&GAxy4zF9o(-w@%QM`AgLFF_CKa&-HyB;r+%yrQMZ@W1`kE=vY0(~vW ze6xm%F?=&F`LD`2O5v-D#Cu*E`WiTaGBy;yWNi@>c**D8ny@~z_8i7WB z0O|4PL3P47Q;GamHe0v;e}0EG!Q}o%HaGPdd@IuGr0oj$Wlu7V1)4 zF2uP3Pk2JC@HNk$L~MS$Nu;DD@|s($;emi|!;oGGEgnkiM$$*sEHL0teD%s1&o|wW zg<|M*pnvLgC!-#53tu28AI$$o zIzm#D^xlBcBExY|kqi*BC5*F&miLE^f3NFFnh49bU6-&ShC!A|*yTjXDu=!fw$)%= zYAHYj$U*EsIE$@i4$OiD_FM&^$Kn%pRXm5nrMql+ja=B<3sHmuxH+`6vzwh_C$at-OU_?nX*q=40#fU{+Nt=P>=8T$l@ops5=?78gz&k0lV?dA3&~%VYk-8-QiRGTM28%!P8ajq^BzN>Tvr+X$jY>!%NI zqrM8%h6yh7cjT!pzq_UuIDd#!o-II{U|p)&KvMmBeK4aN`k|_!!QCUu%SV|q+;jrlai8~m(>;zI!+}@GdN)vc0tHU)%V;W#$){krmW0x1|%J=|O6PqgQ>s6CK z(1Il^twKzAv;JIs^nFN_Mo5TY=h2OqoU_$(CDQ+jQK#O%oC2x)Bfv8axvYjV_txLy zggc}QZMB}P#_~#~$wa8YODd!!SDRA8fR9A(G<%WU8$E&0@gNRAs}5jBLo^Zgv>yGy zjiuG5J3{mab1h+;=aZy^IFUD10BScn+E@7{{{vy~-D9gfDUr4`P6QBazJtZ7VZN;+ z85h4hG}errGE}rG5Vv-z_-ioh+ngK^Y(|ZLw8fHn0zXVtD@|KE;opZI1)oAX3X)h7 zYe1+DZF=!U0hMOVFR94CPz)`SI3Nx`=AJBg^*3`as)_*haL2UJoAi`sFDBl!>tE55 z`&8$u<(yPNb8`65On(hO=0pxeOIXzA@a(OYk>?|`$=5?lKOl4nZq4U?6Iy?zffk=p zA$pRtX{(NjT#0H4W_1g?^$SQkW_V#FnXkjn(zsRY2pgP{Fkr)^#j~z2#A0JH#= z19pJ=i-c99>zm!mBvD79N!H!b!5?JmuCxD?{%pu)M}>a>guxLnJ(b(@TC^Zxd85wk=mDSW4+(Vff1*vZ{ufD zv#@V)@(Yp|EKsM&=p2wOcN+3Rzy$lLfY$BZ=*lzon_NH3mk5oOQ2?%+SX(>)&&mpk zceTQ#b#j#UI!?J2*9z5Jdixnj@bdFzTynm z=};gzw*s}av|JcHldHl1F>=w9efc-rb7+aGNjWd@^dkSEF}iH1^KuG-Vy5ujb~n%J zM?Uv|zF}r)Wbes0R4bsLR)Vz~OdqF>ST;;3%EzxI17M=0cE{$VRU;H@%{1)TF;U-O zUM%eUYYa&E-B+FltfBMol$Zc3cd^<*B3>*TZfriCcloNn138BeWxKo_7D z!{+Fwoc0o&m|)GB6h+aB4r>2_o6!*_8NAleMshPT!vY&P;-}~{wjJQ0?=C=>QY0d{f3MTd^2Ot)SW6Yr zT*o)8Xcz3rHCD-C8|(6bzUf@|>z83EBiADF14+;mVgx?N4*l)bPrVk8PcI*uOWpzC zhV+l$nyvF-uMaBTM2erM*IQXRQkvqeew3l;>|6-I0Dz0nQr z9VkE!1Jp?Fp@2$=-tOi|j8o&&w|PF?7n80{Ub!M5B9A=hVTGkd{?j_M9GL_%byq(+ zI;zV}`xc0>=CB|vbM&6o%`jc6zZ*${KSBYmBt`5h>Trblr?C#S5fTEEse3KGU)=sM zh-;<+K`_vWd+x^mj8>8u@ajDcEGvSL$Dl)!=TG9BBLJRVH>@1ISbG`KHtYh z;pHMad;2iS8on|>t2F}DE+tCc`M%^;ihNR%_>CFOY}0ID_wjDmILRtNo3}{I_Mfi0 zpzDj<7()K8jc1iy#)AXzniLvQ%yZF4ANrqysuTbk#cF0KgZnptbI|IUVVt#8lf}w< zdjcrSe7xidK#q(;n`l`iVmy8?(Dm5^AQFHM1-~}So*orLK@om<{aXMq^haIw7XsL7 zK?`UQ-(Ytr)~iw3|Lq&@RB>I~qW|e|O$l5Td3tmQQX>Tikei_(mNa9qj*f$AQM)hT zEZ&HA5hnlR9oD*wgr)?h#)O#pWG|^A`!t;UuQmY)Agc;CLOypfyNBG_@U-|R_prb# zh-KsKRzd0qSBBhL_NGSh`CdclE>uCVY`{FD)t!(`+<>Cz8s0l z;$J`-1(=#yETG1E9Z@#u7ra1U^;Y`*PPh=hMT7^ZCsKUH<@boFvso&^a8YvgEyBcg zfFvZEc8BfG1yw%kzGr$}m~Xo*gzA)kJqkl^JZjIHG}~eZ)nbY@epRDF-+p%DmTr~+ zQIY#PZ0BptJ$q7qu?mYZVK)WxDjo58*FmIJ@K}3{*BoS*S5;Lt_hK1B+%(JYThHNo zy5gk;lq#RS)?EF?bfq97V>#1Q=sLGpT-6 z{bNy-@V+zAX;0L{-;dJY-gIk)z1eII?QKB!j}QUfVML30ojrKHtu({`TC|%8CBh)= zYdkEiBC;Ldel(Ii&*+8Ac}L3H+qbFlW*-$k0Pc%WxUe_@#o^&${{p}RoZL8fR#vl@8mHVbV_0wC4qTI;zJkw){b z{@;cl2QjBIC0?IJZ%0=V@YZK(XI+KQeYCMuGNq4y<)qX-(l(RbAC0AZ)1n$!p_5n# z{(_`gN2>{gw6AC5JZNZ9OZB4W^!ORE5C_-@ce2B?B`=~S9zjJNtB5irXt`&ld}{il zke_ekBq_N&-1MSIzZ9yL^z(3E{P z^rSu*qjdP7;l~L3K;HN)}Su` zi6=GlL)SYmXMT=Q`go`3rOztX9SMw<^F0{;{K@R6@PNh-1;Q2I_`^!0McSpFjXFG*@N%T-TC;dw@{ApQ_5GMV?&el9yG)2UIFcmL}tDq8MszWh1wqAe zPxZzUDGScQ>le444hwDkGDLBFW)H*Bz^KQ(Ryb~oUJ)k6>U4^ODxOu#0|wf7CWci#n^b@wYfLD$QA%9P+xmsUU|b9s$m#%_Z@E9B_vpb)eRE0u`k3YZ*@0KmGKp=(p!+~oI0 z=>))zuw5RUf9wE9!<(p zX>;G&X(U*d+M08{H{_9UB%`3 z6=4A;5`X-+Eu8YhUP;kYH82VkkdK;q%~>ygD(VhA;sU_vi>*IMF!uOX0@VpQ=G(gc@(eR9iaL8NDRtn<%w@LxbZ zPbx+S6T5`ngiYoZZP!zd@6otCDpRzS@&Q(6i5>@jo?4VOvh+h5v4W*j zil&gff$C&I3)}Z)|2g{sHNe{bK*IuDrg)QlpD9LoQfX!i{w6p$a!q?~Y^i71#LUI4 zbAC`FKkFR5noJl8w0Z=W)Tb;YEUy!EF7BH4J-Dr1cs<_Q%dIt>|F=6~N;mw+XIVDB zIc~17n~s>OruRZVDIxmjNDQ1|?fu;Apb#sJ>O@V_Im4-z25zJpX=l!{ItFXIq7<`krYsNj85CL~Zmq+ynfSJ>~Wj>;h;JeXkunm^` zVmo(cxLbp_l!N$Qmeai>Sht3X_wVssqbp7p^#K#0v#$;77@eO5s>bcpsg?<4zj;un z180?{g;3Y}v>c;k6B$7>?bDI|zslZ*U$Hh}%r%7%FJQcv#z31gG)uT*&(AmxA?)o% z1f;7O`A&Ey%LmG^?9i|7gyeiE8s5m|aUp9mF$5XcYP}peoDq~2@iF0}XePlgxBIg9 zU`ir87t5<-m41y_opgj!PkOn1MYe7pa>^ZICXed&w^|$fMZj1Mwv3Of2XERdyIYFS zQhIGPyD~TMeosJ(0M@K>MSN^{a-AW21NZx`uX&%PXHHJem%?mTqSG_$BZ(j7&Q4BF z4BYe-$GY>hAGQ8cwlJju=6yd56F2XLDvV_1r|HAe4ZqFr8u2gVrlzRXsNZZ1-z9wm zbem7WajZ7RcHxsI#(Qc$aZ8R0lHVZ?Wvw6wlr`Gdx)<}LTHO1zQ@inttmQ@4^iA7k zj;Z&>@6wO;<(q%^MI}l)bXo(_a=6P>S7b(O=^Kh#Ru?+~p$k8b>!@p7A(9U~dgxP@ew3Ow&O7%WJ7a3SE2YModbyX5P0Qy3n4rCij8O#+U~AN2nA zZ?@Cag|5vmoU$j?ZL$QPAwBL$%t-TkYG1WXecB4&%NrIy`N}0dC+io%RdJ84`2k=x zcQ~InkG{(NB7b?j%!)AnI&lR~zt!@iPF6kDHs1=|W_A*Fyf+_HHvTMC9d8x3^IOx) zLrY9|W|0wGsV3I9B=E^GqF1@Od2YRLY;_=YTMZU1|0mG(_wV1q#3~J>%|={(`EsIt zKIJ-}4;|u7Mo-(7tRLzJUXMNzmAO{S{l573gz2~H=f1Na<^ry;m8Si~cX%odhgU*vo%y*z8PtN(K zKY-mn@Kb)WO(QPg4ZN?e?hj=lWWJ?0Y_jQ>g4fqdq8v8l4pGvGuwj`8)PU}eT=;>( zo<35%H8Q5Z2gp0rFh(G;neOQkRn5}7udz56|1nTZ3ff<1o`@*OR!G5tDX(7YPZUK#et?J}B&SGq$HmYQ)58x^d3)SpC zg5FhAd92s9bIX^-nkE@^-W=I2$Ivilsb42}x@>wpyHfuv{TC!Zd42WVjt}B|L^56{ zCzWoiOx{4*+U#hJbR1>-S+Iltz{wTDqy43)vpp^Fz7a^iC%pu)%VaCH2It+Itkfz<$~dsqMonrd4i!J32b5B9mGzh&ZRtqsqwR zhupQ4b}*Umgj58P2!BWeY!kI9?o>|8*SP}kw!ELE4fYqpn2$G1N@` z!<4IPo$C#kfAvE!G#T2gLlcMti3wfOymsyz@=147>jz#1vSVa;z~t(al|f@AO(;$i zn*<1KwEMXEE4#>@(Pv&@%4(d!n$!(|K(S8CR$;u4#q>=YYST^H@tG;zYu+pG9Fn0) zHJLoA|B)*@-BK4G2%39eP?+{?0OO&|w5DX2ul<;f#_xL)A2vt=>*wmhJEP~TR_J_H z{Jxnrr2dS=pmj~m!N`h4iI;CV6Y0pNwTQ2U<_e4OlBS{n>&l^lJ^%!KCw0K5c=K z@2HZy)iV$_g?dZbE9zL0i_%~AkBkSD0TyMx#@>Rr6X!8$TIK0qGTa_S4PHGj~59Wg5a?UzGz$N zP4h+_tGa#PkMRN)er1lvwkEUU0nH9~LXKJ^Mjv(ktga3&;*HZY5rAH4{|oaMg?u~Idg)P@p7rtwd{>{LT*lE(P!(|1*vh}I3Z!w;G6 z@qYDl>b7{X6|Uv1l|(UIQ|q@vP?Ht1b~fABJE_5~qQh9w%g*{j`{g(daWbZ{{FdyI6#z$rY;9{kb^A~{e($}R|>pS|Ih3GP~?WnH|zrCw5g@+a6a|$Ip#hHgQ^U&3{le6>0P_mpY zAd|)HSRgcgIMZ*zi!EO7r0pY3jXJ7FeICwW=j3+uS;kW|5p%o9ewMP8r9I;Lml3+# z5ql`om|#A)nV(Vn5<9=Un_G5p2NkydLJZQ2*#3jU%YwI8Q1pd=*hLxslKsuM{8O`M z3a?eB4{1h~ESY>_Y3Y}@@Mm>pW3trE>u{JPXe#7cU#W{g)SrbWa;8?of~I!MEIl*n zn{X$4R?6iEJv!zDL&1U_-hv&%56mNyUEA7;lXsrddL{;kw4UDF2dwd%mCC_ko*yfN zh$TAdGfnO4e805JhhX$tW?Ayy9oz#@aNs$k(@9aG7o;_pvmFf|o2MJ!mavYznEFGI z9rP@>#Way~Ea0`pN!QhB^95-YA&2%$5F-~N)r#^L9MI~qhjoRmypDLXetmdP_3atT zbmFh9tp}ePxB6C-!X%>K6aeb z6EJIhrvDGvg#elmG&5H;`}IlM*zIvvYSEs9Am2rt^<_qg{=oRwtIuDkn8-CioNR12 z<=6i*_6sziSFL`pDR=jr7kw{cX8OC8@Gd9>X|EuF8T@arB#s4dyHd2(yE5&rVQA5h z*G9EPw~75({}W`VZ38M29;l8;lvyY#`I_r_!B|KmPMus4hNe0p!99jIqV9NBM3|E~ zMPi9?w6QtpZM_HEM@k5x8ffGh&!gJ$GtMqAU1HlRfe5%PmIxFGokDk=$Rnk&(>0+T zM;h2$SZOExK)(u_H>yUV@V4!Yy_v#4*<)d3H%XvZc0~RkV9h&l^{p4?!*F$ViWr83 zZ$XpG)BI(bvLFlJl6BS&fBqeNLaA#)b^9m@5oiZ@T9kQw$U{d2RS}^S@me;UgXsg^ zog$1{aN!!ys*@UJ7XVoS;C)yh4GGlDAE~~jI`o8gIkOT)9C(YJ2)?8q|62}k zdcQn+ZBQbP>X)~(G2QTdm@YGPi!I-mMEFg1(lzi=SWRea#~~MTD!P@y&gR|*^-5tH zujg2&IdN+alCCFM(gC3>U)p0u(fIA>dyTLS0BQr?pSC3kYR5&SlthVaN&Mk?8!NUH zErS*L?EZw{0nwCKajsWJmS)1^_d3Zna*-dz309t`NNr{xj3HN4_X_%7}&{Mc}jm|3igGKZ>&DBMn{ zC#8g8rXAe`hg-zXWulvq+tE!K8xC>vn&?V+gQjdS1tkA<)4N#rN#M0p!~R|Oz<<$z zHEOQD<&9pT!rrCvf94TxhXAXX`fu>blRk)sjj4G<&10=x;!4Z~)Em;r>5w^2FC#3n z8r1{xeIuZ@)?K@nK+b9=gEXnf918KK=fVLESu&+n#gS5irv_eBL9UGse-jHfpBv~H z&`X+{(VQ_Cf`tYgm$VLG$ulVAM*zqkAZxi{FqUtnniK|LVWcOVGCP1j8n#r_NW_OJKSwmA*3>$X921>)F~p<1Ch(zgZ>po2Ro#qOTtNZl`}hyjkP(?pB(Niq^u9#O$*zX+B1<>q7Z> zI~_Zo9a-vR+;F&Ss2yT|fd{=45h-#5JGv_ziIN!#ouQJXtiFNZjQqc$fG-?ZrM26E zc3Br4>7d(K9ArPAkub5n-7Q$FaHP1>?jc3$%m1hm(>*BIl`y}S*_%5~dqfqx&biz- zhLdK+u@|OjdNmMLGmhO*_3c;_!bTw7kT(Mqs|G1~pvWe$-RSS-!Q2VbF%W|+ zf#$*r$tOj>NeoV@o~ESV)>Fn+#K7yyXEKB)?4#ilJh|)8HQ^R1NJ^PmOsF&zthGcg z-(X3u>aFex#S`1o3Gb8?LpR_79rq@po=Y~JNr+&9|DU9>>HAUAkXJK6rsW+w$chfO9I3(Y>Q`J_uiMGj(ft(pi4FYgA@%~u zAEbmp6m}wbqg}&-sM-A?OHwo&jBt{rh^8C<^z3{JsxJtDgoN^UeDkBF}E6 zlx-=%zH-f~S{9?uCY5|(mcp{*N%=}fS@WxYu*4(m%o~RWx^)?$Me}BtkCp?H-d2d*J_ia)Q`pKjl<`1wX{;&6^e=nFcVOP}TL&k~( Sometimes the maintainers will refer to NATS as "nats core" and streaming as "stan" or "streaming." +> Sometimes the maintainers will refer to NATS as "nats core" and streaming as "stan" or "streaming". Messages to the streaming service are opaque byte arrays, just as they are with NATS. However, the streaming server protocol uses protocol buffers to wrap these byte arrays. So if you listen to the NATS traffic the messages will appear as protocol buffers, while the actual data sent and received will simply be byte arrays. NATS streaming uses the concept of a channel to represent an ordered collection of messages. Clients send to and receive from channels instead of subjects. The subjects used by the streaming libraries and server are managed internally. Channels do not currently support wildcard. Channels aren’t raw subjects. Streaming isn’t raw NATS. The streaming libraries hide some of the differences. -Think of channels as a ring buffer. Messages are added until the configured limit is reached. Old messages are removed to make room for new ones. Old messages can expire, based on configuration. Subscriptions don’t affect channel content. Subscriptions are like a cursor on the ring buffer. - -![ring buffer](../resources/ring_buffer.png) +Think of channels as a First In First Out (FIFO) queue. Messages are added until the configured limit is reached. Old messages are removed to make room for new ones. Old messages can expire, based on configuration. Subscriptions don’t affect channel content, that is, when a message is acknowledged, it is not removed from the channel. Positions in the channel are specified in multiple ways: @@ -20,8 +18,6 @@ Positions in the channel are specified in multiple ways: New subscriptions can also specify last received to indicate they only want new messages. Sequence numbers are persistent, when message #1 goes away the oldest message is message #2. Trying to go to a position before the oldest message will be moved to the oldest message. -![starting positions](../resources/start_positions.png) - ## Subscription Types NATS Streaming supports several types of subscriptions: diff --git a/developer/streaming/acks.md b/developer/streaming/acks.md index 72eeb94..c753258 100644 --- a/developer/streaming/acks.md +++ b/developer/streaming/acks.md @@ -14,7 +14,7 @@ sub, err := sc.Subscribe("foo", }, stan.SetManualAckMode(), stan.AckWait(aw)) ``` -## Max In Flight +# Max In Flight Subscribers can set max in flight to rate limit incoming messages. The server will send at most “max in flight” messages before receiving an acknowledgement. Setting max in flight to 1 insures every message is processed in order. diff --git a/developer/streaming/connecting.md b/developer/streaming/connecting.md index 20e83c2..eb12cb6 100644 --- a/developer/streaming/connecting.md +++ b/developer/streaming/connecting.md @@ -1,11 +1,22 @@ # Connecting to NATS Streaming -NATS Streaming is a service on top of NATS. To connect to the service you first connect to NATS and then use the client library to communicate with the server over your NATS connection. Most of the libraries provide a convenience mechanism for connecting in a single step. These convenience methods will take some NATS options, like the server, and perform the NATS connection first, then then run the protocol to connect to the streaming server. +First, it is recommended to understand the relation between Streaming and core NATS. You should familiarize yourself with the [concept](/nats_streaming/relation-to-nats.md). + +NATS Streaming is a service on top of NATS. To connect to the service you first connect to NATS and then use the client library to communicate with the server over your NATS connection. Most of the libraries provide a convenience mechanism for connecting in a single step. These convenience methods will take some NATS options, like the cluster ID, and perform the NATS connection first, then run the protocol to connect to the streaming server. Connecting to a streaming server requires a cluster id, defined by the server configuration, and a client ID defined by the client. +_Client ID should contain only alphanumeric characters, `-` or `_`_ + +Connecting to a server running locally on the default port is as simple as this: + ```go -sc, err := stan.Connect(clusterID, clientID, stan.NatsURL(“nats://localhost:4222”)) +sc, err := stan.Connect(clusterID, clientID) +``` + +If the server runs on port `1234`: +```go +sc, err := stan.Connect(clusterID, clientID, stan.NatsURL(“nats://localhost:1234)) ``` Sometimes you may want to provide NATS settings that aren't available in the streaming libraries connect method. Or, you may want to reuse a NATS connection instead of creating a new one. In this case the libraries generally provide a way to connect to streaming with an existing NATS connection: diff --git a/developer/streaming/durables.md b/developer/streaming/durables.md index e10023b..c5d2550 100644 --- a/developer/streaming/durables.md +++ b/developer/streaming/durables.md @@ -1,7 +1,13 @@ # Durable Subscriptions -Regular subscriptions remember their position while the client is connected. If the client disconnects the position is lost. Durable subscriptions remember their position even if the client is disconnected. Durable subscriptions identify themselves with a name. Connect and disconnect won’t affect the durable subscriptions position in the channel. Unsubscribe will clear the durable subscription. +Regular subscriptions remember their position while the client is connected. If the client disconnects the position is lost. Durable subscriptions remember their position even if the client is disconnected. + +Durable subscriptions identify themselves with a name. Connect and disconnect won’t affect the durable subscriptions position in the channel. ```go sc.Subscribe("foo", func(m *stan.Msg) {...}, stan.DurableName("my-durable")) ``` + +Unsubscribe will cause the server to completely remove the durable subscription. + +Check the [concepts](/nats_streaming/channels/subscriptions/durable.md) section for more information. \ No newline at end of file diff --git a/developer/streaming/embedding.md b/developer/streaming/embedding.md new file mode 100644 index 0000000..5fc5f4c --- /dev/null +++ b/developer/streaming/embedding.md @@ -0,0 +1,111 @@ +# Embedding NATS Streaming + +Embedding a NATS Streaming Server in your own code is easy. Simply import: + +``` + stand "github.com/nats-io/nats-streaming-server/server" +``` + +(Note: we chose `stand` here, but you don't have to use that name) + +Then if you want to use default options, it is as simple as doing: + +``` + s, err := stand.RunServer("mystreamingserver") +``` + +If you want a more advance configuration, then you need to pass options. For instance, let's start the server with a file store instead of memory. + +First import the stores package so we have access to the store type. + +``` + stores "github.com/nats-io/nats-streaming-server/stores" +``` + +Then get the default options and override some of them: + +``` + opts := stand.GetDefaultOptions() + opts.StoreType = stores.TypeFile + opts.FilestoreDir = "datastore" + s, err := stand.RunServerWithOpts(opts, nil) +``` + +However, since the NATS Streaming Server project vendors NATS Server (that it uses as the communication layer with its clients and other servers in the cluster, there are some limitations. + +If you were to import `github.com/nats-io/nats-server/server`, instantiate a NATS `Options` structure, configure it and pass it to the second argument of `RunServerWithOpts`, you would get a compiler error. For instance doing this does not work: + +``` +import ( + natsd "github.com/nats-io/nats-server/server" + stand "github.com/nats-io/nats-streaming-server/server" + stores "github.com/nats-io/nats-streaming-server/stores" +) + +(...) + + nopts := &natsd.Options{} + nopts.Port = 4223 + + s, err := stand.RunServerWithOpts(nil, nopts) +``` + +You would get: + +``` +./myapp.go:36:35: cannot use nopts (type *"myapp/vendor/github.com/nats-io/nats-server/server".Options) as type *"myapp/vendor/github.com/nats-io/nats-streaming-server/vendor/github.com/nats-io/gnatsd/server".Options in argument to "myapp/vendor/github.com/nats-io/nats-streaming-server/server".RunServerWithOpts +``` + +To workaround this issue, the NATS Streaming Server package provides a function `NewNATSOptions()` that is suitable for this approach: + +``` + nopts := stand.NewNATSOptions() + nopts.Port = 4223 + + s, err := stand.RunServerWithOpts(nil, nopts) +``` + +That will work. + +But, if you want to do advanced NATS configuration that requires types or interfaces that belong to the NATS Server package, then this approach won't work. In this case you need to run the NATS Server indepently and have the NATS Streaming Server connects to it. Here is how: + +``` + // This configure the NATS Server using natsd package + nopts := &natsd.Options{} + nopts.HTTPPort = 8222 + nopts.Port = 4223 + + // Setting a customer client authentication requires the NATS Server Authentication interface. + nopts.CustomClientAuthentication = &myCustomClientAuth{} + + // Create the NATS Server + ns := natsd.New(nopts) + + // Start it as a go routine + go ns.Start() + + // Wait for it to be able to accept connections + if !ns.ReadyForConnections(10 * time.Second) { + panic("not able to start") + } + + // Get NATS Streaming Server default options + opts := stand.GetDefaultOptions() + + // Point to the NATS Server with host/port used above + opts.NATSServerURL = "nats://localhost:4223" + + // Now we want to setup the monitoring port for NATS Streaming. + // We still need NATS Options to do so, so create NATS Options + // using the NewNATSOptions() from the streaming server package. + snopts := stand.NewNATSOptions() + snopts.HTTPPort = 8223 + + // Now run the server with the streaming and streaming/nats options. + s, err := stand.RunServerWithOpts(opts, snopts) + if err != nil { + panic(err) + } +``` + +The above seem involved, but it really only if you use very advanced NATS Server options. \ No newline at end of file diff --git a/developer/streaming/protocol.md b/developer/streaming/protocol.md new file mode 100644 index 0000000..a964caa --- /dev/null +++ b/developer/streaming/protocol.md @@ -0,0 +1,265 @@ +# Writing your own client library + +You can find a list of all supported client libraries [here](https://nats.io/download/). There is also links to community contributed clients. + +In the event you would want to write your own NATS Streaming library, you could have a look at existing libraries to understand the flow. But you need to use [Google Protocol Buffers](https://developers.google.com/protocol-buffers/) to exchange protocols between the client and the server. + +## NATS Streaming Protocol + +The NATS streaming protocol sits atop the core NATS protocol and uses [Google's Protocol Buffers](https://developers.google.com/protocol-buffers/). Protocol buffer messages are marshaled into bytes and published as NATS messages on specific subjects described below. In communicating with the NATS Streaming Server, the NATS request/reply pattern is used for all protocol messages that have a corresponding reply. + +### NATS streaming protocol conventions + +**Subject names**: Subject names, including reply subject (INBOX) names, are case-sensitive and must be non-empty alphanumeric strings with no embedded whitespace, and optionally token-delimited using the dot character (`.`), e.g.: + +`FOO`, `BAR`, `foo.bar`, `foo.BAR`, `FOO.BAR` and `FOO.BAR.BAZ` are all valid subject names + +`FOO. BAR`, `foo. .bar` and`foo..bar` are *not- valid subject names + +**Wildcards**: NATS streaming does **not** support wildcards in subject subscriptions + +**Protocol definition**: The fields of NATS streaming protocol messages are defined in the NATS streaming client [protocol file](https://github.com/nats-io/stan.go/blob/master/pb/protocol.proto). + +### NATS streaming protocol messages + +The following table briefly describes the NATS streaming protocol messages. + +Click the name to see more detailed information, including usage: + +#### Protocols + +| Message Name | Sent By | Description +| ------------------------------------------------- |:--------|:-------------------------------------------- +| [`ConnectRequest`](#connectrequest) | Client | Request to connect to the NATS Streaming Server +| [`ConnectResponse`](#connectresponse) | Server | Result of a connection request +| [`SubscriptionRequest`](#subscriptionrequest) | Client | Request sent to subscribe and retrieve data +| [`SubscriptionResponse`](#subscriptionresponse) | Server | Result of a subscription request +| [`UnsubscribeRequest`](#unsubscriberequest) | Client | Unsubscribe from a subject +| [`PubMsg`](#pubmsg) | Client | Publish a message to a subject +| [`PubAck`](#puback) | Server | An acknowledgement that a published message has been processed on the server +| [`MsgProto`](#msgproto) | Server | A message from the NATS Streaming Server to a subscribing client +| [`Ack`](#ack) | Client | Acknowledges that a message has been received +| [`Ping`](#ping) | Client | Ping sent to server to detect connection loss +| [`PingResponse`](#pingresponse) | Server | Result of a Ping +| [`CloseRequest`](#closerequest) | Client | Request sent to close the connection to the NATS Streaming Server +| [`CloseResp`](#closeresponse) | Server | Result of the close request + +The following sections explain each protocol message. + +#### ConnectRequest + +##### Description + +A connection request is sent when a streaming client connects to the NATS Streaming Server. The connection request contains a unique identifier representing the client, and an inbox subject the client will listen on for incoming heartbeats. The identifier **must** be unique; a connection attempt with an identifier currently in use will fail. The inbox subject is the subject where the client receives incoming heartbeats, and responds by publishing an empty NATS message to the reply subject, indicating it is alive. The NATS Streaming Server will return a [ConnectResponse](#connectresponse) message to the reply subject specified in the NATS request message. + +More advanced libraries can set the protocol to 1 and send a connection ID which in combination with ping interval and ping max out allows the library to detect that the connection to the server is lost. + +This request is published to a subject comprised of the `.cluster-id`, for example, if a NATS Streaming Server was started with a cluster-id of `mycluster`, and the default prefix was used, the client publishes to `_STAN.discover.mycluster` + +##### Message Structure + +- `clientID`: A unique identifier for a client +- `heartbeatInbox`: An inbox to which the NATS Streaming Server will send heartbeats for the client to process +- `protocol`: Protocol the client is at +- `connID`: Connection ID, a way to uniquely identify a connection (no connection should ever have the same) +- `pingInterval`: Interval at which client wishes to send PINGs (expressed in seconds) +- `pingMaxOut`: Maximum number of PINGs without a response after which the connection can be considered lost + +[Back to table](#protocols) + + +#### ConnectResponse + +##### Description + +After a `ConnectRequest` is published, the NATS Streaming Server responds with this message on the reply subject of the underlying NATS request. The NATS Streaming Server requires the client to make requests and publish messages on certain subjects (described above), and when a connection is successful, the client saves the information returned to be used in sending other NATS streaming protocol messages. In the event the connection was not successful, an error is returned in the `error` field. + +##### Message Structure + +- `pubPrefix`: Prefix to use when publishing +- `subRequests`: Subject used for subscription requests +- `unsubRequests`: Subject used for unsubscribe requests +- `closeRequests`: Subject for closing a connection +- `error`: An error string, which will be empty/omitted upon success +- `subCloseRequests`: Subject to use for subscription close requests +- `pingRequests`: Subject to use for PING requests +- `pingInterval`: Interval at which client should send PINGs (expressed in seconds). +- `pingMaxOut`: Maximum number of PINGs without a response after which the connection can be considered lost +- `protocol`: Protocol version the server is at +- `publicKey`: Reserved for future use + +[Back to table](#protocols) + + +#### SubscriptionRequest + +##### Description + +A `SubscriptionRequest` is published on the subject returned in the `subRequests` field of a [ConnectResponse](#connectresponse), and creates a subscription to a subject on the NATS Streaming Server. This will return a [SubscriptionResponse](#subscriptionresponse) message to the reply subject specified in the NATS protocol request message. + +##### Message Structure + +- `clientID`: Client ID originally provided in the [ConnectRequest](#connectrequest) +- `subject`: Formal subject to subscribe to, e.g. foo.bar +- `qGroup`: Optional queue group +- `inbox`: Inbox subject to deliver messages on +- `maxInFlight`: Maximum inflight messages without an acknowledgement allowed +- `ackWaitInSecs`: Timeout for receiving an acknowledgement from the client +- `durableName`: Optional durable name which survives client restarts +- `startPosition`: An enumerated type specifying the point in history to start replaying data +- `startSequence`: Optional start sequence number +- `startTimeDelta`: Optional start time + +##### StartPosition enumeration + +- `NewOnly`: Send only new messages +- `LastReceived`: Send only the last received message +- `TimeDeltaStart`: Send messages from duration specified in the `startTimeDelta` field. +- `SequenceStart`: Send messages starting from the sequence in the `startSequence` field. +- `First`: Send all available messages + +[Back to table](#protocols) + + +#### SubscriptionResponse + +##### Description + +The `SubscriptionResponse` message is the response from the `SubscriptionRequest`. After a client has processed an incoming [MsgProto](#msgproto) message, it must send an acknowledgement to the `ackInbox` subject provided here. + +##### Message Structure + +- `ackInbox`: subject the client sends message acknowledgements to the NATS Streaming Server +- `error`: error string, empty/omitted if no error + +[Back to table](#protocols) + + +#### UnsubscribeRequest + +##### Description + +The `UnsubscribeRequest` closes or unsubcribes the subscription from the specified subject. The inbox specified is the `inbox` returned from the NATS Streaming Server in the `SubscriptionResponse`. Depending on which subject this request is sent, the action will result in close (if sent to subject `subCloseRequests`) or unsubscribe (if sent to subject `unsubRequests`) + +##### Message Structure + +- `clientID`: Client ID originally provided in the [ConnectRequest](#connectrequest) +- `subject`: Subject for the subscription +- `inbox`: Inbox subject to identify subscription +- `durableName`: Optional durable name which survives client restarts + +[Back to table](#protocols) + + +#### PubMsg + +##### Description + +The `PubMsg` protocol message is published from a client to the NATS Streaming Server. The GUID must be unique, and is returned in the [PubAck](#puback) message to correlate the success or failure of storing this particular message. + +##### Message Structure + +- `clientID`: Client ID originally provided in the [ConnectRequest](#connectrequest) +- `guid`: a guid generated for this particular message +- `subject`: subject +- `data`: payload +- `connID`: Connection ID. For servers that know about this field, clientID can be omitted + +[Back to table](#protocols) + + +#### PubAck + +##### Description + +The `PubAck` message is an acknowledgement from the NATS Streaming Server that a message has been processed. The message arrives on the subject specified on the reply subject of the NATS message the `PubMsg` was published on. The GUID is the same GUID used in the `PubMsg` being acknowledged. If an error string is present, the message was not persisted by the NATS Streaming Server and no guarantees regarding persistence are honored. `PubAck` messages may be handled asynchronously from their corresponding `PubMsg` in the client. + +##### Message Structure + +- `guid`: GUID of the message being acknowledged by the NATS Streaming Server +- `error`: An error string, empty/omitted if no error + +[Back to table](#protocols) + + +#### MsgProto + +##### Description + +The `MsgProto` message is received by client from the NATS Streaming Server, containing the payload of messages sent by a publisher. A `MsgProto` message that is not acknowledged with an [Ack](#ack) message within the duration specified by the `ackWaitInSecs` field of the subscription request will be redelivered. + +##### Message Structure + +- `sequence`: Globally ordered sequence number for the subject's channel +- `subject`: Subject +- `data`: Payload +- `timestamp`: Time the message was stored in the server. +- `redelivered`: Flag specifying if the message is being redelivered + +[Back to table](#protocols) + + +#### Ack + +##### Description + +An `Ack` message is an acknowledgement from the client that a [MsgProto](#msgproto) message has been considered received. It is published to the `ackInbox` field of the [SubscriptionResponse](#subscriptionresponse). + +##### Message Structure + +- `subject`: Subject of the message being acknowledged +- `sequence`: Sequence of the message being acknowledged + +[Back to table](#protocols) + + +#### Ping + +##### Description + +A `Ping` message is sent to the server at configured interval to check that the connection ID is still valid. This should be used only if client is at protocol 1, and has sent a `connID` in the [ConnectRequest](#connectrequest) protocol. + +##### Message Structure + +- `connID`: The connection ID + +[Back to table](#protocols) + + +#### PingResponse + +##### Description + +This is a response from the server to a `Ping` from the client. If the content is not empty, it will be the error indicating to the client why the connection is no longer valid. + +##### Message Structure + +- `error`: Error string, empty/omitted if no error + +[Back to table](#protocols) + + +#### CloseRequest + +##### Description + +A `CloseRequest` message is published on the `closeRequests` subject from the [ConnectResponse](#connectresponse), and notifies the NATS Streaming Server that the client connection is closing, allowing the server to free up resources. This message should **always** be sent when a client is finished using a connection. + +##### Message Structure + +- `clientID`: Client ID originally provided in the [ConnectRequest](#connectrequest) + +[Back to table](#protocols) + + +#### CloseResponse + +##### Description + +The `CloseResponse` is sent by the NATS Streaming Server on the reply subject of the `CloseRequest` NATS message. This response contains any error that may have occurred with the corresponding close call. + +##### Message Structure + +- `error`: error string, empty/omitted if no error + +[Back to table](#protocols) \ No newline at end of file diff --git a/developer/streaming/publishing.md b/developer/streaming/publishing.md index b9eaa28..bc140fa 100644 --- a/developer/streaming/publishing.md +++ b/developer/streaming/publishing.md @@ -1,6 +1,6 @@ # Publishing to a Channel -The streaming client library can provide a method for publishing synchronously. .These publish methods block until the ACK is returned by the server. An error or exception is used to indicate a timeout or other error. +The streaming client library can provide a method for publishing synchronously. These publish methods block until the ACK is returned by the server. An error or exception is used to indicate a timeout or other error. ```go err := sc.Publish("foo", []byte("Hello World")) @@ -12,4 +12,10 @@ Streaming libraries can also provide a way to publish asynchronously. An ACK cal ackHandler := func(ackedNuid string, err error){ ... } nuid, err := sc.PublishAsync("foo", []byte("Hello World"), ackHandler) -``` \ No newline at end of file +``` + +Even in this mode, the call will still block if the library has a number of published messages without having received an ACK from the server. The default can be changed when creating the connection. + +```go +sc, err := sc.Connect(clusterID, clientName, stan.MaxPubAcksInflight(1000)) +``` diff --git a/developer/streaming/queues.md b/developer/streaming/queues.md index 0cad323..bb86eb4 100644 --- a/developer/streaming/queues.md +++ b/developer/streaming/queues.md @@ -1,11 +1,37 @@ # Queue Subscriptions -Queue subscriptions are created like other subscriptions with the addition of a queue name. All subscriptions, across clients, share the queue based on this unique name. Other subscriptions can receive messages independently of the queue groups. Unsubscribe removes a client from a group, the last unsubscribe kills the group. Max in flight is per subscription. +Queue subscriptions are created like other subscriptions with the addition of a queue name. ```go -qsub1, _ := sc.QueueSubscribe(channelID, +qsub1, _ := sc.QueueSubscribe(channelName, queueName, func(m *stan.Msg) {...}) -qsub2, _ := sc.QueueSubscribe(channelID, +qsub2, _ := sc.QueueSubscribe(channelName, queueName, func(m *stan.Msg) {...}) -``` \ No newline at end of file +``` + +Multiple subscriptions using the same channel and queue name are members of the same queue group. That means that if a message is published on that channel, only one member of the group receives the message. Other subscriptions receive messages independently of the queue groups, that is, a message is delivered to all subscriptions and one member of each queue group. + +To create a durable queue subscription, simply add a durable name: +```go +qsub, err := sc.QueueSubscribe(channelName, + queueName, func(m *stan.Msg) {...}, + stan.DurableName("durable-name")) +``` + +Subscriptions options apply to each member independently, notably, the `AckWait` and `MaxInflight`. Those two members of the same queue group use different options for redelivery and max inflight. +```go +qsub1, _ := sc.QueueSubscribe(channelName, + queueName, func(m *stan.Msg) {...}, + stan.AckWait(5*time.Second), + stan.MaxInflight(5)) + +qsub2, _ := sc.QueueSubscribe(channelName, + queueName, func(m *stan.Msg) {...}, + stan.AckWait(20*time.Second), + stan.MaxInflight(10)) +``` + +If the queue subscription is durable, only the last member calling `Unsubscribe()` will cause the durable queue group to be removed from the server. + +Check the [concepts](/nats_streaming/channels/subscriptions/queue-group.md) section for more information. \ No newline at end of file diff --git a/developer/streaming/receiving.md b/developer/streaming/receiving.md index 0811e06..7841e6c 100644 --- a/developer/streaming/receiving.md +++ b/developer/streaming/receiving.md @@ -9,6 +9,10 @@ Subscriptions come in several forms: * Queue * Queue/Durable +For more details on the various types, check the [concept](/nats_streaming/channels/subscriptions/subscriptions.md) section. + +***Note: message callbacks are invoked serially, one message at a time. If your application does not care about processing ordering and would prefer the messages to be dispatched concurrently, it is the application responsibility to move them to some internal queue to be picked up by threads/go routines.*** + Subscriptions set their starting position on creation using position or time. For example, in Go you can start at: * The last message received @@ -43,4 +47,26 @@ var startTime time.Time sub, err := sc.Subscribe("foo", func(m *stan.Msg) {...}, stan.StartAtTime(startTime)) -``` \ No newline at end of file +``` + +To set the delay after which the server should attempt to redeliver a message for which it has not receive an acknowledgment: + +```go +sub, err := sc.Subscribe("foo", + func(m *stan.Msg) {...}, + stan.AckWait(20*time.Second)) +``` + +When an application wishes to stop receiving, but want to maintain the connection opened, the subscription should be closed. There are two ways to stop a subscription, either "close" it, or "unsubscribe" it. For non durable subscriptions, this is equivalent since the subscription will be completely removed. For durable subscriptions, close means that the server will stop delivering, but remember the durable subscription. Unsubscribe, however, means that the server will remove the state of this subscription. + +To simply close: +```go +err := sub.Close() +``` + +To unsubscribe: +```go +err := sub.Unsubscribe() +``` + +_Note: If a connection is closed without explicitly closing the subscriptions, the subscriptions are implicitly closed, not unsubscribed._ diff --git a/nats_streaming/channels/channels.md b/nats_streaming/channels/channels.md new file mode 100644 index 0000000..f38ef3f --- /dev/null +++ b/nats_streaming/channels/channels.md @@ -0,0 +1,7 @@ +# Channels + +Channels are at the heart of the NATS Streaming Server. Channels are subjects clients send data to and consume from. + +***Note: NATS Streaming server does not support wildcard for channels, that is, one cannot subscribe on `foo.*`, or `>`, etc...*** + +The number of channels can be limited (and is by default) through configuration. Messages produced to a channel are stored in a message log inside this channel. \ No newline at end of file diff --git a/nats_streaming/channels/message-log.md b/nats_streaming/channels/message-log.md new file mode 100644 index 0000000..058f632 --- /dev/null +++ b/nats_streaming/channels/message-log.md @@ -0,0 +1,5 @@ +# Message Log + +You can view a message log as a First In First Out (FIFO) queue. Messages are appended to the end of the log. If a limit is set globally for all channels, or specifically for this channel, when the limit is reached, older messages are removed to make room for the new ones. + +But except for the administrative size/age limit set for a message log, messages are not removed due to consumers consuming them. In fact, messages are stored regardless of the presence of subscriptions on that channel. diff --git a/nats_streaming/channels/subscriptions/durable.md b/nats_streaming/channels/subscriptions/durable.md new file mode 100644 index 0000000..16c0864 --- /dev/null +++ b/nats_streaming/channels/subscriptions/durable.md @@ -0,0 +1,9 @@ +# Durable + +If an application wishes to resume message consumption from where it previously stopped, it needs to create a durable subscription. It does so by providing a durable name, which is combined with the client ID provided when the client created its connection. The server then maintain the state for this subscription even after the client connection is closed. + +***Note: The starting position given by the client when restarting a durable subscription is ignored.*** + +When the application wants to stop receiving messages on a durable subscription, it should close - but *not unsubscribe*- this subscription. If a given client library does not have the option to close a subscription, the application should close the connection instead. + +When the application wants to delete the subscription, it must unsubscribe it. Once unsubscribed, the state is removed and it is then possible to re-use the durable name, but it will be considered a brand new durable subscription, with the start position being the one given by the client when creating the durable subscription. \ No newline at end of file diff --git a/nats_streaming/channels/subscriptions/queue-group.md b/nats_streaming/channels/subscriptions/queue-group.md new file mode 100644 index 0000000..527b8ae --- /dev/null +++ b/nats_streaming/channels/subscriptions/queue-group.md @@ -0,0 +1,11 @@ +# Queue Group + +When consumers want to consume from the same channel but each receive a different message, as opposed to all receiving the same messages, they need to create a queue subscription. When a queue group name is specified, the server will send each messages from the log to a single consumer in the group. The distribution of these messages is not specified, therefore applications should not rely on an expected delivery scheme. + +After the first queue member is created, any other member joining the group will receive messages based on where the server is in the message log for that particular group. That means that starting position given by joining members is ignored by the server. + +When the last member of the group leaves (subscription unsubscribed/closed/or connection closed), the group is removed from the server. The next application creating a subscription with the same name will create a new group, starting at the start position given in the subscription request. + +A queue subscription can also be durable. For that, the client needs to provide a queue and durable name. The behavior is, as you would expect, a combination of queue and durable subscription. Unlike a durable subscription, though, the client ID is not part of the queue group name. It makes sense, because since client ID must be unique, it would prevent more than one connection to participate in the queue group. The main difference between a queue subscription and a durable one, is that when the last member leaves the group, the state of the group will be maintained by the server. Later, when a member rejoins the group, the delivery will resume. + +***Note: For a durable queue subscription, the last member to * unsubscribe * (not simply close) causes the group to be removed from the server.*** diff --git a/nats_streaming/channels/subscriptions/redelivery.md b/nats_streaming/channels/subscriptions/redelivery.md new file mode 100644 index 0000000..ac76c8b --- /dev/null +++ b/nats_streaming/channels/subscriptions/redelivery.md @@ -0,0 +1,11 @@ +# Redelivery + +When the server sends a message to a consumer, it expects to receive an ACK from this consumer. The consumer is the one specifying how long the server should wait before resending all unacknowledged messages to the consumer. + +When the server restarts and recovers unacknowledged messages for a subscription, it will first attempt to redelivery those messages before sending new messages. However, if during the initial redelivery some messages don't make it to the client, the server cannot know that and will enable delivery of new messages. + +***So it is possible for an application to receive redelivered messages mixed with new messages. This is typically what happens outside of the server restart scenario.*** + +For queue subscriptions, if a member has unacknowledged messages, when this member `AckWait` (which is the duration given to the server before the server should attempt to redeliver unacknowledged messages) time elapses, the messages are redelivered to any other member in the group (including itself). + +If a queue member leaves the group, its unacknowledged messages are redistributed to other queue members. \ No newline at end of file diff --git a/nats_streaming/channels/subscriptions/regular.md b/nats_streaming/channels/subscriptions/regular.md new file mode 100644 index 0000000..a3be9c6 --- /dev/null +++ b/nats_streaming/channels/subscriptions/regular.md @@ -0,0 +1,3 @@ +# Regular + +The state of these subscriptions is removed when they are unsubscribed or closed (which is equivalent for this type of subscription) or the client connection is closed (explicitly by the client, or closed by the server due to timeout). They do, however, survive a *server* failure (if running with a persistent store). diff --git a/nats_streaming/channels/subscriptions/subscriptions.md b/nats_streaming/channels/subscriptions/subscriptions.md new file mode 100644 index 0000000..3fea537 --- /dev/null +++ b/nats_streaming/channels/subscriptions/subscriptions.md @@ -0,0 +1,13 @@ +# Subscriptions + +A client creates a subscription on a given channel. Remember, there is no support for wildcards, so a subscription is really tied to +one and only one channel. The server will maintain the subscription state on behalf of the client until the later closes the subscription (or its connection). + +If there are messages in the log for this channel, messages will be sent to the consumer when the subscription is created. The server will +send up to the maximum number of inflight messages as given by the client when creating the subscription. + +When receiving ACKs from the consumer, the server will then deliver more messages, if more are available. + +A subscription can be created to start at any point in the message log, either by message sequence, or by time. + +Following pages describe all types of subscription. \ No newline at end of file diff --git a/nats_streaming/client-connections.md b/nats_streaming/client-connections.md new file mode 100644 index 0000000..fd04ede --- /dev/null +++ b/nats_streaming/client-connections.md @@ -0,0 +1,9 @@ +# Client Connections + +As described, clients are not directly connected to the streaming server. Instead, they send connection requests. The request includes a `client ID` which is used by the server to uniquely identify, and restrict, a given client. That is, no two connections with the same client ID will be able to run concurrently. + +This client ID links a given connection to its published messages, subscriptions, especially durable subscriptions. Indeed, durable subscriptions are stored as a combination of the client ID and durable name. More on durable subscriptions later. + +It is also used to resolve the issue of not having direct client connections to the server. For instance, say that a client crashes without closing the connection. It later restarts with the same client ID. The server will detect that this client ID is already in-use. It will try to contact that known client to its original private inbox. If the server does not receive a response - which would be the case if the client crashed - it will replace the old client with this new one.
+ +Otherwise, the server would reject the connection request since the client ID is already in-use. \ No newline at end of file diff --git a/nats_streaming/clustering/auto-configuration.md b/nats_streaming/clustering/auto-configuration.md new file mode 100644 index 0000000..2a19c63 --- /dev/null +++ b/nats_streaming/clustering/auto-configuration.md @@ -0,0 +1,17 @@ +# Auto Configuration + +We can also bootstrap a NATS Streaming cluster by starting one server as the seed node using the `-cluster_bootstrap` flag. This node will elect itself leader, so it's important to avoid starting multiple servers as seed. Once a seed node is started, other servers will automatically join the cluster. If the server is recovering, it will use the recovered cluster configuration. + +Here is an example of starting three servers in a cluster by starting one as the seed and letting the others automatically join: + +``` +nats-streaming-server -store file -dir store-a -clustered -cluster_bootstrap -nats_server nats://localhost:4222 + +nats-streaming-server -store file -dir store-b -clustered -nats_server nats://localhost:4222 + +nats-streaming-server -store file -dir store-c -clustered -nats_server nats://localhost:4222 +``` + +For a given cluster ID, if more than one server is started with `cluster_bootstrap` set to true, each server with this parameter will report the misconfiguration and exit. + +The very first server that bootstrapped the cluster can be restarted, however, the operator must remove the datastores of the other servers that were incorrectly started with the bootstrap parameter before attempting to restart them. If they are restarted -even without the `-cluster_bootstrap` parameter- but with existing state, they will once again start as a leader. diff --git a/nats_streaming/clustering/clustering.md b/nats_streaming/clustering/clustering.md new file mode 100644 index 0000000..4125f77 --- /dev/null +++ b/nats_streaming/clustering/clustering.md @@ -0,0 +1,9 @@ +# Clustering + +NATS Streaming Server supports clustering and data replication, implemented with the [Raft consensus algorithm](https://raft.github.io/), for the purposes of high availability. + +There are two ways to bootstrap a cluster: with an explicit cluster configuration or with "auto" configuration using a seed node. With the first, we provide the IDs of the nodes participating in the cluster. In this case, the participating nodes will elect a leader. With the second, we start one server as a seed node, which will elect itself as leader, and subsequent servers will automatically join the seed (note that this also works with the explicit cluster configuration once the leader has been established). With the second method, we need to be careful to avoid starting multiple servers as seed as this will result in a split-brain. Both of these configuration methods are shown in the sections below. + +It is recommended to run an odd number of servers in a cluster with a minimum of three servers to avoid split-brain scenarios. Note that if less than a majority of servers are available, the cluster cannot make progress, e.g. if two nodes go down in a cluster of three, the cluster is unavailable until at least one node comes back. + +Note about Channels Partitioning and Clustering. These two features are mutually exclusive. Trying to start a server with channels Partitioning and Clustering enabled will result in a startup error. Clustering requires all channels to be replicated in the cluster. \ No newline at end of file diff --git a/nats_streaming/clustering/configuration.md b/nats_streaming/clustering/configuration.md new file mode 100644 index 0000000..6160294 --- /dev/null +++ b/nats_streaming/clustering/configuration.md @@ -0,0 +1,95 @@ +# Configuration + +We can bootstrap a NATS Streaming cluster by providing the cluster topology using the `-cluster_peers` flag. This is simply the set of node IDs participating in the cluster. Note that once a leader is established, we can start subsequent servers without providing this configuration as they will automatically join the leader. If the server is recovering, it will use the recovered cluster configuration. + +Here is an example of starting three servers in a cluster. For this example, we run a separate NATS server which the Streaming servers connect to. + +``` +nats-streaming-server -store file -dir store-a -clustered -cluster_node_id a -cluster_peers b,c -nats_server nats://localhost:4222 + +nats-streaming-server -store file -dir store-b -clustered -cluster_node_id b -cluster_peers a,c -nats_server nats://localhost:4222 + +nats-streaming-server -store file -dir store-c -clustered -cluster_node_id c -cluster_peers a,b -nats_server nats://localhost:4222 +``` + +Note that once a leader is elected, subsequent servers can be started without providing the cluster configuration. They will automatically join the cluster. Similarly, the cluster node ID does not need to be provided as one will be automatically assigned. As long as the file store is used, this ID will be recovered on restart. + +``` +nats-streaming-server -store file -dir store-d -clustered -nats_server nats://localhost:4222 +``` + +The equivalent clustering configurations can be specified in a configuration file under the `cluster` group. See the [Configuring](#configuring) section for more information. + +Here is an example of a cluster of 3 nodes using the following configuration files. The nodes are running on `host1`, `host2` and `host3` respectively. + +NOTE If you have an existing NATS cluster and want to run NATS Streaming Cluster on top of that, see details at the end of this section. + +On `host1`, this configuration indicates that the server will accept client connections on port 4222. It will accept route connections on port 6222. It creates 2 routes, to `host2` and `host3` cluster port. + +It defines the NATS Streaming cluster name as `mycluster`, uses a store file that points to the `store` directory. The `cluster` section inside `streaming` makes the NATS Streaming server run in cluster mode. This configuration explicitly define each node id (`a` for `host1`) and list its peers. +``` +# NATS specific configuration +port: 4222 +cluster { + listen: 0.0.0.0:6222 + routes: ["nats://host2:6222", "nats://host3:6222"] +} + +# NATS Streaming specific configuration +streaming { + id: mycluster + store: file + dir: store + cluster { + node_id: "a" + peers: ["b", "c"] + } +} +``` + +Below is the configuration for the server running on `host2`. Notice how the routes are now to `host1` and `host3`. The other thing that changed is the node id that is set to `b` and peers are updated accordingly to `a` and `c`. + +Note that the `dir` configuration is also `store` but these are local directories and do not (actually must not) be shared. Each node will have its own copy of the datastore. You could have each configuration have a different value for `dir` if desired. +``` +# NATS specific configuration +port: 4222 +cluster { + listen: 0.0.0.0:6222 + routes: ["nats://host1:6222", "nats://host3:6222"] +} + +# NATS Streaming specific configuration +streaming { + id: mycluster + store: file + dir: store + cluster { + node_id: "b" + peers: ["a", "c"] + } +} +``` + +As you would expect, for `host3`, the routes are now to `host1` and `host2` and the node id is `c` while its peers +are `a` and `b`. +``` +# NATS specific configuration +port: 4222 +cluster { + listen: 0.0.0.0:6222 + routes: ["nats://host1:6222", "nats://host2:6222"] +} + +# NATS Streaming specific configuration +streaming { + id: mycluster + store: file + dir: store + cluster { + node_id: "c" + peers: ["a", "b"] + } +} +``` + +In the example above, the configuration assumes no existing NATS Cluster and therefore configure the NATS routes between each node. Should you want to use an existing NATS cluster, do not include the "NATS specific configuration" section, instead, add `nats_server_url` in the `streaming` section to point to the NATS server you want. \ No newline at end of file diff --git a/nats_streaming/clustering/containers.md b/nats_streaming/clustering/containers.md new file mode 100644 index 0000000..71cf971 --- /dev/null +++ b/nats_streaming/clustering/containers.md @@ -0,0 +1,9 @@ +# Containers + +When running the docker image of NATS Streaming Server, you will want to specify a mounted volume so that the data can be recovered. Your `-dir` parameter then points to a directory inside that mounted volume. However, after a restart you may get a failure with a message similar to this: +``` +[FTL] STREAM: Failed to start: streaming state was recovered but cluster log path "mycluster/a" is empty +``` +This is because the server recovered the streaming state (as pointed by `-dir` and located in the mounted volume), but did not recover the RAFT specific state that is by default stored in a directory named after your cluster id, relative to the current directory starting the executable. In the context of a container, this data will be lost after the container is stopped. + +In order to avoid this issue, you need to specify the `-cluster_log_path` and ensure that it points to the mounted volume so that the RAFT state can be recovered along with the Streaming state. diff --git a/nats_streaming/clustering/supported-stores.md b/nats_streaming/clustering/supported-stores.md new file mode 100644 index 0000000..0fb3a3e --- /dev/null +++ b/nats_streaming/clustering/supported-stores.md @@ -0,0 +1,9 @@ +# Supported Stores + +In order to run NATS Streaming Server in clustered mode, you need to specify a persistent store. At this time you have the choice between `FILE` and `SQL` + +The NATS Streaming stores server meta information, messages and subscriptions to the storage you configure using the `--store` option. + +However, in clustered mode, we use RAFT for leader election. The raft layer uses its own stores which are currently necessarily file based. The location of the RAFT stores defaults to the current directory under a sub-directory named after the cluster ID, or you can configure it using `--cluster_log_path`. + +This means that even if you select an SQL Store, there will still be a need for storing data on the file system. diff --git a/nats_streaming/fault-tolerance/active-server.md b/nats_streaming/fault-tolerance/active-server.md new file mode 100644 index 0000000..6ae7b8d --- /dev/null +++ b/nats_streaming/fault-tolerance/active-server.md @@ -0,0 +1,7 @@ +# Active Server + +There is a single Active server in the group. This server was the first to obtain the exclusive lock for storage. For the `FileStore` implementation, it means trying to get an advisory lock for a file located in the shared datastore. For the `SQLStore` implementation, a special table is used in which the owner of the lock updates a column. Other instances will steal the lock if the column is not updated for a certain amount of time. + +If the elected server fails to grab this lock because it is already locked, it will go back to standby. + +***Only the active server accesses the store and service all clients.*** diff --git a/nats_streaming/fault-tolerance/failover.md b/nats_streaming/fault-tolerance/failover.md new file mode 100644 index 0000000..c1dce65 --- /dev/null +++ b/nats_streaming/fault-tolerance/failover.md @@ -0,0 +1,9 @@ +# Failover + +When the active server fails, all standby servers will try to activate. The process consists of trying to get an exclusive lock on the storage. + +The first server that succeeds will become active and go through the process of recovering the store and service clients. It is as if a server in standalone mode was automatically restarted. + +All other servers that failed to get the store lock will go back to standby mode and stay in this mode until they stop receiving heartbeats from the current active server. + +It is possible that a standby trying to activate is not able to immediately acquire the store lock. When that happens, it goes back into standby mode, but if it fails to receive heartbeats from an active server, it will try again to acquire the store lock. The interval is random but as of now set to a bit more than a second. diff --git a/nats_streaming/fault-tolerance/ft.md b/nats_streaming/fault-tolerance/ft.md new file mode 100644 index 0000000..b3833ce --- /dev/null +++ b/nats_streaming/fault-tolerance/ft.md @@ -0,0 +1,15 @@ +# Fault Tolerance + +To minimize the single point of failure, NATS Streaming server can be run in Fault Tolerance mode. It works by having a group of servers with one acting as the active server (accessing the store) and handling all communication with clients, and all others acting as standby servers. + +It is important to note that is not possible to run Nats Stream as Fault Tolerance mode and Clustering mode at the same time. + +To start a server in Fault Tolerance (FT) mode, you specify an FT group name. + +Here is an example on how starting 2 servers in FT mode running on the same host and embedding the NATS servers: + +``` +nats-streaming-server -store file -dir datastore -ft_group "ft" -cluster nats://localhost:6222 -routes nats://localhost:6223 -p 4222 + +nats-streaming-server -store file -dir datastore -ft_group "ft" -cluster nats://localhost:6223 -routes nats://localhost:6222 -p 4223 +``` \ No newline at end of file diff --git a/nats_streaming/fault-tolerance/shared-state.md b/nats_streaming/fault-tolerance/shared-state.md new file mode 100644 index 0000000..3226b49 --- /dev/null +++ b/nats_streaming/fault-tolerance/shared-state.md @@ -0,0 +1,3 @@ +# Shared State + +Actual file replication to multiple disks is not handled by the Streaming server. This - if required - needs to be handled by the user. For the FileStore implementation that we currently provide, the data store needs to be mounted by all servers in the FT group (e.g. an NFS Mount (Gluster in Google Cloud or EFS in Amazon). diff --git a/nats_streaming/fault-tolerance/standby-server.md b/nats_streaming/fault-tolerance/standby-server.md new file mode 100644 index 0000000..da8fff6 --- /dev/null +++ b/nats_streaming/fault-tolerance/standby-server.md @@ -0,0 +1,3 @@ +# Standby servers + +There can be as many as you want standby servers on the same group. These servers do not access the store and do not receive any data from the streaming clients. They are just running waiting for the detection of the active server failure. diff --git a/nats_streaming/gettingstarted/configuring.md b/nats_streaming/gettingstarted/configuring.md new file mode 100644 index 0000000..9e3f666 --- /dev/null +++ b/nats_streaming/gettingstarted/configuring.md @@ -0,0 +1,3 @@ +## Configuration and Administration + +NATS Streaming provides a rich set of commands and parameters to configure all aspects of the server. Please refer to the [README](https://github.com/nats-io/nats-streaming-server#configuring) for further information on usage, configuration, and administration. \ No newline at end of file diff --git a/nats_streaming/gettingstarted/install.md b/nats_streaming/gettingstarted/install.md new file mode 100644 index 0000000..390b20b --- /dev/null +++ b/nats_streaming/gettingstarted/install.md @@ -0,0 +1,81 @@ +# NATS Streaming Server Installation + +NATS philosophy is simplicity. Installation is just decompressing a zip file and copying the binary to an appropriate directory; you can also use your favorite package manager. + +## Installing via a Package Manager + +On Mac OS: +``` +> brew install nats-streaming-server +``` + +Via Docker: +``` +> docker pull nats-streaming +``` + +## Installing a release build + +You can find the latest release of `nats-streaming-server` [here](https://github.com/nats-io/nats-streaming-server/releases/latest). + +Download the zip file matching your systems architecture, and unzip. For this example, assuming version 0.14.2 of the server, and a Linux AMD64: +``` +> curl -L https://github.com/nats-io/nats-streaming-server/releases/download/v0.14.2/nats-streaming-server-v0.14.2-linux-amd64.zip -o nats-streaming-server.zip + +> unzip nats-streaming-server.zip -d tmp +Archive: nats-streaming-server.zip + creating: tmp/nats-streaming-server-v0.14.2-linux-amd64/ + inflating: tmp/nats-streaming-server-v0.14.2-linux-amd64/README.md + inflating: tmp/nats-streaming-server-v0.14.2-linux-amd64/LICENSE + inflating: tmp/nats-streaming-server-v0.14.2-linux-amd64/nats-streaming-server + +> cp tmp/nats-streaming-server-v0.14.2-linux-amd64/nats-streaming-server /usr/local/bin +``` + +## Installing from the source + +If you have go installed, installing the binary is easy: + +``` +> go get github.com/nats-io/nats-streaming-server +``` + +This mechanism will install a build of master, which almost certainly will not be a released version. If you are a developer and want to play with the latest, this is the easiest way of obtaining it. + +You can run the test suite from the `nats-streaming-server` root directory: +``` +go test -v -p=1 ./... +``` + +Some of the store tests require a SQL server to be running. To skip those, use this command instead: +``` +go test -v -p=1 ./... -sql=false +``` + +## Testing Your Installation + +To test your installation (provided the executable is visible to your shell): + +``` +> nats-streaming-server +[58061] 2019/05/22 13:56:45.463562 [INF] STREAM: Starting nats-streaming-server[test-cluster] version 0.14.2 +[58061] 2019/05/22 13:56:45.463639 [INF] STREAM: ServerID: Avb51sMf9imRPVVwv6Ts0v +[58061] 2019/05/22 13:56:45.463657 [INF] STREAM: Go version: go1.11.10 +[58061] 2019/05/22 13:56:45.463659 [INF] STREAM: Git commit: [not set] +[58061] 2019/05/22 13:56:45.464086 [INF] Starting nats-server version 1.4.1 +[58061] 2019/05/22 13:56:45.464092 [INF] Git commit [not set] +[58061] 2019/05/22 13:56:45.464310 [INF] Listening for client connections on 0.0.0.0:4222 +[58061] 2019/05/22 13:56:45.464328 [INF] Server is ready +[58061] 2019/05/22 13:56:45.495045 [INF] STREAM: Recovering the state... +[58061] 2019/05/22 13:56:45.495055 [INF] STREAM: No recovered state +[58061] 2019/05/22 13:56:45.749604 [INF] STREAM: Message store is MEMORY +[58061] 2019/05/22 13:56:45.749658 [INF] STREAM: ---------- Store Limits ---------- +[58061] 2019/05/22 13:56:45.749664 [INF] STREAM: Channels: 100 * +[58061] 2019/05/22 13:56:45.749668 [INF] STREAM: --------- Channels Limits -------- +[58061] 2019/05/22 13:56:45.749671 [INF] STREAM: Subscriptions: 1000 * +[58061] 2019/05/22 13:56:45.749675 [INF] STREAM: Messages : 1000000 * +[58061] 2019/05/22 13:56:45.749678 [INF] STREAM: Bytes : 976.56 MB * +[58061] 2019/05/22 13:56:45.749682 [INF] STREAM: Age : unlimited * +[58061] 2019/05/22 13:56:45.749686 [INF] STREAM: Inactivity : unlimited * +[58061] 2019/05/22 13:56:45.749690 [INF] STREAM: ---------------------------------- +``` diff --git a/nats_streaming/gettingstarted/intro.md b/nats_streaming/gettingstarted/intro.md new file mode 100644 index 0000000..6be1634 --- /dev/null +++ b/nats_streaming/gettingstarted/intro.md @@ -0,0 +1,6 @@ +# Getting Started + +The best way to get the NATS Streaming Server is to use one of the pre-built release binaries which are available for OSX, Linux (x86-64/ARM), Windows. + +Of course you can build the latest version of the server from the master branch. The master branch will always build and pass tests, but may not work correctly in your environment. You will first need Go installed on your machine (version 1.11+ is required) to build the NATS Streaming server. + diff --git a/nats_streaming/gettingstarted/process-signaling.md b/nats_streaming/gettingstarted/process-signaling.md new file mode 100644 index 0000000..a3e88b8 --- /dev/null +++ b/nats_streaming/gettingstarted/process-signaling.md @@ -0,0 +1,27 @@ +# Process Signaling + +On Unix systems, the NATS Streaming Server responds to the following signals: + +| Signal | Result | +| --------------- | ------------------------------------- | +| SIGKILL | Kills the process immediately | +| SIGINT, SIGTERM | Stops the server gracefully | +| SIGUSR1 | Reopens the log file for log rotation | + +The `nats-streaming-server` binary can be used to send these signals to running NATS Streaming Servers using the `-sl` flag: + +```sh +# Reopen log file for log rotation +nats-streaming-server -sl reopen + +# Stop the server +nats-streaming-server -sl quit +``` + +If there are multiple `nats-streaming-server` processes running, specify a PID: + +```sh +nats-streaming-server -sl quit= +``` + +See the [Windows Service](#windows-service) section for information on signaling the NATS Streaming Server on Windows. diff --git a/nats_streaming/gettingstarted/run.md b/nats_streaming/gettingstarted/run.md new file mode 100644 index 0000000..c24918b --- /dev/null +++ b/nats_streaming/gettingstarted/run.md @@ -0,0 +1,96 @@ +# Getting Started with NATS Streaming + +This tutorial demonstrates NATS Streaming using example [Go NATS Streaming clients](https://github.com/nats-io/stan.go.git). + +## Prerequisites + +- [Set up your Git environment](https://help.github.com/articles/set-up-git/). +- [Set up your Go environment](https://golang.org/doc/install). + +## Setup + +Download and install the [NATS Streaming Server](https://github.com/nats-io/nats-streaming-server/releases). + +Clone the following repositories: + +- NATS Streaming Server: `git clone https://github.com/nats-io/nats-streaming-server.git` +- NATS Streaming Client: `git clone https://github.com/nats-io/stan.go.git` + +## Start the NATS Streaming Server + +Two options: + +Run the binary that you downloaded, for example: `$ ./nats-streaming-server` + +Or, run from source: + +```sh +> cd $GOPATH/src/github.com/nats-io/nats-streaming-server +> go run nats-streaming-server.go +``` + +You should see the following, indicating that the NATS Streaming Server is running: + +```sh +> go run nats-streaming-server.go +[59232] 2019/05/22 14:24:54.426344 [INF] STREAM: Starting nats-streaming-server[test-cluster] version 0.14.2 +[59232] 2019/05/22 14:24:54.426423 [INF] STREAM: ServerID: 3fpvAuXHo3C66Rkd4rmfFX +[59232] 2019/05/22 14:24:54.426440 [INF] STREAM: Go version: go1.11.10 +[59232] 2019/05/22 14:24:54.426442 [INF] STREAM: Git commit: [not set] +[59232] 2019/05/22 14:24:54.426932 [INF] Starting nats-server version 1.4.1 +[59232] 2019/05/22 14:24:54.426937 [INF] Git commit [not set] +[59232] 2019/05/22 14:24:54.427104 [INF] Listening for client connections on 0.0.0.0:4222 +[59232] 2019/05/22 14:24:54.427108 [INF] Server is ready +[59232] 2019/05/22 14:24:54.457604 [INF] STREAM: Recovering the state... +[59232] 2019/05/22 14:24:54.457614 [INF] STREAM: No recovered state +[59232] 2019/05/22 14:24:54.711407 [INF] STREAM: Message store is MEMORY +[59232] 2019/05/22 14:24:54.711465 [INF] STREAM: ---------- Store Limits ---------- +[59232] 2019/05/22 14:24:54.711471 [INF] STREAM: Channels: 100 * +[59232] 2019/05/22 14:24:54.711474 [INF] STREAM: --------- Channels Limits -------- +[59232] 2019/05/22 14:24:54.711478 [INF] STREAM: Subscriptions: 1000 * +[59232] 2019/05/22 14:24:54.711481 [INF] STREAM: Messages : 1000000 * +[59232] 2019/05/22 14:24:54.711485 [INF] STREAM: Bytes : 976.56 MB * +[59232] 2019/05/22 14:24:54.711488 [INF] STREAM: Age : unlimited * +[59232] 2019/05/22 14:24:54.711492 [INF] STREAM: Inactivity : unlimited * +[59232] 2019/05/22 14:24:54.711495 [INF] STREAM: ---------------------------------- +``` + +## Run the publisher client + +Publish several messages. For each publication you should get a result. + +```sh +> cd $GOPATH/src/github.com/nats-io/stan.go/examples/stan-pub +> go run main.go foo "msg one" +Published [foo] : 'msg one' +> go run main.go foo "msg two" +Published [foo] : 'msg two' +> go run main.go foo "msg three" +Published [foo] : 'msg three' +``` + +## Run the subscriber client + +Use the `--all` flag to receive all published messages. + +```sh +> cd $GOPATH/src/github.com/nats-io/stan.go/examples/stan-sub +> go run main.go --all -c test-cluster -id myID foo +Connected to nats://localhost:4222 clusterID: [test-cluster] clientID: [myID] +subscribing with DeliverAllAvailable +Listening on [foo], clientID=[myID], qgroup=[] durable=[] +[#1] Received on [foo]: 'sequence:1 subject:"foo" data:"msg one" timestamp:1465962202884478817 ' +[#2] Received on [foo]: 'sequence:2 subject:"foo" data:"msg two" timestamp:1465962208545003897 ' +[#3] Received on [foo]: 'sequence:3 subject:"foo" data:"msg three" timestamp:1465962215567601196 +``` + +## Explore other subscription options + +```sh + --seq Start at seqno + --all Deliver all available messages + --last Deliver starting with last published message + --since Deliver messages in last interval (e.g. 1s, 1hr, https://golang.org/pkg/time/#ParseDuration) + --durable Durable subscriber name + --unsubscribe Unsubscribe the durable on exit +``` diff --git a/nats_streaming/nats-streaming-tls.md b/nats_streaming/gettingstarted/tls.md similarity index 100% rename from nats_streaming/nats-streaming-tls.md rename to nats_streaming/gettingstarted/tls.md diff --git a/nats_streaming/gettingstarted/windows-service.md b/nats_streaming/gettingstarted/windows-service.md new file mode 100644 index 0000000..6a52b78 --- /dev/null +++ b/nats_streaming/gettingstarted/windows-service.md @@ -0,0 +1,28 @@ +# Windows Service + +The NATS Streaming Server supports running as a Windows service. There is currently no installer and instead users should use `sc.exe` to install the service: + +Here is how to create and start a NATS Streaming Server named `nats-streaming-server`. Note that the server flags should be passed in when creating the service. +```sh +sc.exe create nats-streaming-server binPath="\"\nats-streaming-server.exe\" [NATS Streaming flags]" +sc.exe start nats-streaming-server +``` + +You can create several instances, giving it a unique name. For instance, this is how you would create two services, named `nss1` and `nss2`, each one with its own set of parameters. +``` +sc.exe create nss1 binPath="\"c:\nats-io\nats-streaming\nats-streaming-server.exe\" --syslog --syslog_name=nss1 -p 4222" + +sc.exe create nss2 binPath="\"c:\nats-io\nats-streaming\nats-streaming-server.exe\" --syslog --syslog_name=nss2 -p 4223" +``` +By default, when no logfile is specified, the server will use the system log. The default event source name is `NATS-Streaming-Server`. However, you can specify the name you want, which is especially useful when installing more than one service as described above. + +Once the service is running, it can be controlled using `sc.exe` or `nats-streaming-server.exe -sl`: + +```batch +REM Stop the server +nats-streaming-server.exe -sl quit +``` +The above commands will default to controlling the service named `nats-streaming-server`. If the service has another name, it can be specified like this: +```batch +nats-streaming-server.exe -sl quit= +``` diff --git a/nats_streaming/nats-streaming-intro.md b/nats_streaming/intro.md similarity index 81% rename from nats_streaming/nats-streaming-intro.md rename to nats_streaming/intro.md index 1ce104c..9d1ea42 100644 --- a/nats_streaming/nats-streaming-intro.md +++ b/nats_streaming/intro.md @@ -32,15 +32,8 @@ digraph nats_streaming { In addition to the features of the core NATS platform, NATS Streaming provides the following: -- **Enhanced message protocol** - NATS Streaming implements its own enhanced message format using [Google Protocol Buffers] (https://developers.google.com/protocol-buffers/). These messages are transmitted as binary message payloads via core NATS platform, and thus require no changes to the basic NATS protocol. NATS Streaming messages contain the following fields: - - Sequence - a globally ordered sequence number for the subject's channel - - Subject - The NATS Streaming delivery subject - - Reply - The optional "reply-to" subject - - Data - The message payload - - Timestamp - the received timestamp, in nanoseconds. - - Redelivered - A flag signifying whether this message has been redelivered by the server - - CRC32 - An optional IEEE CRC32 -- **Message/event persistence** - NATS Streaming offers configurable message persistence either in-memory or via flat files. The storage subsystem uses a public interface that allows contributors to develop their own custom implementations. +- **Enhanced message protocol** - NATS Streaming implements its own enhanced message format using [Google Protocol Buffers](https://developers.google.com/protocol-buffers/). These messages are transmitted as binary message payloads via core NATS platform, and thus require no changes to the basic NATS protocol. +- **Message/event persistence** - NATS Streaming offers configurable message persistence: in-memory, flat files or database. The storage subsystem uses a public interface that allows contributors to develop their own custom implementations. - **At-least-once-delivery** - NATS Streaming offers message acknowledgements between publisher and server (for publish operations) and between subscriber and server (to confirm message delivery). Messages are persisted by the server in memory or secondary storage (or other external storage) and will be redelivered to eligible subscribing clients as needed. - **Publisher rate limiting** - NATS Streaming provides a connection option called `MaxPubAcksInFlight` that effectively limits the number of unacknowledged messages that a publisher may have in-flight at any given time. When this maximum is reached, further async publish calls will block until the number of unacknowledged messages falls below the specified limit. - **Rate matching/limiting per subscriber** - Subscriptions may specify a `MaxInFlight` option that designates the maximum number of outstanding acknowledgements (messages that have been delivered but not acknowledged) that NATS Streaming will allow for a given subscription. When this limit is reached, NATS Streaming will suspend delivery of messages to this subscription until the number of unacknowledged messages falls below the specified limit. @@ -52,10 +45,11 @@ In addition to the features of the core NATS platform, NATS Streaming provides t - A specific message sequence number - **Durable subscriptions** - Subscriptions may also specify a "durable name" which will survive client restarts. Durable subscriptions cause the server to track the last acknowledged message sequence number for a client and durable name. When the client restarts/resubscribes, and uses the same client ID and durable name, the server will resume delivery beginning with the earliest unacknowledged message for this durable subscription. + ## Installation -NATS provides a [server binary](nats-streaming-install.md) for Linux, Mac, and Windows. You can install the server from source on any platform you choose. +NATS provides a [server binary](gettingstarted/install.md) for Linux, Mac, and Windows. You can install the server from source on any platform you choose. ## Usage, Configuration and Administration -NATS Streaming provides a rich set of commands and parameters to configure all aspects of the server. Please refer to the [README](https://github.com/nats-io/nats-streaming-server/) for further info on usage, configuration, and administration. +NATS Streaming provides a rich set of commands and parameters to configure all aspects of the server. Please refer to the [README](https://github.com/nats-io/nats-streaming-server#configuring) for further information on usage, configuration, and administration. \ No newline at end of file diff --git a/nats_streaming/monitoring/enabling.md b/nats_streaming/monitoring/enabling.md new file mode 100644 index 0000000..675cc58 --- /dev/null +++ b/nats_streaming/monitoring/enabling.md @@ -0,0 +1,24 @@ +### Enabling + +To enable the monitoring server, start the NATS Streaming Server with the monitoring flag -m (or -ms) and specify the monitoring port. + +Monitoring options +``` +-m, --http_port PORT HTTP PORT for monitoring +-ms,--https_port PORT Use HTTPS PORT for monitoring (requires TLS cert and key) +``` +To enable monitoring via the configuration file, use `http: "host:port"` or `https: "host:port"` (there is no explicit configuration flag for the monitoring interface). + +For example, after running this: +``` +nats-streaming-server -m 8222 +``` +you should see that the NATS Streaming server starts with the HTTP monitoring port enabled: + +``` +(...) +[53359] 2017/12/18 17:44:31.594407 [INF] Starting http monitor on 0.0.0.0:8222 +[53359] 2017/12/18 17:44:31.594462 [INF] Listening for client connections on 0.0.0.0:4222 +(...) +``` +You can then point your browser (or curl) to [http://localhost:8222/streaming](http://localhost:8222/streaming) diff --git a/nats_streaming/monitoring/endpoints.md b/nats_streaming/monitoring/endpoints.md new file mode 100644 index 0000000..fdd042a --- /dev/null +++ b/nats_streaming/monitoring/endpoints.md @@ -0,0 +1,319 @@ +### Endpoints + +The following sections describe each supported monitoring endpoint: serverz, storez, clientsz, and channelsz. + +#### /serverz + +The endpoint [http://localhost:8222/streaming/serverz](http://localhost:8222/streaming/serverz) reports various general statistics. +``` +{ + "cluster_id": "test-cluster", + "server_id": "JEzjfVQS4JIEzM7lZmWHm9", + "version": "0.14.2", + "go": "go1.11.10", + "state": "STANDALONE", + "now": "2019-05-21T11:09:35.364637-06:00", + "start_time": "2019-05-21T11:09:24.204869-06:00", + "uptime": "11s", + "clients": 0, + "subscriptions": 0, + "channels": 0, + "total_msgs": 0, + "total_bytes": 0 +} +``` + +In clustering mode, there is an additional field that indicates the RAFT role of the given node. Here is an example: +``` +{ + "cluster_id": "test-cluster", + "server_id": "t9W9zbOIIi5Y9Guppxl0lF", + "version": "0.14.2", + "go": "go1.11.10", + "state": "CLUSTERED", + "role": "Follower", + "now": "2019-05-21T11:10:15.765261-06:00", + "start_time": "2019-05-21T11:10:12.21284-06:00", + "uptime": "3s", + "clients": 0, + "subscriptions": 0, + "channels": 0, + "total_msgs": 0, + "total_bytes": 0 +} +``` +The possible values are: `Leader`, `Follower` or `Candidate`. + +#### /storez + +The endpoint [http://localhost:8222/streaming/storez](http://localhost:8222/streaming/storez) reports information about the store. +``` +{ + "cluster_id": "test-cluster", + "server_id": "8AjZq57k4JY7cfKEvuZ8iF", + "now": "2019-04-16T09:57:32.857406-06:00", + "type": "MEMORY", + "limits": { + "max_channels": 100, + "max_msgs": 1000000, + "max_bytes": 1024000000, + "max_age": 0, + "max_subscriptions": 1000, + "max_inactivity": 0 + }, + "total_msgs": 130691, + "total_bytes": 19587140 +} +``` + +#### /clientsz + +The endpoint [http://localhost:8222/streaming/clientsz](http://localhost:8222/streaming/clientsz) reports more detailed information about the connected clients. + +It uses a paging mechanism which defaults to 1024 clients. + +You can control these via URL arguments (limit and offset). For example: [http://localhost:8222/streaming/clientsz?limit=1&offset=1](http://localhost:8222/streaming/clientsz?limit=1&offset=1). +``` +{ + "cluster_id": "test-cluster", + "server_id": "J3Odi0wXYKWKFWz5D5uhH9", + "now": "2017-06-07T14:47:44.495254605+02:00", + "offset": 1, + "limit": 1, + "count": 1, + "total": 11, + "clients": [ + { + "id": "benchmark-sub-0", + "hb_inbox": "_INBOX.jAHSY3hcL5EGFQGYmfayQK" + } + ] +} +``` +You can also report detailed subscription information on a per client basis using `subs=1`. For example: [http://localhost:8222/streaming/clientsz?limit=1&offset=1&subs=1](http://localhost:8222/streaming/clientsz?limit=1&offset=1&subs=1). +``` +{ + "cluster_id": "test-cluster", + "server_id": "J3Odi0wXYKWKFWz5D5uhH9", + "now": "2017-06-07T14:48:06.157468748+02:00", + "offset": 1, + "limit": 1, + "count": 1, + "total": 11, + "clients": [ + { + "id": "benchmark-sub-0", + "hb_inbox": "_INBOX.jAHSY3hcL5EGFQGYmfayQK", + "subscriptions": { + "foo": [ + { + "client_id": "benchmark-sub-0", + "inbox": "_INBOX.jAHSY3hcL5EGFQGYmfayvC", + "ack_inbox": "_INBOX.J3Odi0wXYKWKFWz5D5uhem", + "is_durable": false, + "is_offline": false, + "max_inflight": 1024, + "ack_wait": 30, + "last_sent": 505597, + "pending_count": 0, + "is_stalled": false + } + ] + } + } + ] +} +``` +You can select a specific client based on its client ID with `client=`, and get also get detailed statistics with `subs=1`. For example: [http://localhost:8222/streaming/clientsz?client=me&subs=1](http://localhost:8222/streaming/clientsz?client=me&subs=1). +``` +{ + "id": "me", + "hb_inbox": "_INBOX.HG0uDuNtAPxJQ1lVjIC2sr", + "subscriptions": { + "foo": [ + { + "client_id": "me", + "inbox": "_INBOX.HG0uDuNtAPxJQ1lVjIC389", + "ack_inbox": "_INBOX.Q9iH2gsDPN57ZEvqswiYSL", + "is_durable": false, + "is_offline": false, + "max_inflight": 1024, + "ack_wait": 30, + "last_sent": 0, + "pending_count": 0, + "is_stalled": false + } + ] + } +} +``` + +#### /channelsz + +The endpoint [http://localhost:8222/streaming/channelsz](http://localhost:8222/streaming/channelsz) reports the list of channels. +``` +{ + "cluster_id": "test-cluster", + "server_id": "J3Odi0wXYKWKFWz5D5uhH9", + "now": "2017-06-07T14:48:41.680592041+02:00", + "offset": 0, + "limit": 1024, + "count": 2, + "total": 2, + "names": [ + "bar" + "foo" + ] +} +``` +It uses a paging mechanism which defaults to 1024 channels. + +You can control these via URL arguments (limit and offset). For example: [http://localhost:8222/streaming/channelsz?limit=1&offset=1](http://localhost:8222/streaming/channelsz?limit=1&offset=1). +``` +{ + "cluster_id": "test-cluster", + "server_id": "J3Odi0wXYKWKFWz5D5uhH9", + "now": "2017-06-07T14:48:41.680592041+02:00", + "offset": 1, + "limit": 1, + "count": 1, + "total": 2, + "names": [ + "foo" + ] +} +``` +You can also get the list of subscriptions with `subs=1`. For example: [http://localhost:8222/streaming/channelsz?limit=1&offset=0&subs=1](http://localhost:8222/streaming/channelsz?limit=1&offset=0&subs=1). +``` +{ + "cluster_id": "test-cluster", + "server_id": "J3Odi0wXYKWKFWz5D5uhH9", + "now": "2017-06-07T15:01:02.166116959+02:00", + "offset": 0, + "limit": 1, + "count": 1, + "total": 2, + "channels": [ + { + "name": "bar", + "msgs": 0, + "bytes": 0, + "first_seq": 0, + "last_seq": 0, + "subscriptions": [ + { + "client_id": "me", + "inbox": "_INBOX.S7kTJjOcToXiJAzGWgINit", + "ack_inbox": "_INBOX.Y04G5pZxlint3yPXrSTjTV", + "is_durable": false, + "is_offline": false, + "max_inflight": 1024, + "ack_wait": 30, + "last_sent": 0, + "pending_count": 0, + "is_stalled": false + } + ] + } + ] +} +``` +You can select a specific channel based on its name with `channel=name`. For example: [http://localhost:8222/streaming/channelsz?channel=foo](http://localhost:8222/streaming/channelsz?channel=foo). +``` +{ + "name": "foo", + "msgs": 649234, + "bytes": 97368590, + "first_seq": 1, + "last_seq": 649234 +} +``` +And again, you can get detailed subscriptions with `subs=1`. For example: [http://localhost:8222/streaming/channelsz?channel=foo&subs=1](http://localhost:8222/streaming/channelsz?channel=foo&subs=1). +``` +{ + "name": "foo", + "msgs": 704770, + "bytes": 105698990, + "first_seq": 1, + "last_seq": 704770, + "subscriptions": [ + { + "client_id": "me", + "inbox": "_INBOX.jAHSY3hcL5EGFQGYmfayvC", + "ack_inbox": "_INBOX.J3Odi0wXYKWKFWz5D5uhem", + "is_durable": false, + "is_offline": false, + "max_inflight": 1024, + "ack_wait": 30, + "last_sent": 704770, + "pending_count": 0, + "is_stalled": false + }, + { + "client_id": "me2", + "inbox": "_INBOX.jAHSY3hcL5EGFQGYmfaywG", + "ack_inbox": "_INBOX.J3Odi0wXYKWKFWz5D5uhjV", + "is_durable": false, + "is_offline": false, + "max_inflight": 1024, + "ack_wait": 30, + "last_sent": 704770, + "pending_count": 0, + "is_stalled": false + }, + (...) + ] +} +``` + +For durables that are currently running, the `is_offline` field is set to `false`. Here is an example: +``` +{ + "name": "foo", + "msgs": 0, + "bytes": 0, + "first_seq": 0, + "last_seq": 0, + "subscriptions": [ + { + "client_id": "me", + "inbox": "_INBOX.P23kNGFnwC7KRg3jIMB3IL", + "ack_inbox": "_STAN.ack.pLyMpEyg7dgGZBS7jGXC02.foo.pLyMpEyg7dgGZBS7jGXCaw", + "durable_name": "dur", + "is_durable": true, + "is_offline": false, + "max_inflight": 1024, + "ack_wait": 30, + "last_sent": 0, + "pending_count": 0, + "is_stalled": false + } + ] +} +``` + +When that same durable goes offline, `is_offline` is be set to `true`. Although the client is possibly no longer connected (and would not appear in the `clientsz` endpoint), the `client_id` field is still displayed here. +``` +{ + "name": "foo", + "msgs": 0, + "bytes": 0, + "first_seq": 0, + "last_seq": 0, + "subscriptions": [ + { + "client_id": "me", + "inbox": "_INBOX.P23kNGFnwC7KRg3jIMB3IL", + "ack_inbox": "_STAN.ack.pLyMpEyg7dgGZBS7jGXC02.foo.pLyMpEyg7dgGZBS7jGXCaw", + "durable_name": "dur", + "is_durable": true, + "is_offline": true, + "max_inflight": 1024, + "ack_wait": 30, + "last_sent": 0, + "pending_count": 0, + "is_stalled": false + } + ] +} +``` diff --git a/nats_streaming/monitoring/monitoring.md b/nats_streaming/monitoring/monitoring.md new file mode 100644 index 0000000..4ef4fff --- /dev/null +++ b/nats_streaming/monitoring/monitoring.md @@ -0,0 +1,3 @@ +## Monitoring + +To monitor the NATS streaming system, a lightweight HTTP server is used on a dedicated monitoring port. The monitoring server provides several endpoints, all returning a JSON object. diff --git a/nats_streaming/nats-streaming-install.md b/nats_streaming/nats-streaming-install.md deleted file mode 100644 index 8717bef..0000000 --- a/nats_streaming/nats-streaming-install.md +++ /dev/null @@ -1,102 +0,0 @@ -# Install and Run NATS Streaming Server - -In this tutorial you install and run the NATS Streaming server (`nats-streaming-server`). -You can follow this same procedure anytime you want to run the NATS Streaming server. - -### Install the NATS Streaming server - -There are numerous ways to install the NATS Streaming server. - -#### GitHub releases - -The latest official release binaries are always available on the [GitHub releases page](https://github.com/nats-io/nats-streaming-server/releases). -The following platforms are available: - -- Linux (x86, x86_64, ARM) -- Windows (x86, x86_64) -- macOS - -The following methods may also be used. _Please note that these methods may not install the latest released version_: - -#### Go - -Make sure [your Go environment is set up](https://golang.org/doc/install) - -```sh -% go get github.com/nats-io/nats-streaming-server -``` - -Note that this method may not install the latest released version. - -#### Docker Hub - -The latest [official Docker image](https://hub.docker.com/_/nats-streaming/) is always available on Docker Hub. - -#### Windows - -On Windows, the NATS Streaming server can also be installed via [Chocolatey](https://chocolatey.org/packages/nats-streaming-server): - -```sh -% choco install nats-streaming-server -``` - -#### macOS - -On macOS, the NATS Streaming server can be installed via [Homebrew](http://brewformulas.org/NatsStreamingServer): - -```sh -% brew install nats-streaming-server -``` - -### Start the NATS Streaming server - -You can invoke the NATS Streaming server binary, with no options and no configuration file, to start a server with acceptable standalone defaults (no authentication, no clustering). - -```sh -% nats-streaming-server -``` - -When the server starts successfully, you will see that the NATS Streaming server listens for client connections on TCP Port 4222: - -```sh -[18085] 2016/10/31 13:11:44.059012 [INF] Starting nats-streaming-server[test-cluster] version 0.3.1 -[18085] 2016/10/31 13:11:44.059830 [INF] Starting nats-server version 0.9.4 -[18085] 2016/10/31 13:11:44.061544 [INF] Listening for client connections on 0.0.0.0:4222 -[18085] 2016/10/31 13:11:44.061966 [INF] Server is ready -[18085] 2016/10/31 13:11:44.396819 [INF] STAN: Message store is MEMORY -[18085] 2016/10/31 13:11:44.396832 [INF] STAN: --------- Store Limits --------- -[18085] 2016/10/31 13:11:44.396837 [INF] STAN: Channels: 100 * -[18085] 2016/10/31 13:11:44.396839 [INF] STAN: -------- channels limits ------- -[18085] 2016/10/31 13:11:44.396842 [INF] STAN: Subscriptions: 1000 * -[18085] 2016/10/31 13:11:44.396844 [INF] STAN: Messages : 1000000 * -[18085] 2016/10/31 13:11:44.396855 [INF] STAN: Bytes : 976.56 MB * -[18085] 2016/10/31 13:11:44.396858 [INF] STAN: Age : unlimited * -[18085] 2016/10/31 13:11:44.396859 [INF] STAN: -------------------------------- -``` - -### Start the NATS Streaming Server with NATS monitoring enabled (optional) - -The NATS Streaming server exposes the monitoring interface of its embedded NATS Server (`nats-server`) on port 8222. - -```sh -% nats-streaming-server -m 8222 -``` - -If you run the NATS Streaming server with monitoring enabled, you see the following messages: - -```sh -[18122] 2016/10/31 13:13:10.048663 [INF] Starting nats-streaming-server[test-cluster] version 0.3.1 -[18122] 2016/10/31 13:13:10.048843 [INF] Starting nats-server version 0.9.4 -[18122] 2016/10/31 13:13:10.048890 [INF] Starting http monitor on 0.0.0.0:8222 -[18122] 2016/10/31 13:13:10.048968 [INF] Listening for client connections on 0.0.0.0:4222 -[18122] 2016/10/31 13:13:10.048992 [INF] Server is ready -[18122] 2016/10/31 13:13:10.388282 [INF] STAN: Message store is MEMORY -[18122] 2016/10/31 13:13:10.388301 [INF] STAN: --------- Store Limits --------- -[18122] 2016/10/31 13:13:10.388309 [INF] STAN: Channels: 100 * -[18122] 2016/10/31 13:13:10.388312 [INF] STAN: -------- channels limits ------- -[18122] 2016/10/31 13:13:10.388316 [INF] STAN: Subscriptions: 1000 * -[18122] 2016/10/31 13:13:10.388319 [INF] STAN: Messages : 1000000 * -[18122] 2016/10/31 13:13:10.388333 [INF] STAN: Bytes : 976.56 MB * -[18122] 2016/10/31 13:13:10.388338 [INF] STAN: Age : unlimited * -[18122] 2016/10/31 13:13:10.388341 [INF] STAN: -------------------------------- -``` diff --git a/nats_streaming/nats-streaming-protocol.md b/nats_streaming/nats-streaming-protocol.md deleted file mode 100644 index 3533a23..0000000 --- a/nats_streaming/nats-streaming-protocol.md +++ /dev/null @@ -1,190 +0,0 @@ -# NATS Streaming Protocol - - -The NATS streaming protocol sits atop the core [NATS protocol](/documentation/internals/nats-protocol) and uses [Google's Protocol Buffers](https://developers.google.com/protocol-buffers/). Protocol buffer messages are marshaled into bytes and published as NATS messages on specific subjects described below. In communicating with the NATS Streaming Server, the NATS [request/reply](/documentation/writing_applications/concepts) pattern is used for all protocol messages that have a corresponding reply. - -## NATS streaming protocol conventions - -**Subject names**: Subject names, including reply subject (INBOX) names, are case-sensitive and must be non-empty alphanumeric strings with no embedded whitespace, and optionally token-delimited using the dot character (`.`), e.g.: - -`FOO`, `BAR`, `foo.bar`, `foo.BAR`, `FOO.BAR` and `FOO.BAR.BAZ` are all valid subject names - -`FOO. BAR`, `foo. .bar` and`foo..bar` are *not- valid subject names - -**Wildcards**: NATS streaming does **not*- support wildcards in subject subscriptions - -**Protocol definition**: The fields of NATS streaming protocol messages are defined in the go-nats-streaming [protocol file](https://github.com/nats-io/go-nats-streaming/blob/master/pb/protocol.proto). - -## NATS streaming protocol messages - -The following table briefly describes the NATS streaming protocol messages. - -Click the name to see more detailed information, including usage: - -| Message Name | Sent By | Description -| --------------------------------- |:--------|:-------------------------------------------- -| [`ConnectRequest`](#CONNREQ) | Client | Request to connect to the NATS Streaming Server -| [`ConnectResponse`](#CONNRESP) | Server | Result of a connection request -| [`SubscriptionRequest`](#SUBREQ) | Client | Request sent to subscribe and retrieve data -| [`SubscriptionResponse`](#SUBRESP)| Server | Result of a subscription request -| [`UnsubscribeRequest`](#UNSUBREQ) | Client | Unsubscribe from a subject -| [`PubMsg`](#PUBMSG) | Client | Publish a message to a subject, with optional reply subject -| [`PubAck`](#PUBACK) | Server | An acknowledgement that a published message has been processed on the server -| [`MsgProto`](#MSGPROTO) | Server | A message from the NATS Streaming Server to a subscribing client -| [`Ack`](#ACK) | Client | Acknowledges that a message has been received -| [`CloseRequest`](#CLOSEREQ) | Client | Request sent to close the connection to the NATS Streaming Server -| [`CloseResp`](#CLOSERESP) | Server | Result of the close request - -The following sections explain each protocol message. - -## ConnectRequest - -#### Description - -A connection request is sent when a streaming client connects to the NATS Streaming Server. The connection request contains a unique identifier representing the client, and an inbox subject the client will listen on for incoming heartbeats. The identifier **must*- be unique; a connection attempt with an identifier currently in use will fail. The inbox subject is the subject where the client receives incoming heartbeats, and responds by publishing an empty NATS message to the reply subject, indicating it is alive. The NATS Streaming Server will return a [ConnectResponse](#CONNRESP) message to the reply subject specified in the NATS request message. - -This request is published to a subject comprised of the `.cluster-id`, for example, if a NATS Streaming Server was started with a cluster-id of `mycluster`, and the default prefix was used, the client publishes to `_STAN.discover.mycluster` - -#### Message Structure - -- `clientID`: A unique identifier for a client -- `heartbeatInbox`: An inbox to which the NATS Streaming Server will send heartbeats for the client to process - -## ConnectResponse - -#### Description - -After a `ConnectRequest` is published, the NATS Streaming Server responds with this message on the reply subject of the underlying NATS request. The NATS Streaming Server requires the client to make requests and publish messages on certain subjects (described above), and when a connection is successful, the client saves the information returned to be used in sending other NATS streaming protocol messages. In the event the connection was not successful, an error is returned in the `error` field. - -#### Message Structure - -- `pubPrefix`: Prefix to use when publishing -- `subRequests`: Subject used for subscription requests -- `unsubRequests`: Subject used for unsubscribe requests -- `closeRequests`: Subject for closing a connection -- `error`: An error string, which will be empty/omitted upon success -- `publicKey`: Reserved for future use - -## SubscriptionRequest - -#### Description - -A `SubscriptionRequest` is published on the subject returned in the `subRequests` field of a [`ConnectResponse`](#CONNRESP), and creates a subscription to a subject on the NATS Streaming Server. This will return a [SubscriptionResponse](#SUBRESP) message to the reply subject specified in the NATS protocol request message. - -#### Message Structure - -- `clientID`: Client ID originally provided in the [ConnectRequest](#CONNREQ) -- `subject`: Formal subject to subscribe to, e.g. foo.bar -- `qGroup`: Optional queue group -- `inbox`: Inbox subject to deliver messages on -- `maxInFlight`: Maximum inflight messages without an acknowledgement allowed -- `ackWaitInSecs`: Timeout for receiving an acknowledgement from the client -- `durableName`: Optional durable name which survives client restarts -- `startPosition`: An enumerated type specifying the point in history to start replaying data -- `startSequence`: Optional start sequence number -- `startTimeDelta`: Optional start time - -#### StartPosition enumeration - -- `NewOnly`: Send only new messages -- `LastReceived`: Send only the last received message -- `TimeDeltaStart`: Send messages from duration specified in the `startTimeDelta` field. -- `SequenceStart`: Send messages starting from the sequence in the `startSequence` field. -- `First`: Send all available messages - -## SubscriptionResponse - -#### Description - -The `SubscriptionResponse` message is the response from the `SubscriptionRequest`. After a client has processed an incoming [MsgProto](#MSGPROTO) message, it must send an acknowledgement to the `ackInbox` subject provided here. - -#### Message Structure - -- `ackInbox`: subject the client sends message acknowledgements to the NATS Streaming Server -- `error`: error string, empty/omitted if no error - -## UnsubscribeRequest - -#### Description - -The `UnsubscribeRequest` unsubcribes the connection from the specified subject. The inbox specified is the `inbox` returned from the NATS Streaming Server in the `SubscriptionResponse`. - -#### Message Structure - -- `clientID`: Client ID originally provided in the [ConnectRequest](#CONNREQ) -- `subject`: Subject for the subscription -- `inbox`: Inbox subject to identify subscription -- `durableName`: Optional durable name which survives client restarts - -## PubMsg - -#### Description - -The `PubMsg` protocol message is published from a client to the NATS Streaming Server. The GUID must be unique, and is returned in the [PubAck](#PUBACK) message to correlate the success or failure of storing this particular message. - -#### Message Structure - -- `clientID`: Client ID originally provided in the [ConnectRequest](#CONNREQ) -- `guid`: a guid generated for this particular message -- `subject`: subject -- `reply`: optional reply subject -- `data`: payload -- `sha256`: optional sha256 of payload data - -## PubAck - -#### Description - -The `PubAck` message is an acknowledgement from the NATS Streaming Server that a message has been processed. The message arrives on the subject specified on the reply subject of the NATS message the `PubMsg` was published on. The GUID is the same GUID used in the `PubMsg` being acknowledged. If an error string is present, the message was not persisted by the NATS Streaming Server and no guarantees regarding persistence are honored. `PubAck` messages may be handled asynchronously from their corresponding `PubMsg` in the client. - -#### Message Structure - -- `guid`: GUID of the message being acknowledged by the NATS Streaming Server -- `error`: An error string, empty/omitted if no error - -## MsgProto - -#### Description - -The `MsgProto` message is received by client from the NATS Streaming Server, containing the payload of messages sent by a publisher. A `MsgProto` message that is not acknowledged with an [Ack](#ACK) message within the duration specified by the `ackWaitInSecs` field of the subscription request will be redelivered. - -#### Message Structure - -- `sequence`: Globally ordered sequence number for the subject's channel -- `subject`: Subject -- `reply`: Optional reply -- `data`: Payload -- `timestamp`: Time the message was stored in the server. -- `redelivered`: Flag specifying if the message is being redelivered -- `CRC32`: Optional IEEE CRC32 - -## Ack - -#### Description - -An `Ack` message is an acknowledgement from the client that a [MsgProto](#MSGPROTO) message has been considered received. It is published to the `ackInbox` field of the [`SubscriptionResponse`](#SUBRESP). - -#### Message Structure - -- `subject`: Subject of the message being acknowledged -- `sequence`: Sequence of the message being acknowledged - -## CloseRequest - -#### Description - -A `CloseRequest` message is published on the `closeRequests` subject from the [`ConnectResponse`](#CONNRESP), and notifies the NATS Streaming Server that the client connection is closing, allowing the server to free up resources. This message should **always*- be sent when a client is finished using a connection. - -#### Message Structure - -- `clientID`: Client ID originally provided in the [ConnectRequest](#CONNREQ) - -## CloseResponse - -#### Description - -The `CloseResponse` is sent by the NATS Streaming Server on the reply subject of the `CloseRequest` NATS message. This response contains any error that may have occurred with the corresponding close call. - -#### Message Structure - -- `error`: error string, empty/omitted if no error diff --git a/nats_streaming/nats-streaming-quickstart.md b/nats_streaming/nats-streaming-quickstart.md deleted file mode 100644 index 5812482..0000000 --- a/nats_streaming/nats-streaming-quickstart.md +++ /dev/null @@ -1,82 +0,0 @@ -# Getting Started with NATS Streaming - -This tutorial demonstrates NATS Streaming using example [Go NATS Streaming clients](https://github.com/nats-io/go-nats-streaming.git). - -## Prerequisites - -- [Set up your Git environment](https://help.github.com/articles/set-up-git/). -- [Set up your Go environment](https://golang.org/doc/install). - -## Setup - -Download and install the [NATS Streaming Server](https://github.com/nats-io/nats-streaming-server/releases). - -Clone the following repositories: - -- NATS Streaming Server: `git clone https://github.com/nats-io/nats-streaming-server.git` -- NATS Streaming Client: `git clone https://github.com/nats-io/go-nats-streaming.git` - -## Start the NATS Streaming Server - -Two options: - -Run the binary that you downloaded, for example: `$ ./nats-streaming-server` - -Or, run from source: - -```sh -% cd $GOPATH/src/github.com/nats-io/nats-streaming-server -% go run nats-streaming-server.go -``` - -You should see the following, indicating that the NATS Streaming Server is running: - -```sh -% go run nats-streaming-server.go -[89999] 2016/06/25 08:54:35.399071 [INF] Starting nats-streaming-server[test-cluster] version 0.1.0 -[89999] 2016/06/25 08:54:35.399315 [INF] Starting nats-server version 0.9.0.beta -[89999] 2016/06/25 08:54:35.399326 [INF] Listening for client connections on localhost:4222 -[89999] 2016/06/25 08:54:35.400721 [INF] Server is ready -[89999] 2016/06/25 08:54:35.737589 [INF] STAN: Message store is MEMORY -[89999] 2016/06/25 08:54:35.737610 [INF] STAN: Maximum of 1000000 will be stored -``` - -## Run the publisher client - -Publish several messages. For each publication you should get a result. - -```sh -% cd $GOPATH/src/github.com/nats-io/go-nats-streaming/examples/stan-pub -% go run main.go foo "msg one" -Published [foo] : 'msg one' -% go run main.go foo "msg two" -Published [foo] : 'msg two' -% go run main.go foo "msg three" -Published [foo] : 'msg three' -``` - -## Run the subscriber client - -Use the `--all` flag to receive all published messages. - -```sh -% cd $GOPATH/src/github.com/nats-io/go-nats-streaming/examples/stan-sub -% go run main.go --all -c test-cluster -id myID foo -Connected to nats://localhost:4222 clusterID: [test-cluster] clientID: [myID] -subscribing with DeliverAllAvailable -Listening on [foo], clientID=[myID], qgroup=[] durable=[] -[#1] Received on [foo]: 'sequence:1 subject:"foo" data:"msg one" timestamp:1465962202884478817 ' -[#2] Received on [foo]: 'sequence:2 subject:"foo" data:"msg two" timestamp:1465962208545003897 ' -[#3] Received on [foo]: 'sequence:3 subject:"foo" data:"msg three" timestamp:1465962215567601196 -``` - -## Explore other subscription options - -```sh - --seq Start at seqno - --all Deliver all available messages - --last Deliver starting with last published message - --since Deliver messages in last interval (e.g. 1s, 1hr, https://golang.org/pkg/time/#ParseDuration) - --durable Durable subscriber name - --unsubscribe Unsubscribe the durable on exit -``` diff --git a/nats_streaming/partitioning.md b/nats_streaming/partitioning.md new file mode 100644 index 0000000..1e6ab4c --- /dev/null +++ b/nats_streaming/partitioning.md @@ -0,0 +1,119 @@ +# Partitioning + +***Note, this feature is incompatible with Clustering mode. Trying to start a server with Partitioning and Clustering enabled will result in a startup error.*** + +It is possible to limit the list of channels a server can handle. This can be used to: + +* Prevent creation of unwanted channels +* Share the load between several servers running with the same cluster ID + +In order to do so, you need to enable the `partitioning` parameter in the configuration file, and also specify the list of allowed channels in the `channels` section of the `store_limits` configuration. + +Channels don't need to override any limit, but they need to be specified for the server to service only these channels. + +Here is an example: + +``` +partitioning: true +store_limits: { + channels: { + "foo": {} + "bar": {} + # Use of wildcards in configuration is allowed. However, applications cannot + # publish to, or subscribe to, wildcard channels. + "baz.*": {} + } +} +``` + +When partitioning is enabled, multiple servers with the same cluster ID can coexist on the same NATS network, each server handling its own set of channels. ***Note however that in this mode, state is not replicated as it is in Clustering mode. The only communication between servers is to report if a given channel is handled in more than one serve.r*** + +## Wildcards + +NATS Streaming does not support sending or subscribing to wildcard channels (such as `foo.*`). + +However, it is possible to use wildcards to define the partition that a server can handle. For instance, with the following configuration: +``` +partitioning: true +store_limits: { + channels: { + "foo.*": {} + "bar.>": {} + } +} +``` +The streaming server would accept subscriptions or published messages to channels such as: +1. `foo.bar` +2. `bar.baz` +3. `bar.baz.bat` +4. ... + +But would ignore messages or subscriptions on: + +1. `foo` +2. `foo.bar.baz` +3. `bar` +4. `some.other.channel` +5. ... + +## A given channel must be defined in a single server + +When a server starts, it sends its list of channels to all other servers on the same cluster in an attempt to detect duplicate channels. When a server receives this list and finds that it has a channel in common, it will return an error to the emitting server, which will then fail to start. + +However, on startup, it is possible that the underlying NATS cluster is not fully formed. The server would not get any response from the rest of the cluster and therefore start successfully and service clients. Anytime a Streaming server detects that a NATS server was added to the NATS cluster, it will resend its list of channels. It means that currently running servers may suddenly fail with a message regarding duplicate channels. Having the same channel on different servers means that a subscription would be created on all servers handling the channel, but only one server will receive and process message acknowledgements. Other servers would then redeliver messages (since they would not get the acknowledgements), which would cause duplicates. + +***In order to avoid issues with channels existing on several servers, it is ultimately the responsibility of the administrator to ensure that channels are unique.*** + +## Fault Tolerance and Partitioning + +You can easily combine the Fault Tolerance and Partitioning feature. + +To illustrate, suppose that we want two partitions, one for `foo.>` and one for `bar.>`. + +The configuration for the first server `foo.conf` would look like: +``` +partitioning: true +store_limits: { + channels: { + foo.>: {} + } +} +``` + +The second configuration `bar.conf` would be: +``` +partitioning: true +store_limits: { + channels: { + bar.>: {} + } +} +``` + +If you remember, Fault Tolerance is configured by specifying a name (`ft_group_name`). Suppose there is an NFS mount called `/nss/datastore` on both `host1` and `host2`. + +Starting an FT pair for the partition `foo` could look like this: +``` +host1$ nats-streaming-server -store file -dir /nss/datastore/foodata -sc foo.conf -ft_group_name foo -cluster nats://host1:6222 -routes nats://host2:6222,nats://host2:6223 + +host2$ nats-streaming-server -store file -dir /nss/datastore/foodata -sc foo.conf -ft_group_name foo -cluster nats://host2:6222 -routes nats://host1:6222,nats://host1:6223 +``` +Notice that each server on each note points to each other (the `-routes` parameter). The reason why we also point to `6223` will be explained later. They both listen for routes connections on their host's `6222` port. + +We now start the FT pair for `bar`. Since we are running from the same machines (we don't have to), we need to use a different port: +``` +host1$ nats-streaming-server -store file -dir /nss/datastore/bardata -sc bar.conf -ft_group_name bar -p 4223 -cluster nats://host1:6223 -routes nats://host2:6222,nats://host2:6223 + +host2$ nats-streaming-server -store file -dir /nss/datastore/bardata -sc bar.conf -ft_group_name bar -p 4223 -cluster nats://host2:6223 -routes nats://host1:6222,nats://host1:6223 +``` +You will notice that the `-routes` parameter points to both `6222` and `6223`, this is so that both partitions belong to the same cluster and be view as "one" by a Streaming application connecting to this cluster. Effectively, we have created a full mesh of 4 NATS servers that can all communicate with each other. Two of these servers are backups for servers running on the same FT group. + +## Applications behavior + +When an application connects, it specifies a cluster ID. If several servers are running with that same cluster ID, the application will be able to publish/subscribe to any channel handled by the cluster (as long as those servers are all connected to the NATS network). + +A published message will be received by only the server that has that channel defined. If no server is handling this channel, no specific error is returned, instead the publish call will timeout. Same goes for message acknowledgements. Only the server handling the subscription on this channel should receive those. + +However, other client requests (such as connection and subscription requests) are received by all servers. For connections, all servers handle them and the client library will receive a response from all servers in the cluster, but use the first one that it received. + +For subscriptions, a server receiving the request for a channel that it does not handle will simply ignore the request. Again, if no server handle this channel, the client's subscription request will simply time out. \ No newline at end of file diff --git a/nats_streaming/relation-to-nats.md b/nats_streaming/relation-to-nats.md new file mode 100644 index 0000000..cce74fa --- /dev/null +++ b/nats_streaming/relation-to-nats.md @@ -0,0 +1,12 @@ +# Relation to NATS + +NATS Streaming Server by default embeds a [NATS](https://github.com/nats-io/nats-server) server. That is, the Streaming server is not a server per-se, but instead, a client to a NATS Server.
+It means that Streaming clients are not directly connected to the streaming server, but instead communicate with the streaming server *through* NATS Server. + +This detail is important when it comes to Streaming clients connections to the Streaming server. Indeed, since there is no direct connection, the server knows if a client is connected based on heartbeats. + +***It is therefore strongly recommended for clients to close their connection when the application exit, otherwise the server will consider these clients connected (sending data, etc...) until it detects missing heartbeats.*** + +The streaming server creates internal subscriptions on specific subjects to communicate with its clients and/or other servers. + +Note that NATS clients and NATS Streaming clients cannot exchange data between each other. That is, if a streaming client publishes on `foo`, a NATS client subscribing on that same subject will not receive the messages. Streaming messages are NATS messages made of a protobuf. The streaming server is expected to send ACKs back to producers and receive ACKs from consumers. If messages were freely exchanged with the NATS clients, this would cause problems. diff --git a/nats_streaming/store-encryption.md b/nats_streaming/store-encryption.md new file mode 100644 index 0000000..9dbf744 --- /dev/null +++ b/nats_streaming/store-encryption.md @@ -0,0 +1,33 @@ +# Store Encryption + +The server can be configured to encrypt a message's payload when storing them, providing encryption at rest. This can be done from the command line or from the configuration file. Check `encrypt` and `encryption_key` in the [Configuring](#configuring) section. + +It is recommended to provide the encryption key through the environment variable `NATS_STREAMING_ENCRYPTION_KEY` instead of `encryption_key`. If encryption is enabled and `NATS_STREAMING_ENCRYPTION_KEY` is found, this will take precedence over `encryption_key` value. + +You can pass this from the command line this way: +``` +$ env NATS_STREAMING_ENCRYPTION_KEY="mykey" nats-streaming-server -store file -dir datastore -encrypt +``` + +We currently support two ciphers for encryption: [AES](https://godoc.org/crypto/aes) and [CHACHA](https://godoc.org/golang.org/x/crypto/chacha20poly1305). +The default selected cipher depends on the platform. For ARM, we use `CHACHA`, otherwise we default to `AES`. You can always override that decision by explicitly specifying the cipher like this: +``` +$ env NATS_STREAMING_ENCRYPTION_KEY="mykey" nats-streaming-server -store file -dir datastore -encrypt -encryption_cipher "CHACHA" +``` +or, to select `AES`: +``` +$ env NATS_STREAMING_ENCRYPTION_KEY="mykey" nats-streaming-server -store file -dir datastore -encrypt -encryption_cipher "AES" +``` + +Note that only message payload is encrypted, all other data stored by NATS Streaming server is not. + +When running in clustering mode (see below), the server uses RAFT, which uses its own log files. Those will be encrypted too. + +Starting a server with `encrypt` against a datastore that was not encrypted may result in failures when it comes to decrypt a message, which may not happen immediately upon startup. Instead, +it will happen when attempting to deliver messages to consumers. However, when possible, the server will detect if the data was not encrypted and return the data without attempting to decrypt it. +The server will also detect which cipher was used to encrypt the data and use the proper cipher to decrypt, even if this is not the currently selected cipher. + +If the data is encrypted with a key and the server is restarted with a different key, the server will fail to decrypt messages when attempting to load them from the store. + +Performance considerations: As expected, encryption is likely to decrease performance, but by how much is hard to define. In some performance tests on a MacbookPro 2.8 GHz Intel Core i7 with SSD, we have +observed as little as 1% decrease to more than 30%. In addition to CPU cycles required for encryption, the encrypted payload is bigger, which result in more data being stored or read. diff --git a/nats_streaming/store-interface.md b/nats_streaming/store-interface.md new file mode 100644 index 0000000..bf0f591 --- /dev/null +++ b/nats_streaming/store-interface.md @@ -0,0 +1,15 @@ +# Store Interface + +Every store implementation follows the [Store interface](https://github.com/nats-io/nats-streaming-server/blob/master/stores/store.go). + +On startup, the server creates a unique instance of the `Store`. The constructor of a store implementation can do some initialization and configuration check, but *must not* access, or attempt to recover, the storage at this point. This is important because when the server runs on Fault Tolerance mode, the storage must be shared across many servers but only one server can be using it. + +After instantiating the store, the server will then call `Recover()` in order to recover the persisted state. For implementations that do not support persistence, such as the provided `MemoryStore`, this call will simply return `nil` (without error) to indicate that no state was recovered. + +The `Store` is used to add/delete clients, create/lookup channels, etc... + +Creating/looking up a channel will return a `ChannelStore`, which points to two other interfaces, the `SubStore` and `MsgStore`. These stores, for a given channel, handle subscriptions and messages respectively. + +If you wish to contribute to a new store type, your implementation must include all these interfaces. For stores that allow recovery (such as file store as opposed to memory store), there are additional structures that have been defined and should be returned by `Recover()`. + +The memory and the provided file store implementations both use a generic store implementation to avoid code duplication. When writing your own store implementation, you can do the same for APIs that don't need to do more than what the generic implementation provides. You can check [MemStore](https://github.com/nats-io/nats-streaming-server/blob/master/stores/memstore.go) and [FileStore](https://github.com/nats-io/nats-streaming-server/blob/master/stores/filestore.go) implementations for more details. diff --git a/nats_streaming/nats-streaming-swarm.md b/nats_streaming/swarm.md similarity index 100% rename from nats_streaming/nats-streaming-swarm.md rename to nats_streaming/swarm.md