1. Riak Crashing on Data Load from CSV and 2. How to get Ripple to load js functions

Kresten Krab Thorup krab at trifork.com
Tue May 22 13:02:15 EDT 2012


It looks like you have run out of file descriptors. The crash is an emfile.

What does 'ulimit -n' print?

Kresten

On 22/05/2012, at 18.24, "Gustav Gennrich" <ggennrich at gmail.com> wrote:

> Hi Dave,
>
> This is some output from the crash.log file on the nodes. app.config file is at the end:
>
> Node 1 (Hosting Riak Control):
>
> 2012-05-18 04:58:25 =ERROR REPORT====
> {mochiweb_socket_server,310,{acceptor_error,{error,accept_failed}}}2012-05-18 04:58:25 =ERROR REPORT====
> {mochiweb_socket_server,310,{acceptor_error,{error,accept_failed}}}2012-05-18 04:58:25 =ERROR REPORT====
> {mochiweb_socket_server,310,{acceptor_error,{error,accept_failed}}}2012-05-18 04:58:25 =ERROR REPORT====
> {mochiweb_socket_server,310,{acceptor_error,{error,accept_failed}}}2012-05-18 04:58:26 =ERROR REPORT====
> {mochiweb_socket_server,310,{acceptor_error,{error,accept_failed}}}2012-05-18 04:58:26 =ERROR REPORT====
> {mochiweb_socket_server,310,{acceptor_error,{error,accept_failed}}}2012-05-18 04:58:26 =ERROR REPORT====
> {mochiweb_socket_server,310,{acceptor_error,{error,accept_failed}}}2012-05-18 04:58:26 =ERROR REPORT====
> {mochiweb_socket_server,310,{acceptor_error,{error,accept_failed}}}2012-05-18 04:58:26 =ERROR REPORT====
> {mochiweb_socket_server,310,{acceptor_error,{error,accept_failed}}}2012-05-18 04:58:26 =ERROR REPORT====
> {mochiweb_socket_server,310,{acceptor_error,{error,accept_failed}}}2012-05-18 04:58:26 =ERROR REPORT====
> {mochiweb_socket_server,310,{acceptor_error,{error,accept_failed}}}2012-05-18 04:58:26 =ERROR REPORT====
> {mochiweb_socket_server,310,{acceptor_error,{error,accept_failed}}}2012-05-18 04:58:26 =ERROR REPORT====
> {mochiweb_socket_server,310,{acceptor_error,{error,accept_failed}}}2012-05-18 04:58:26 =ERROR REPORT====
> {mochiweb_socket_server,310,{acceptor_error,{error,accept_failed}}}2012-05-18 04:58:27 =ERROR REPORT====
> {mochiweb_socket_server,310,{acceptor_error,{error,accept_failed}}}2012-05-18 04:59:00 =ERROR REPORT====
> ** Generic server riak_kv_stat terminating
> ** Last message in was {'$gen_cast',{update,{get_fsm,undefined,1158,1,4559},63504536340}}
> ** When Server state == {state,{spiral,63504536340,[1,0,0,0,0,0,0,0,0,0,0,0,0,0,14,0,0,0,0,0,0,0,0,0,0,0,8,0,0,11,0,0,0,0,0,14,0,0,0,17,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]},{spiral,63504536326,[7,0,0,0,0,0,0,0,0,0,0,0,4,0,0,7,0,0,0,0,0,9,0,0,0,11,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]},65,38,{spiral,63504536326,[84,0,0,0,0,0,0,0,0,0,0,0,84,0,0,84,0,0,0,0,0,126,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]},378,{spiral,63504536326,[7,0,0,0,0,0,0,0,0,0,0,0,4,0,0,7,0,0,0,0,0,9,0,0,0,11,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]},38,{spiral,63504536326,[1,0,0,0,0,0,0,0,0,0,0,0,1,0,0,1,0,0,0,0,0,4,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]},7,{spiral,63504536301,[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]},0,{spiral,63504536326,[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]},1,29,18,{slide,63504536301,60,60,"/var/lib/riak/slide-data/3053/1337.331501.51152",{file_descriptor,prim_file,{#Port<0.2186418>,128}},63504536326},{slide,63504536301,60,60,"/var/lib/riak/slide-data/3053/1337.331501.51714",{file_descriptor,prim_file,{#Port<0.2186419>,177}},63504536326},{slide,63504536301,60,60,"/var/lib/riak/slide-data/3053/1337.331501.52049",{file_descriptor,prim_file,{#Port<0.2186417>,69}},63504536326},{slide,63504536301,60,60,"/var/lib/riak/slide-data/3053/1337.331501.52247",{file_descriptor,prim_file,{#Port<0.2186420>,40}},63504536326},{spiral,63504536301,[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]},0,0,{spiral,63504536301,[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]},0,undefined,0,0,undefined,undefined,0,0,true}
> ** Reason for termination ==
> ** {emfile,[{erlang,open_port,[{spawn,"/bin/sh -s unix:cmd 2>&1"},[stream]]},{os,start_port_srv_handle,1},{os,start_port_srv_loop,0}]}
> 2012-05-18 04:59:00 =CRASH REPORT====
>  crasher:
>    initial call: gen:init_it/7
>    pid: <0.31225.398>
>    registered_name: riak_kv_stat
>    exception exit: {emfile,[{erlang,open_port,[{spawn,"/bin/sh -s unix:cmd 2>&1"},[stream]]},{os,start_port_srv_handle,1},{os,start_port_srv_loop,0}]}
>      in function  gen_server2:terminate/6
>      in call from proc_lib:init_p_do_apply/3
>    ancestors: [riak_kv_sup,<0.323.0>]
>    messages: []
>    links: [#Port<0.2186419>,#Port<0.2186420>,<0.420.0>,#Port<0.2186418>]
>    dictionary: []
>    trap_exit: true
>    status: running
>    heap_size: 377
>    stack_size: 24
>    reductions: 41467
>  neighbours:
> 2012-05-18 04:59:00 =SUPERVISOR REPORT====
>     Supervisor: {local,riak_kv_sup}
>     Context:    child_terminated
>     Reason:     {emfile,[{erlang,open_port,[{spawn,"/bin/sh -s unix:cmd 2>&1"},[stream]]},{os,start_port_srv_handle,1},{os,start_port_srv_loop,0}]}
>     Offender:   [{pid,<0.31225.398>},{name,riak_kv_stat},{mfargs,{riak_kv_stat,start_link,[]}},{restart_type,permanent},{shutdown,5000},{child_type,worker}]
>
> 2012-05-18 04:59:00 =CRASH REPORT====
>  crasher:
>    initial call: riak_core_vnode:init/1
>    pid: <0.22390.463>
>    registered_name: []
>    exception exit: {{badmatch,{error,emfile}},[{riak_kv_vnode,init,1},{riak_core_vnode,init,1},{gen_fsm,init_it,6},{proc_lib,init_p_do_apply,3}]}
>      in function  gen_fsm:init_it/6
>      in call from proc_lib:init_p_do_apply/3
>    ancestors: [riak_core_vnode_sup,riak_core_sup,<0.91.0>]
>    messages: []
>    links: [<0.95.0>]
>    dictionary: []
>    trap_exit: true
>    status: running
>    heap_size: 987
>    stack_size: 24
>    reductions: 2734
>  neighbours:
> 2012-05-18 04:59:00 =SUPERVISOR REPORT====
>     Supervisor: {local,riak_core_vnode_sup}
>     Context:    child_terminated
>     Reason:     {{badmatch,{error,emfile}},[{riak_kv_vnode,init,1},{riak_core_vnode,init,1},{gen_fsm,init_it,6},{proc_lib,init_p_do_apply,3}]}
>     Offender:   [{pid,<0.22390.463>},{name,undefined},{mfargs,{riak_core_vnode,start_link,undefined}},{restart_type,temporary},{shutdown,300000},{child_type,worker}]
>
> 2012-05-18 04:59:00 =ERROR REPORT====
> ** Generic server riak_core_vnode_manager terminating
> ** Last message in was {137015778499772148581595453067151533092743675904,riak_kv_vnode,get_vnode}
> ** When Server state == {state,40798511124,[{{riak_kv_vnode,0},undefined},{{riak_kv_vnode,11417981541647679048466287755595961091061972992},undefined},{{riak_kv_vnode,22835963083295358096932575511191922182123945984},undefined},{{riak_kv_vnode,34253944624943037145398863266787883273185918976},undefined},{{riak_kv_vnode,45671926166590716193865151022383844364247891968},undefined},{{riak_kv_vnode,57089907708238395242331438777979805455309864960},undefined},{{riak_kv_vnode,68507889249886074290797726533575766546371837952},undefined},{{riak_kv_vnode,79925870791533753339264014289171727637433810944},undefined},{{riak_kv_vnode,91343852333181432387730302044767688728495783936},undefined},{{riak_kv_vnode,102761833874829111436196589800363649819557756928},undefined},{{riak_kv_vnode,114179815416476790484662877555959610910619729920},undefined},{{riak_kv_vnode,125597796958124469533129165311555572001681702912},undefined},{{riak_kv_vnode,137015778499772148581595453067151533092743675904},undefined},{{riak_kv_vnode,148433760041419827630061740822747494183805648896},undefined},{{riak_kv_vnode,159851741583067506678528028578343455274867621888},undefined},{{riak_kv_vnode,171269723124715185726994316333939416365929594880},undefined},{{riak_kv_vnode,182687704666362864775460604089535377456991567872},undefined},{{riak_kv_vnode,194105686208010543823926891845131338548053540864},undefined},{{riak_kv_vnode,205523667749658222872393179600727299639115513856},undefined},{{riak_kv_vnode,216941649291305901920859467356323260730177486848},undefined},{{riak_kv_vnode,228359630832953580969325755111919221821239459840},undefined},{{riak_kv_vnode,239777612374601260017792042867515182912301432832},undefined},{{riak_kv_vnode,251195593916248939066258330623111144003363405824},undefined},{{riak_kv_vnode,262613575457896618114724618378707105094425378816},undefined},{{riak_kv_vnode,274031556999544297163190906134303066185487351808},undefined},{{riak_kv_vnode,285449538541191976211657193889899027276549324800},undefined},{{riak_kv_vnode,296867520082839655260123481645494988367611297792},undefined},{{riak_kv_vnode,308285501624487334308589769401090949458673270784},undefined},{{riak_kv_vnode,319703483166135013357056057156686910549735243776},undefined},{{riak_kv_vnode,331121464707782692405522344912282871640797216768},undefined},{{riak_kv_vnode,342539446249430371453988632667878832731859189760},undefined},{{riak_kv_vnode,353957427791078050502454920423474793822921162752},undefined},{{riak_kv_vnode,365375409332725729550921208179070754913983135744},undefined},{{riak_kv_vnode,376793390874373408599387495934666716005045108736},undefined},{{riak_kv_vnode,388211372416021087647853783690262677096107081728},undefined},{{riak_kv_vnode,399629353957668766696320071445858638187169054720},undefined},{{riak_kv_vnode,411047335499316445744786359201454599278231027712},undefined},{{riak_kv_vnode,422465317040964124793252646957050560369293000704},undefined},{{riak_kv_vnode,433883298582611803841718934712646521460354973696},undefined},{{riak_kv_vnode,445301280124259482890185222468242482551416946688},undefined},{{riak_kv_vnode,456719261665907161938651510223838443642478919680},undefined},{{riak_kv_vnode,468137243207554840987117797979434404733540892672},undefined},{{riak_kv_vnode,479555224749202520035584085735030365824602865664},undefined},{{riak_kv_vnode,490973206290850199084050373490626326915664838656},undefined},{{riak_kv_vnode,502391187832497878132516661246222288006726811648},undefined},{{riak_kv_vnode,513809169374145557180982949001818249097788784640},undefined},{{riak_kv_vnode,525227150915793236229449236757414210188850757632},undefined},{{riak_kv_vnode,536645132457440915277915524513010171279912730624},undefined},{{riak_kv_vnode,548063113999088594326381812268606132370974703616},undefined},{{riak_kv_vnode,559481095540736273374848100024202093462036676608},undefined},{{riak_kv_vnode,570899077082383952423314387779798054553098649600},undefined},{{riak_kv_vnode,582317058624031631471780675535394015644160622592},undefined},{{riak_kv_vnode,593735040165679310520246963290989976735222595584},undefined},{{riak_kv_vnode,605153021707326989568713251046585937826284568576},undefined},{{riak_kv_vnode,616571003248974668617179538802181898917346541568},undefined},{{riak_kv_vnode,627988984790622347665645826557777860008408514560},undefined},{{riak_kv_vnode,639406966332270026714112114313373821099470487552},undefined},{{riak_kv_vnode,650824947873917705762578402068969782190532460544},undefined},{{riak_kv_vnode,662242929415565384811044689824565743281594433536},undefined},{{riak_kv_vnode,673660910957213063859510977580161704372656406528},undefined},{{riak_kv_vnode,685078892498860742907977265335757665463718379520},undefined},{{riak_kv_vnode,696496874040508421956443553091353626554780352512},undefined},{{riak_kv_vnode,707914855582156101004909840846949587645842325504},undefined},{{riak_kv_vnode,719332837123803780053376128602545548736904298496},undefined},{{riak_kv_vnode,730750818665451459101842416358141509827966271488},undefined},{{riak_kv_vnode,742168800207099138150308704113737470919028244480},undefined},{{riak_kv_vnode,753586781748746817198774991869333432010090217472},undefined},{{riak_kv_vnode,765004763290394496247241279624929393101152190464},undefined},{{riak_kv_vnode,776422744832042175295707567380525354192214163456},undefined},{{riak_kv_vnode,787840726373689854344173855136121315283276136448},undefined},{{riak_kv_vnode,799258707915337533392640142891717276374338109440},undefined},{{riak_kv_vnode,810676689456985212441106430647313237465400082432},undefined},{{riak_kv_vnode,822094670998632891489572718402909198556462055424},undefined},{{riak_kv_vnode,833512652540280570538039006158505159647524028416},undefined},{{riak_kv_vnode,844930634081928249586505293914101120738586001408},undefined},{{riak_kv_vnode,856348615623575928634971581669697081829647974400},undefined},{{riak_kv_vnode,867766597165223607683437869425293042920709947392},undefined},{{riak_kv_vnode,879184578706871286731904157180889004011771920384},undefined},{{riak_kv_vnode,890602560248518965780370444936484965102833893376},undefined},{{riak_kv_vnode,902020541790166644828836732692080926193895866368},undefined},{{riak_kv_vnode,913438523331814323877303020447676887284957839360},undefined},{{riak_kv_vnode,924856504873462002925769308203272848376019812352},undefined},{{riak_kv_vnode,936274486415109681974235595958868809467081785344},undefined},{{riak_kv_vnode,947692467956757361022701883714464770558143758336},undefined},{{riak_kv_vnode,959110449498405040071168171470060731649205731328},undefined},{{riak_kv_vnode,970528431040052719119634459225656692740267704320},undefined},{{riak_kv_vnode,981946412581700398168100746981252653831329677312},undefined},{{riak_kv_vnode,993364394123348077216567034736848614922391650304},undefined},{{riak_kv_vnode,1004782375664995756265033322492444576013453623296},undefined},{{riak_kv_vnode,1016200357206643435313499610248040537104515596288},undefined},{{riak_kv_vnode,1027618338748291114361965898003636498195577569280},undefined},{{riak_kv_vnode,1039036320289938793410432185759232459286639542272},undefined},{{riak_kv_vnode,1050454301831586472458898473514828420377701515264},undefined},{{riak_kv_vnode,1061872283373234151507364761270424381468763488256},undefined},{{riak_kv_vnode,1073290264914881830555831049026020342559825461248},undefined},{{riak_kv_vnode,1084708246456529509604297336781616303650887434240},undefined},{{riak_kv_vnode,1096126227998177188652763624537212264741949407232},undefined},{{riak_kv_vnode,1107544209539824867701229912292808225833011380224},undefined},{{riak_kv_vnode,1118962191081472546749696200048404186924073353216},undefined},{{riak_kv_vnode,1130380172623120225798162487804000148015135326208},undefined},{{riak_kv_vnode,1141798154164767904846628775559596109106197299200},undefined},{{riak_kv_vnode,1153216135706415583895095063315192070197259272192},undefined},{{riak_kv_vnode,1164634117248063262943561351070788031288321245184},undefined},{{riak_kv_vnode,1176052098789710941992027638826383992379383218176},undefined},{{riak_kv_vnode,1187470080331358621040493926581979953470445191168},undefined},{{riak_kv_vnode,1198888061873006300088960214337575914561507164160},undefined},{{riak_kv_vnode,1210306043414653979137426502093171875652569137152},undefined},{{riak_kv_vnode,1221724024956301658185892789848767836743631110144},undefined},{{riak_kv_vnode,1233142006497949337234359077604363797834693083136},undefined},{{riak_kv_vnode,1244559988039597016282825365359959758925755056128},undefined},{{riak_kv_vnode,1255977969581244695331291653115555720016817029120},undefined},{{riak_kv_vnode,1267395951122892374379757940871151681107879002112},undefined},{{riak_kv_vnode,1278813932664540053428224228626747642198940975104},undefined},{{riak_kv_vnode,1290231914206187732476690516382343603290002948096},undefined},{{riak_kv_vnode,1301649895747835411525156804137939564381064921088},undefined},{{riak_kv_vnode,1313067877289483090573623091893535525472126894080},undefined},{{riak_kv_vnode,1324485858831130769622089379649131486563188867072},undefined},{{riak_kv_vnode,1335903840372778448670555667404727447654250840064},undefined},{{riak_kv_vnode,1347321821914426127719021955160323408745312813056},undefined},{{riak_kv_vnode,1358739803456073806767488242915919369836374786048},undefined},{{riak_kv_vnode,1370157784997721485815954530671515330927436759040},undefined},{{riak_kv_vnode,1381575766539369164864420818427111292018498732032},undefined},{{riak_kv_vnode,1392993748081016843912887106182707253109560705024},undefined},{{riak_kv_vnode,1404411729622664522961353393938303214200622678016},undefined},{{riak_kv_vnode,1415829711164312202009819681693899175291684651008},undefined},{{riak_kv_vnode,1427247692705959881058285969449495136382746624000},undefined},{{riak_kv_vnode,1438665674247607560106752257205091097473808596992},undefined},{{riak_kv_vnode,1450083655789255239155218544960687058564870569984},undefined},{{riak_pipe_vnode,0},undefined},{{riak_pipe_vnode,11417981541647679048466287755595961091061972992},undefined},{{riak_pipe_vnode,22835963083295358096932575511191922182123945984},undefined},{{riak_pipe_vnode,34253944624943037145398863266787883273185918976},undefined},{{riak_pipe_vnode,45671926166590716193865151022383844364247891968},undefined},{{riak_pipe_vnode,57089907708238395242331438777979805455309864960},undefined},{{riak_pipe_vnode,68507889249886074290797726533575766546371837952},undefined},{{riak_pipe_vnode,79925870791533753339264014289171727637433810944},undefined},{{riak_pipe_vnode,91343852333181432387730302044767688728495783936},undefined},{{riak_pipe_vnode,102761833874829111436196589800363649819557756928},undefined},{{riak_pipe_vnode,114179815416476790484662877555959610910619729920},undefined},{{riak_pipe_vnode,125597796958124469533129165311555572001681702912},undefined},{{riak_pipe_vnode,137015778499772148581595453067151533092743675904},undefined},{{riak_pipe_vnode,148433760041419827630061740822747494183805648896},undefined},{{riak_pipe_vnode,159851741583067506678528028578343455274867621888},undefined},{{riak_pipe_vnode,171269723124715185726994316333939416365929594880},undefined},{{riak_pipe_vnode,182687704666362864775460604089535377456991567872},undefined},{{riak_pipe_vnode,194105686208010543823926891845131338548053540864},undefined},{{riak_pipe_vnode,205523667749658222872393179600727299639115513856},undefined},{{riak_pipe_vnode,216941649291305901920859467356323260730177486848},undefined},{{riak_pipe_vnode,228359630832953580969325755111919221821239459840},undefined},{{riak_pipe_vnode,239777612374601260017792042867515182912301432832},undefined},{{riak_pipe_vnode,251195593916248939066258330623111144003363405824},undefined},{{riak_pipe_vnode,262613575457896618114724618378707105094425378816},undefined},{{riak_pipe_vnode,274031556999544297163190906134303066185487351808},undefined},{{riak_pipe_vnode,285449538541191976211657193889899027276549324800},undefined},{{riak_pipe_vnode,296867520082839655260123481645494988367611297792},undefined},{{riak_pipe_vnode,308285501624487334308589769401090949458673270784},undefined},{{riak_pipe_vnode,319703483166135013357056057156686910549735243776},undefined},{{riak_pipe_vnode,331121464707782692405522344912282871640797216768},undefined},{{riak_pipe_vnode,342539446249430371453988632667878832731859189760},undefined},{{riak_pipe_vnode,353957427791078050502454920423474793822921162752},undefined},{{riak_pipe_vnode,365375409332725729550921208179070754913983135744},undefined},{{riak_pipe_vnode,376793390874373408599387495934666716005045108736},undefined},{{riak_pipe_vnode,388211372416021087647853783690262677096107081728},undefined},{{riak_pipe_vnode,399629353957668766696320071445858638187169054720},undefined},{{riak_pipe_vnode,411047335499316445744786359201454599278231027712},undefined},{{riak_pipe_vnode,422465317040964124793252646957050560369293000704},undefined},{{riak_pipe_vnode,433883298582611803841718934712646521460354973696},undefined},{{riak_pipe_vnode,445301280124259482890185222468242482551416946688},undefined},{{riak_pipe_vnode,456719261665907161938651510223838443642478919680},undefined},{{riak_pipe_vnode,468137243207554840987117797979434404733540892672},undefined},{{riak_pipe_vnode,479555224749202520035584085735030365824602865664},undefined},{{riak_pipe_vnode,490973206290850199084050373490626326915664838656},undefined},{{riak_pipe_vnode,502391187832497878132516661246222288006726811648},undefined},{{riak_pipe_vnode,513809169374145557180982949001818249097788784640},undefined},{{riak_pipe_vnode,525227150915793236229449236757414210188850757632},undefined},{{riak_pipe_vnode,536645132457440915277915524513010171279912730624},undefined},{{riak_pipe_vnode,548063113999088594326381812268606132370974703616},undefined},{{riak_pipe_vnode,559481095540736273374848100024202093462036676608},undefined},{{riak_pipe_vnode,570899077082383952423314387779798054553098649600},undefined},{{riak_pipe_vnode,582317058624031631471780675535394015644160622592},undefined},{{riak_pipe_vnode,593735040165679310520246963290989976735222595584},undefined},{{riak_pipe_vnode,605153021707326989568713251046585937826284568576},undefined},{{riak_pipe_vnode,616571003248974668617179538802181898917346541568},undefined},{{riak_pipe_vnode,627988984790622347665645826557777860008408514560},undefined},{{riak_pipe_vnode,639406966332270026714112114313373821099470487552},undefined},{{riak_pipe_vnode,650824947873917705762578402068969782190532460544},undefined},{{riak_pipe_vnode,662242929415565384811044689824565743281594433536},undefined},{{riak_pipe_vnode,673660910957213063859510977580161704372656406528},undefined},{{riak_pipe_vnode,685078892498860742907977265335757665463718379520},undefined},{{riak_pipe_vnode,696496874040508421956443553091353626554780352512},undefined},{{riak_pipe_vnode,707914855582156101004909840846949587645842325504},undefined},{{riak_pipe_vnode,719332837123803780053376128602545548736904298496},undefined},{{riak_pipe_vnode,730750818665451459101842416358141509827966271488},undefined},{{riak_pipe_vnode,742168800207099138150308704113737470919028244480},undefined},{{riak_pipe_vnode,753586781748746817198774991869333432010090217472},undefined},{{riak_pipe_vnode,765004763290394496247241279624929393101152190464},undefined},{{riak_pipe_vnode,776422744832042175295707567380525354192214163456},undefined},{{riak_pipe_vnode,787840726373689854344173855136121315283276136448},undefined},{{riak_pipe_vnode,799258707915337533392640142891717276374338109440},undefined},{{riak_pipe_vnode,810676689456985212441106430647313237465400082432},undefined},{{riak_pipe_vnode,822094670998632891489572718402909198556462055424},undefined},{{riak_pipe_vnode,833512652540280570538039006158505159647524028416},undefined},{{riak_pipe_vnode,844930634081928249586505293914101120738586001408},undefined},{{riak_pipe_vnode,856348615623575928634971581669697081829647974400},undefined},{{riak_pipe_vnode,867766597165223607683437869425293042920709947392},undefined},{{riak_pipe_vnode,879184578706871286731904157180889004011771920384},undefined},{{riak_pipe_vnode,890602560248518965780370444936484965102833893376},undefined},{{riak_pipe_vnode,902020541790166644828836732692080926193895866368},undefined},{{riak_pipe_vnode,913438523331814323877303020447676887284957839360},undefined},{{riak_pipe_vnode,924856504873462002925769308203272848376019812352},undefined},{{riak_pipe_vnode,936274486415109681974235595958868809467081785344},undefined},{{riak_pipe_vnode,947692467956757361022701883714464770558143758336},undefined},{{riak_pipe_vnode,959110449498405040071168171470060731649205731328},undefined},{{riak_pipe_vnode,970528431040052719119634459225656692740267704320},undefined},{{riak_pipe_vnode,981946412581700398168100746981252653831329677312},undefined},{{riak_pipe_vnode,993364394123348077216567034736848614922391650304},undefined},{{riak_pipe_vnode,1004782375664995756265033322492444576013453623296},undefined},{{riak_pipe_vnode,1016200357206643435313499610248040537104515596288},undefined},{{riak_pipe_vnode,1027618338748291114361965898003636498195577569280},undefined},{{riak_pipe_vnode,1039036320289938793410432185759232459286639542272},undefined},{{riak_pipe_vnode,1050454301831586472458898473514828420377701515264},undefined},{{riak_pipe_vnode,1061872283373234151507364761270424381468763488256},undefined},{{riak_pipe_vnode,1073290264914881830555831049026020342559825461248},undefined},{{riak_pipe_vnode,1084708246456529509604297336781616303650887434240},undefined},{{riak_pipe_vnode,1096126227998177188652763624537212264741949407232},undefined},{{riak_pipe_vnode,1107544209539824867701229912292808225833011380224},undefined},{{riak_pipe_vnode,1118962191081472546749696200048404186924073353216},undefined},{{riak_pipe_vnode,1130380172623120225798162487804000148015135326208},undefined},{{riak_pipe_vnode,1141798154164767904846628775559596109106197299200},undefined},{{riak_pipe_vnode,1153216135706415583895095063315192070197259272192},undefined},{{riak_pipe_vnode,1164634117248063262943561351070788031288321245184},undefined},{{riak_pipe_vnode,1176052098789710941992027638826383992379383218176},undefined},{{riak_pipe_vnode,1187470080331358621040493926581979953470445191168},undefined},{{riak_pipe_vnode,1198888061873006300088960214337575914561507164160},undefined},{{riak_pipe_vnode,1210306043414653979137426502093171875652569137152},undefined},{{riak_pipe_vnode,1221724024956301658185892789848767836743631110144},undefined},{{riak_pipe_vnode,1233142006497949337234359077604363797834693083136},undefined},{{riak_pipe_vnode,1244559988039597016282825365359959758925755056128},undefined},{{riak_pipe_vnode,1255977969581244695331291653115555720016817029120},undefined},{{riak_pipe_vnode,1267395951122892374379757940871151681107879002112},undefined},{{riak_pipe_vnode,1278813932664540053428224228626747642198940975104},undefined},{{riak_pipe_vnode,1290231914206187732476690516382343603290002948096},undefined},{{riak_pipe_vnode,1301649895747835411525156804137939564381064921088},undefined},{{riak_pipe_vnode,1313067877289483090573623091893535525472126894080},undefined},{{riak_pipe_vnode,1324485858831130769622089379649131486563188867072},undefined},{{riak_pipe_vnode,1335903840372778448670555667404727447654250840064},undefined},{{riak_pipe_vnode,1347321821914426127719021955160323408745312813056},undefined},{{riak_pipe_vnode,1358739803456073806767488242915919369836374786048},undefined},{{riak_pipe_vnode,1370157784997721485815954530671515330927436759040},undefined},{{riak_pipe_vnode,1381575766539369164864420818427111292018498732032},undefined},{{riak_pipe_vnode,1392993748081016843912887106182707253109560705024},undefined},{{riak_pipe_vnode,1404411729622664522961353393938303214200622678016},undefined},{{riak_pipe_vnode,1415829711164312202009819681693899175291684651008},undefined},{{riak_pipe_vnode,1427247692705959881058285969449495136382746624000},undefined},{{riak_pipe_vnode,1438665674247607560106752257205091097473808596992},undefined},{{riak_pipe_vnode,1450083655789255239155218544960687058564870569984},undefined},{{riak_search_vnode,0},undefined},{{riak_search_vnode,11417981541647679048466287755595961091061972992},undefined},{{riak_search_vnode,22835963083295358096932575511191922182123945984},undefined},{{riak_search_vnode,34253944624943037145398863266787883273185918976},undefined},{{riak_search_vnode,45671926166590716193865151022383844364247891968},undefined},{{riak_search_vnode,57089907708238395242331438777979805455309864960},undefined},{{riak_search_vnode,68507889249886074290797726533575766546371837952},undefined},{{riak_search_vnode,79925870791533753339264014289171727637433810944},undefined},{{riak_search_vnode,91343852333181432387730302044767688728495783936},undefined},{{riak_search_vnode,102761833874829111436196589800363649819557756928},undefined},{{riak_search_vnode,114179815416476790484662877555959610910619729920},undefined},{{riak_search_vnode,125597796958124469533129165311555572001681702912},undefined},{{riak_search_vnode,137015778499772148581595453067151533092743675904},undefined},{{riak_search_vnode,148433760041419827630061740822747494183805648896},undefined},{{riak_search_vnode,159851741583067506678528028578343455274867621888},undefined},{{riak_search_vnode,171269723124715185726994316333939416365929594880},undefined},{{riak_search_vnode,182687704666362864775460604089535377456991567872},undefined},{{riak_search_vnode,194105686208010543823926891845131338548053540864},undefined},{{riak_search_vnode,205523667749658222872393179600727299639115513856},undefined},{{riak_search_vnode,216941649291305901920859467356323260730177486848},undefined},{{riak_search_vnode,228359630832953580969325755111919221821239459840},undefined},{{riak_search_vnode,239777612374601260017792042867515182912301432832},undefined},{{riak_search_vnode,251195593916248939066258330623111144003363405824},undefined},{{riak_search_vnode,262613575457896618114724618378707105094425378816},undefined},{{riak_search_vnode,274031556999544297163190906134303066185487351808},undefined},{{riak_search_vnode,285449538541191976211657193889899027276549324800},undefined},{{riak_search_vnode,296867520082839655260123481645494988367611297792},undefined},{{riak_search_vnode,308285501624487334308589769401090949458673270784},undefined},{{riak_search_vnode,319703483166135013357056057156686910549735243776},undefined},{{riak_search_vnode,331121464707782692405522344912282871640797216768},undefined},{{riak_search_vnode,342539446249430371453988632667878832731859189760},undefined},{{riak_search_vnode,353957427791078050502454920423474793822921162752},undefined},{{riak_search_vnode,365375409332725729550921208179070754913983135744},undefined},{{riak_search_vnode,376793390874373408599387495934666716005045108736},undefined},{{riak_search_vnode,388211372416021087647853783690262677096107081728},undefined},{{riak_search_vnode,399629353957668766696320071445858638187169054720},undefined},{{riak_search_vnode,411047335499316445744786359201454599278231027712},undefined},{{riak_search_vnode,422465317040964124793252646957050560369293000704},undefined},{{riak_search_vnode,433883298582611803841718934712646521460354973696},undefined},{{riak_search_vnode,445301280124259482890185222468242482551416946688},undefined},{{riak_search_vnode,456719261665907161938651510223838443642478919680},undefined},{{riak_search_vnode,468137243207554840987117797979434404733540892672},undefined},{{riak_search_vnode,479555224749202520035584085735030365824602865664},undefined},{{riak_search_vnode,490973206290850199084050373490626326915664838656},undefined},{{riak_search_vnode,502391187832497878132516661246222288006726811648},undefined},{{riak_search_vnode,513809169374145557180982949001818249097788784640},undefined},{{riak_search_vnode,525227150915793236229449236757414210188850757632},undefined},{{riak_search_vnode,536645132457440915277915524513010171279912730624},undefined},{{riak_search_vnode,548063113999088594326381812268606132370974703616},undefined},{{riak_search_vnode,559481095540736273374848100024202093462036676608},undefined},{{riak_search_vnode,570899077082383952423314387779798054553098649600},undefined},{{riak_search_vnode,582317058624031631471780675535394015644160622592},undefined},{{riak_search_vnode,593735040165679310520246963290989976735222595584},undefined},{{riak_search_vnode,605153021707326989568713251046585937826284568576},undefined},{{riak_search_vnode,616571003248974668617179538802181898917346541568},undefined},{{riak_search_vnode,627988984790622347665645826557777860008408514560},undefined},{{riak_search_vnode,639406966332270026714112114313373821099470487552},undefined},{{riak_search_vnode,650824947873917705762578402068969782190532460544},undefined},{{riak_search_vnode,662242929415565384811044689824565743281594433536},undefined},{{riak_search_vnode,673660910957213063859510977580161704372656406528},undefined},{{riak_search_vnode,685078892498860742907977265335757665463718379520},undefined},{{riak_search_vnode,696496874040508421956443553091353626554780352512},undefined},{{riak_search_vnode,707914855582156101004909840846949587645842325504},undefined},{{riak_search_vnode,719332837123803780053376128602545548736904298496},undefined},{{riak_search_vnode,730750818665451459101842416358141509827966271488},undefined},{{riak_search_vnode,742168800207099138150308704113737470919028244480},undefined},{{riak_search_vnode,753586781748746817198774991869333432010090217472},undefined},{{riak_search_vnode,765004763290394496247241279624929393101152190464},undefined},{{riak_search_vnode,776422744832042175295707567380525354192214163456},undefined},{{riak_search_vnode,787840726373689854344173855136121315283276136448},undefined},{{riak_search_vnode,799258707915337533392640142891717276374338109440},undefined},{{riak_search_vnode,810676689456985212441106430647313237465400082432},undefined},{{riak_search_vnode,822094670998632891489572718402909198556462055424},undefined},{{riak_search_vnode,833512652540280570538039006158505159647524028416},undefined},{{riak_search_vnode,844930634081928249586505293914101120738586001408},undefined},{{riak_search_vnode,856348615623575928634971581669697081829647974400},undefined},{{riak_search_vnode,867766597165223607683437869425293042920709947392},undefined},{{riak_search_vnode,879184578706871286731904157180889004011771920384},undefined},{{riak_search_vnode,890602560248518965780370444936484965102833893376},undefined},{{riak_search_vnode,902020541790166644828836732692080926193895866368},undefined},{{riak_search_vnode,913438523331814323877303020447676887284957839360},undefined},{{riak_search_vnode,924856504873462002925769308203272848376019812352},undefined},{{riak_search_vnode,936274486415109681974235595958868809467081785344},undefined},{{riak_search_vnode,947692467956757361022701883714464770558143758336},undefined},{{riak_search_vnode,959110449498405040071168171470060731649205731328},undefined},{{riak_search_vnode,970528431040052719119634459225656692740267704320},undefined},{{riak_search_vnode,981946412581700398168100746981252653831329677312},undefined},{{riak_search_vnode,993364394123348077216567034736848614922391650304},undefined},{{riak_search_vnode,1004782375664995756265033322492444576013453623296},undefined},{{riak_search_vnode,1016200357206643435313499610248040537104515596288},undefined},{{riak_search_vnode,1027618338748291114361965898003636498195577569280},undefined},{{riak_search_vnode,1039036320289938793410432185759232459286639542272},undefined},{{riak_search_vnode,1050454301831586472458898473514828420377701515264},undefined},{{riak_search_vnode,1061872283373234151507364761270424381468763488256},undefined},{{riak_search_vnode,1073290264914881830555831049026020342559825461248},undefined},{{riak_search_vnode,1084708246456529509604297336781616303650887434240},undefined},{{riak_search_vnode,1096126227998177188652763624537212264741949407232},undefined},{{riak_search_vnode,1107544209539824867701229912292808225833011380224},undefined},{{riak_search_vnode,1118962191081472546749696200048404186924073353216},undefined},{{riak_search_vnode,1130380172623120225798162487804000148015135326208},undefined},{{riak_search_vnode,1141798154164767904846628775559596109106197299200},undefined},{{riak_search_vnode,1153216135706415583895095063315192070197259272192},undefined},{{riak_search_vnode,1164634117248063262943561351070788031288321245184},undefined},{{riak_search_vnode,1176052098789710941992027638826383992379383218176},undefined},{{riak_search_vnode,1187470080331358621040493926581979953470445191168},undefined},{{riak_search_vnode,1198888061873006300088960214337575914561507164160},undefined},{{riak_search_vnode,1210306043414653979137426502093171875652569137152},undefined},{{riak_search_vnode,1221724024956301658185892789848767836743631110144},undefined},{{riak_search_vnode,1233142006497949337234359077604363797834693083136},undefined},{{riak_search_vnode,1244559988039597016282825365359959758925755056128},undefined},{{riak_search_vnode,1255977969581244695331291653115555720016817029120},undefined},{{riak_search_vnode,1267395951122892374379757940871151681107879002112},undefined},{{riak_search_vnode,1278813932664540053428224228626747642198940975104},undefined},{{riak_search_vnode,1290231914206187732476690516382343603290002948096},undefined},{{riak_search_vnode,1301649895747835411525156804137939564381064921088},undefined},{{riak_search_vnode,1313067877289483090573623091893535525472126894080},undefined},{{riak_search_vnode,1324485858831130769622089379649131486563188867072},undefined},{{riak_search_vnode,1335903840372778448670555667404727447654250840064},undefined},{{riak_search_vnode,1347321821914426127719021955160323408745312813056},undefined},{{riak_search_vnode,1358739803456073806767488242915919369836374786048},undefined},{{riak_search_vnode,1370157784997721485815954530671515330927436759040},undefined},{{riak_search_vnode,1381575766539369164864420818427111292018498732032},undefined},{{riak_search_vnode,1392993748081016843912887106182707253109560705024},undefined},{{riak_search_vnode,1404411729622664522961353393938303214200622678016},undefined},{{riak_search_vnode,1415829711164312202009819681693899175291684651008},undefined},{{riak_search_vnode,1427247692705959881058285969449495136382746624000},undefined},{{riak_search_vnode,1438665674247607560106752257205091097473808596992},undefined},{{riak_search_vnode,1450083655789255239155218544960687058564870569984},undefined}],[]}
> ** Reason for termination ==
> ** {{badmatch,{error,{{badmatch,{error,emfile}},[{riak_kv_vnode,init,1},{riak_core_vnode,init,1},{gen_fsm,init_it,6},{proc_lib,init_p_do_apply,3}]}}},[{riak_core_vnode_manager,get_vnode,3},{riak_core_vnode_manager,handle_call,3},{gen_server,handle_msg,5},{proc_lib,init_p_do_apply,3}]}
> 2012-05-18 04:59:00 =CRASH REPORT====
>  crasher:
>    initial call: riak_core_vnode_manager:init/1
>    pid: <0.18742.398>
>    registered_name: riak_core_vnode_manager
>    exception exit: {{badmatch,{error,{{badmatch,{error,emfile}},[{riak_kv_vnode,init,1},{riak_core_vnode,init,1},{gen_fsm,init_it,6},{proc_lib,init_p_do_apply,3}]}}},[{riak_core_vnode_manager,get_vnode,3},{riak_core_vnode_manager,handle_call,3},{gen_server,handle_msg,5},{proc_lib,init_p_do_apply,3}]}
>      in function  gen_server:terminate/6
>      in call from proc_lib:init_p_do_apply/3
>    ancestors: [riak_core_sup,<0.91.0>]
>    messages: []
>    links: [<0.92.0>]
>    dictionary: []
>    trap_exit: false
>    status: running
>    heap_size: 17711
>    stack_size: 24
>    reductions: 546975
>  neighbours:
> 2012-05-18 04:59:00 =SUPERVISOR REPORT====
>     Supervisor: {local,riak_core_sup}
>     Context:    child_terminated
>     Reason:     {{badmatch,{error,{{badmatch,{error,emfile}},[{riak_kv_vnode,init,1},{riak_core_vnode,init,1},{gen_fsm,init_it,6},{proc_lib,init_p_do_apply,3}]}}},[{riak_core_vnode_manager,get_vnode,3},{riak_core_vnode_manager,handle_call,3},{gen_server,handle_msg,5},{proc_lib,init_p_do_apply,3}]}
>     Offender:   [{pid,<0.18742.398>},{name,riak_core_vnode_manager},{mfargs,{riak_core_vnode_manager,start_link,[]}},{restart_type,permanent},{shutdown,5000},{child_type,worker}]
>
> 2012-05-18 04:59:00 =CRASH REPORT====
>  crasher:
>    initial call: riak_core_vnode_proxy:init/1
>    pid: <0.21306.398>
>    registered_name: proxy_riak_kv_vnode_137015778499772148581595453067151533092743675904
>    exception exit: {{{badmatch,{error,{{badmatch,{error,emfile}},[{riak_kv_vnode,init,1},{riak_core_vnode,init,1},{gen_fsm,init_it,6},{proc_lib,init_p_do_apply,3}]}}},[{riak_core_vnode_manager,get_vnode,3},{riak_core_vnode_manager,handle_call,3},{gen_server,handle_msg,5},{proc_lib,init_p_do_apply,3}]},{gen_server,call,[riak_core_vnode_manager,{137015778499772148581595453067151533092743675904,riak_kv_vnode,get_vnode},infinity]}}
>      in function  gen_server:call/3
>      in call from riak_core_vnode_proxy:get_vnode_pid/1
>      in call from riak_core_vnode_proxy:handle_proxy/2
>      in call from riak_core_vnode_proxy:loop/2
>      in call from proc_lib:init_p_do_apply/3
>    ancestors: [riak_core_vnode_proxy_sup,riak_core_sup,<0.91.0>]
>    messages: []
>    links: [<0.20960.398>]
>    dictionary: []
>    trap_exit: false
>    status: running
>    heap_size: 377
>    stack_size: 24
>    reductions: 89
>  neighbours:
>
> Node 2:
>
> 2012-05-18 04:49:22 =ERROR REPORT====
> webmachine error: path="/buckets/contacts/index/email_bin/event at panchabuta.com<mailto:event at panchabuta.com>"
> {error,{error,{case_clause,{error,timeout,[[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[<<"N8XpooDmyVDCLPqGiAhjDVsXNqC">>],[],[],[],[],[],[]]}},[{riak_kv_wm_index,produce_index_results,2},{webmachine_resource,resource_call,3},{webmachine_resource,do,3},{webmachine_decision_core,resource_call,1},{webmachine_decision_core,decision,1},{webmachine_decision_core,handle_request,2},{webmachine_mochiweb,loop,1},{mochiweb_http,parse_headers,5}]}}
> 2012-05-18 04:49:34 =ERROR REPORT====
> ** Node 'riak at 10.38.24.196<mailto:riak at 10.38.24.196>' not responding **
> ** Removing (timedout) connection **
> 2012-05-18 04:49:34 =SUPERVISOR REPORT====
>     Supervisor: {local,riak_kv_put_fsm_sup}
>     Context:    child_terminated
>     Reason:     {{nodedown,'riak at 10.38.24.196<mailto:riak at 10.38.24.196>'},{gen_server,call,[{riak_search_vnode_master,'riak at 10.38.24.196<mailto:riak at 10.38.24.196>'},{riak_vnode_req_v1,1004782375664995756265033322492444576013453623296,{server,undefined,undefined},{delete_v1,[{<<"contacts">>,<<"page_keys">>,<<"njxqbdh1xgelrafoxahmlxjsuy7">>,<<"KDad4R2BVEdZXf0msX5OoTCr3lq">>,[{p,[0]}],1337330961353632}]}},infinity]}}
>     Offender:   [{pid,<0.2228.397>},{name,undefined},{mfargs,{riak_kv_put_fsm,start_link,undefined}},{restart_type,temporary},{shutdown,5000},{child_type,worker}]
>
> 2012-05-18 04:49:41 =ERROR REPORT====
> webmachine error: path="/buckets/contacts/index/email_bin/inspector at bchouseinspections.ca<mailto:inspector at bchouseinspections.ca>"
> {error,{error,{case_clause,{error,timeout,[[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]]}},[{riak_kv_wm_index,produce_index_results,2},{webmachine_resource,resource_call,3},{webmachine_resource,do,3},{webmachine_decision_core,resource_call,1},{webmachine_decision_core,decision,1},{webmachine_decision_core,handle_request,2},{webmachine_mochiweb,loop,1},{mochiweb_http,parse_headers,5}]}}
> 2012-05-18 04:50:23 =ERROR REPORT====
> webmachine error: path="/buckets/contacts/index/email_bin/event at panchabuta.com<mailto:event at panchabuta.com>"
> {error,{error,{case_clause,{error,timeout,[[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[<<"N8XpooDmyVDCLPqGiAhjDVsXNqC">>],[],[],[]]}},[{riak_kv_wm_index,produce_index_results,2},{webmachine_resource,resource_call,3},{webmachine_resource,do,3},{webmachine_decision_core,resource_call,1},{webmachine_decision_core,decision,1},{webmachine_decision_core,handle_request,2},{webmachine_mochiweb,loop,1},{mochiweb_http,parse_headers,5}]}}
> 2012-05-18 04:50:49 =ERROR REPORT====
> ** Node 'riak at 10.38.24.196<mailto:riak at 10.38.24.196>' not responding **
> ** Removing (timedout) connection **
> 2012-05-18 04:52:49 =ERROR REPORT====
> ** Node 'riak at 10.38.24.196<mailto:riak at 10.38.24.196>' not responding **
> ** Removing (timedout) connection **
> 2012-05-18 04:52:49 =SUPERVISOR REPORT====
>     Supervisor: {local,riak_kv_put_fsm_sup}
>     Context:    child_terminated
>     Reason:     {{nodedown,'riak at 10.38.24.196<mailto:riak at 10.38.24.196>'},{gen_server,call,[{riak_search_vnode_master,'riak at 10.38.24.196<mailto:riak at 10.38.24.196>'},{riak_vnode_req_v1,1050454301831586472458898473514828420377701515264,{server,undefined,undefined},{delete_v1,[{<<"sites">>,<<"updated_at">>,<<"2012-05-18">>,<<"fa3FeLb9UYhf0K1VdmR7CMdGCV">>,[{p,[0]}],1337331089082754}]}},infinity]}}
>     Offender:   [{pid,<0.3077.397>},{name,undefined},{mfargs,{riak_kv_put_fsm,start_link,undefined}},{restart_type,temporary},{shutdown,5000},{child_type,worker}]
>
> 2012-05-18 04:52:49 =SUPERVISOR REPORT====
>     Supervisor: {local,riak_kv_put_fsm_sup}
>     Context:    child_terminated
>     Reason:     {{nodedown,'riak at 10.38.24.196<mailto:riak at 10.38.24.196>'},{gen_server,call,[{riak_search_vnode_master,'riak at 10.38.24.196<mailto:riak at 10.38.24.196>'},{riak_vnode_req_v1,45671926166590716193865151022383844364247891968,{server,undefined,undefined},{delete_v1,[{<<"sites">>,<<"page_keys">>,<<"m9ksizkajltxgqeck2dix0y6vzs">>,<<"5Cd4M1awqLwGsRKJi2QkVB79h3g">>,[{p,[23]}],1337331105707883}]}},infinity]}}
>     Offender:   [{pid,<0.3102.397>},{name,undefined},{mfargs,{riak_kv_put_fsm,start_link,undefined}},{restart_type,temporary},{shutdown,5000},{child_type,worker}]
>
> 2012-05-18 04:52:49 =SUPERVISOR REPORT====
>     Supervisor: {local,riak_kv_put_fsm_sup}
>     Context:    child_terminated
>     Reason:     {{nodedown,'riak at 10.38.24.196<mailto:riak at 10.38.24.196>'},{gen_server,call,[{riak_search_vnode_master,'riak at 10.38.24.196<mailto:riak at 10.38.24.196>'},{riak_vnode_req_v1,1084708246456529509604297336781616303650887434240,{server,undefined,undefined},{index_v1,[{<<"sites">>,<<"num_contacts">>,<<"0000000000">>,<<"Z6bUs5DmmdRggHv1XjXZe30a1Zz">>,[{p,[0]}],1337331091385105}]}},infinity]}}
>     Offender:   [{pid,<0.3083.397>},{name,undefined},{mfargs,{riak_kv_put_fsm,start_link,undefined}},{restart_type,temporary},{shutdown,5000},{child_type,worker}]
>
> ....
>
> 2012-05-18 04:59:01 =ERROR REPORT====
> ** Generic server riak_core_vnode_manager terminating
> ** Last message in was {947692467956757361022701883714464770558143758336,riak_kv_vnode,get_vnode}
> ** When Server state == {state,40799215793,[{{riak_kv_vnode,0},undefined},{{riak_kv_vnode,11417981541647679048466287755595961091061972992},undefined},{{riak_kv_vnode,22835963083295358096932575511191922182123945984},undefined},{{riak_kv_vnode,34253944624943037145398863266787883273185918976},undefined},{{riak_kv_vnode,45671926166590716193865151022383844364247891968},undefined},{{riak_kv_vnode,57089907708238395242331438777979805455309864960},undefined},{{riak_kv_vnode,68507889249886074290797726533575766546371837952},undefined},{{riak_kv_vnode,79925870791533753339264014289171727637433810944},undefined},{{riak_kv_vnode,91343852333181432387730302044767688728495783936},undefined},{{riak_kv_vnode,102761833874829111436196589800363649819557756928},undefined},{{riak_kv_vnode,114179815416476790484662877555959610910619729920},undefined},{{riak_kv_vnode,125597796958124469533129165311555572001681702912},undefined},{{riak_kv_vnode,137015778499772148581595453067151533092743675904},undefined},{{riak_kv_vnode,148433760041419827630061740822747494183805648896},undefined},{{riak_kv_vnode,159851741583067506678528028578343455274867621888},undefined},{{riak_kv_vnode,171269723124715185726994316333939416365929594880},undefined},{{riak_kv_vnode,182687704666362864775460604089535377456991567872},undefined},{{riak_kv_vnode,194105686208010543823926891845131338548053540864},undefined},{{riak_kv_vnode,205523667749658222872393179600727299639115513856},undefined},{{riak_kv_vnode,216941649291305901920859467356323260730177486848},undefined},{{riak_kv_vnode,228359630832953580969325755111919221821239459840},undefined},{{riak_kv_vnode,239777612374601260017792042867515182912301432832},undefined},{{riak_kv_vnode,251195593916248939066258330623111144003363405824},undefined},{{riak_kv_vnode,262613575457896618114724618378707105094425378816},undefined},{{riak_kv_vnode,274031556999544297163190906134303066185487351808},undefined},{{riak_kv_vnode,285449538541191976211657193889899027276549324800},undefined},{{riak_kv_vnode,296867520082839655260123481645494988367611297792},undefined},{{riak_kv_vnode,308285501624487334308589769401090949458673270784},undefined},{{riak_kv_vnode,319703483166135013357056057156686910549735243776},undefined},{{riak_kv_vnode,331121464707782692405522344912282871640797216768},undefined},{{riak_kv_vnode,342539446249430371453988632667878832731859189760},undefined},{{riak_kv_vnode,353957427791078050502454920423474793822921162752},undefined},{{riak_kv_vnode,365375409332725729550921208179070754913983135744},undefined},{{riak_kv_vnode,376793390874373408599387495934666716005045108736},undefined},{{riak_kv_vnode,388211372416021087647853783690262677096107081728},undefined},{{riak_kv_vnode,399629353957668766696320071445858638187169054720},undefined},{{riak_kv_vnode,411047335499316445744786359201454599278231027712},undefined},{{riak_kv_vnode,422465317040964124793252646957050560369293000704},undefined},{{riak_kv_vnode,433883298582611803841718934712646521460354973696},undefined},{{riak_kv_vnode,445301280124259482890185222468242482551416946688},undefined},{{riak_kv_vnode,456719261665907161938651510223838443642478919680},undefined},{{riak_kv_vnode,468137243207554840987117797979434404733540892672},undefined},{{riak_kv_vnode,479555224749202520035584085735030365824602865664},undefined},{{riak_kv_vnode,490973206290850199084050373490626326915664838656},undefined},{{riak_kv_vnode,502391187832497878132516661246222288006726811648},undefined},{{riak_kv_vnode,513809169374145557180982949001818249097788784640},undefined},{{riak_kv_vnode,525227150915793236229449236757414210188850757632},undefined},{{riak_kv_vnode,536645132457440915277915524513010171279912730624},undefined},{{riak_kv_vnode,548063113999088594326381812268606132370974703616},undefined},{{riak_kv_vnode,559481095540736273374848100024202093462036676608},undefined},{{riak_kv_vnode,570899077082383952423314387779798054553098649600},undefined},{{riak_kv_vnode,582317058624031631471780675535394015644160622592},undefined},{{riak_kv_vnode,593735040165679310520246963290989976735222595584},undefined},{{riak_kv_vnode,605153021707326989568713251046585937826284568576},undefined},{{riak_kv_vnode,616571003248974668617179538802181898917346541568},undefined},{{riak_kv_vnode,627988984790622347665645826557777860008408514560},undefined},{{riak_kv_vnode,639406966332270026714112114313373821099470487552},undefined},{{riak_kv_vnode,650824947873917705762578402068969782190532460544},undefined},{{riak_kv_vnode,662242929415565384811044689824565743281594433536},undefined},{{riak_kv_vnode,673660910957213063859510977580161704372656406528},undefined},{{riak_kv_vnode,685078892498860742907977265335757665463718379520},undefined},{{riak_kv_vnode,696496874040508421956443553091353626554780352512},undefined},{{riak_kv_vnode,707914855582156101004909840846949587645842325504},undefined},{{riak_kv_vnode,719332837123803780053376128602545548736904298496},undefined},{{riak_kv_vnode,730750818665451459101842416358141509827966271488},undefined},{{riak_kv_vnode,742168800207099138150308704113737470919028244480},undefined},{{riak_kv_vnode,753586781748746817198774991869333432010090217472},undefined},{{riak_kv_vnode,765004763290394496247241279624929393101152190464},undefined},{{riak_kv_vnode,776422744832042175295707567380525354192214163456},undefined},{{riak_kv_vnode,787840726373689854344173855136121315283276136448},undefined},{{riak_kv_vnode,799258707915337533392640142891717276374338109440},undefined},{{riak_kv_vnode,810676689456985212441106430647313237465400082432},undefined},{{riak_kv_vnode,822094670998632891489572718402909198556462055424},undefined},{{riak_kv_vnode,833512652540280570538039006158505159647524028416},undefined},{{riak_kv_vnode,844930634081928249586505293914101120738586001408},undefined},{{riak_kv_vnode,856348615623575928634971581669697081829647974400},undefined},{{riak_kv_vnode,867766597165223607683437869425293042920709947392},undefined},{{riak_kv_vnode,879184578706871286731904157180889004011771920384},undefined},{{riak_kv_vnode,890602560248518965780370444936484965102833893376},undefined},{{riak_kv_vnode,902020541790166644828836732692080926193895866368},undefined},{{riak_kv_vnode,913438523331814323877303020447676887284957839360},undefined},{{riak_kv_vnode,924856504873462002925769308203272848376019812352},undefined},{{riak_kv_vnode,936274486415109681974235595958868809467081785344},undefined},{{riak_kv_vnode,947692467956757361022701883714464770558143758336},undefined},{{riak_kv_vnode,959110449498405040071168171470060731649205731328},undefined},{{riak_kv_vnode,970528431040052719119634459225656692740267704320},undefined},{{riak_kv_vnode,981946412581700398168100746981252653831329677312},undefined},{{riak_kv_vnode,993364394123348077216567034736848614922391650304},undefined},{{riak_kv_vnode,1004782375664995756265033322492444576013453623296},undefined},{{riak_kv_vnode,1016200357206643435313499610248040537104515596288},undefined},{{riak_kv_vnode,1027618338748291114361965898003636498195577569280},undefined},{{riak_kv_vnode,1039036320289938793410432185759232459286639542272},undefined},{{riak_kv_vnode,1050454301831586472458898473514828420377701515264},undefined},{{riak_kv_vnode,1061872283373234151507364761270424381468763488256},undefined},{{riak_kv_vnode,1073290264914881830555831049026020342559825461248},undefined},{{riak_kv_vnode,1084708246456529509604297336781616303650887434240},undefined},{{riak_kv_vnode,1096126227998177188652763624537212264741949407232},undefined},{{riak_kv_vnode,1107544209539824867701229912292808225833011380224},undefined},{{riak_kv_vnode,1118962191081472546749696200048404186924073353216},undefined},{{riak_kv_vnode,1130380172623120225798162487804000148015135326208},undefined},{{riak_kv_vnode,1141798154164767904846628775559596109106197299200},undefined},{{riak_kv_vnode,1153216135706415583895095063315192070197259272192},undefined},{{riak_kv_vnode,1164634117248063262943561351070788031288321245184},undefined},{{riak_kv_vnode,1176052098789710941992027638826383992379383218176},undefined},{{riak_kv_vnode,1187470080331358621040493926581979953470445191168},undefined},{{riak_kv_vnode,1198888061873006300088960214337575914561507164160},undefined},{{riak_kv_vnode,1210306043414653979137426502093171875652569137152},undefined},{{riak_kv_vnode,1221724024956301658185892789848767836743631110144},undefined},{{riak_kv_vnode,1233142006497949337234359077604363797834693083136},undefined},{{riak_kv_vnode,1244559988039597016282825365359959758925755056128},undefined},{{riak_kv_vnode,1255977969581244695331291653115555720016817029120},undefined},{{riak_kv_vnode,1267395951122892374379757940871151681107879002112},undefined},{{riak_kv_vnode,1278813932664540053428224228626747642198940975104},undefined},{{riak_kv_vnode,1290231914206187732476690516382343603290002948096},undefined},{{riak_kv_vnode,1301649895747835411525156804137939564381064921088},undefined},{{riak_kv_vnode,1313067877289483090573623091893535525472126894080},undefined},{{riak_kv_vnode,1324485858831130769622089379649131486563188867072},undefined},{{riak_kv_vnode,1335903840372778448670555667404727447654250840064},undefined},{{riak_kv_vnode,1347321821914426127719021955160323408745312813056},undefined},{{riak_kv_vnode,1358739803456073806767488242915919369836374786048},undefined},{{riak_kv_vnode,1370157784997721485815954530671515330927436759040},undefined},{{riak_kv_vnode,1381575766539369164864420818427111292018498732032},undefined},{{riak_kv_vnode,1392993748081016843912887106182707253109560705024},undefined},{{riak_kv_vnode,1404411729622664522961353393938303214200622678016},undefined},{{riak_kv_vnode,1415829711164312202009819681693899175291684651008},undefined},{{riak_kv_vnode,1427247692705959881058285969449495136382746624000},undefined},{{riak_kv_vnode,1438665674247607560106752257205091097473808596992},undefined},{{riak_kv_vnode,1450083655789255239155218544960687058564870569984},undefined},{{riak_pipe_vnode,0},undefined},{{riak_pipe_vnode,11417981541647679048466287755595961091061972992},undefined},{{riak_pipe_vnode,22835963083295358096932575511191922182123945984},undefined},{{riak_pipe_vnode,34253944624943037145398863266787883273185918976},undefined},{{riak_pipe_vnode,45671926166590716193865151022383844364247891968},undefined},{{riak_pipe_vnode,57089907708238395242331438777979805455309864960},undefined},{{riak_pipe_vnode,68507889249886074290797726533575766546371837952},undefined},{{riak_pipe_vnode,79925870791533753339264014289171727637433810944},undefined},{{riak_pipe_vnode,91343852333181432387730302044767688728495783936},undefined},{{riak_pipe_vnode,102761833874829111436196589800363649819557756928},undefined},{{riak_pipe_vnode,114179815416476790484662877555959610910619729920},undefined},{{riak_pipe_vnode,125597796958124469533129165311555572001681702912},undefined},{{riak_pipe_vnode,137015778499772148581595453067151533092743675904},undefined},{{riak_pipe_vnode,148433760041419827630061740822747494183805648896},undefined},{{riak_pipe_vnode,159851741583067506678528028578343455274867621888},undefined},{{riak_pipe_vnode,171269723124715185726994316333939416365929594880},undefined},{{riak_pipe_vnode,182687704666362864775460604089535377456991567872},undefined},{{riak_pipe_vnode,194105686208010543823926891845131338548053540864},undefined},{{riak_pipe_vnode,205523667749658222872393179600727299639115513856},undefined},{{riak_pipe_vnode,216941649291305901920859467356323260730177486848},undefined},{{riak_pipe_vnode,228359630832953580969325755111919221821239459840},undefined},{{riak_pipe_vnode,239777612374601260017792042867515182912301432832},undefined},{{riak_pipe_vnode,251195593916248939066258330623111144003363405824},undefined},{{riak_pipe_vnode,262613575457896618114724618378707105094425378816},undefined},{{riak_pipe_vnode,274031556999544297163190906134303066185487351808},undefined},{{riak_pipe_vnode,285449538541191976211657193889899027276549324800},undefined},{{riak_pipe_vnode,296867520082839655260123481645494988367611297792},undefined},{{riak_pipe_vnode,308285501624487334308589769401090949458673270784},undefined},{{riak_pipe_vnode,319703483166135013357056057156686910549735243776},undefined},{{riak_pipe_vnode,331121464707782692405522344912282871640797216768},undefined},{{riak_pipe_vnode,342539446249430371453988632667878832731859189760},undefined},{{riak_pipe_vnode,353957427791078050502454920423474793822921162752},undefined},{{riak_pipe_vnode,365375409332725729550921208179070754913983135744},undefined},{{riak_pipe_vnode,376793390874373408599387495934666716005045108736},undefined},{{riak_pipe_vnode,388211372416021087647853783690262677096107081728},undefined},{{riak_pipe_vnode,399629353957668766696320071445858638187169054720},undefined},{{riak_pipe_vnode,411047335499316445744786359201454599278231027712},undefined},{{riak_pipe_vnode,422465317040964124793252646957050560369293000704},undefined},{{riak_pipe_vnode,433883298582611803841718934712646521460354973696},undefined},{{riak_pipe_vnode,445301280124259482890185222468242482551416946688},undefined},{{riak_pipe_vnode,456719261665907161938651510223838443642478919680},undefined},{{riak_pipe_vnode,468137243207554840987117797979434404733540892672},undefined},{{riak_pipe_vnode,479555224749202520035584085735030365824602865664},undefined},{{riak_pipe_vnode,490973206290850199084050373490626326915664838656},undefined},{{riak_pipe_vnode,502391187832497878132516661246222288006726811648},undefined},{{riak_pipe_vnode,513809169374145557180982949001818249097788784640},undefined},{{riak_pipe_vnode,525227150915793236229449236757414210188850757632},undefined},{{riak_pipe_vnode,536645132457440915277915524513010171279912730624},undefined},{{riak_pipe_vnode,548063113999088594326381812268606132370974703616},undefined},{{riak_pipe_vnode,559481095540736273374848100024202093462036676608},undefined},{{riak_pipe_vnode,570899077082383952423314387779798054553098649600},undefined},{{riak_pipe_vnode,582317058624031631471780675535394015644160622592},undefined},{{riak_pipe_vnode,593735040165679310520246963290989976735222595584},undefined},{{riak_pipe_vnode,605153021707326989568713251046585937826284568576},undefined},{{riak_pipe_vnode,616571003248974668617179538802181898917346541568},undefined},{{riak_pipe_vnode,627988984790622347665645826557777860008408514560},undefined},{{riak_pipe_vnode,639406966332270026714112114313373821099470487552},undefined},{{riak_pipe_vnode,650824947873917705762578402068969782190532460544},undefined},{{riak_pipe_vnode,662242929415565384811044689824565743281594433536},undefined},{{riak_pipe_vnode,673660910957213063859510977580161704372656406528},undefined},{{riak_pipe_vnode,685078892498860742907977265335757665463718379520},undefined},{{riak_pipe_vnode,696496874040508421956443553091353626554780352512},undefined},{{riak_pipe_vnode,707914855582156101004909840846949587645842325504},undefined},{{riak_pipe_vnode,719332837123803780053376128602545548736904298496},undefined},{{riak_pipe_vnode,730750818665451459101842416358141509827966271488},undefined},{{riak_pipe_vnode,742168800207099138150308704113737470919028244480},undefined},{{riak_pipe_vnode,753586781748746817198774991869333432010090217472},undefined},{{riak_pipe_vnode,765004763290394496247241279624929393101152190464},undefined},{{riak_pipe_vnode,776422744832042175295707567380525354192214163456},undefined},{{riak_pipe_vnode,787840726373689854344173855136121315283276136448},undefined},{{riak_pipe_vnode,799258707915337533392640142891717276374338109440},undefined},{{riak_pipe_vnode,810676689456985212441106430647313237465400082432},undefined},{{riak_pipe_vnode,822094670998632891489572718402909198556462055424},undefined},{{riak_pipe_vnode,833512652540280570538039006158505159647524028416},undefined},{{riak_pipe_vnode,844930634081928249586505293914101120738586001408},undefined},{{riak_pipe_vnode,856348615623575928634971581669697081829647974400},undefined},{{riak_pipe_vnode,867766597165223607683437869425293042920709947392},undefined},{{riak_pipe_vnode,879184578706871286731904157180889004011771920384},undefined},{{riak_pipe_vnode,890602560248518965780370444936484965102833893376},undefined},{{riak_pipe_vnode,902020541790166644828836732692080926193895866368},undefined},{{riak_pipe_vnode,913438523331814323877303020447676887284957839360},undefined},{{riak_pipe_vnode,924856504873462002925769308203272848376019812352},undefined},{{riak_pipe_vnode,936274486415109681974235595958868809467081785344},undefined},{{riak_pipe_vnode,947692467956757361022701883714464770558143758336},undefined},{{riak_pipe_vnode,959110449498405040071168171470060731649205731328},undefined},{{riak_pipe_vnode,970528431040052719119634459225656692740267704320},undefined},{{riak_pipe_vnode,981946412581700398168100746981252653831329677312},undefined},{{riak_pipe_vnode,993364394123348077216567034736848614922391650304},undefined},{{riak_pipe_vnode,1004782375664995756265033322492444576013453623296},undefined},{{riak_pipe_vnode,1016200357206643435313499610248040537104515596288},undefined},{{riak_pipe_vnode,1027618338748291114361965898003636498195577569280},undefined},{{riak_pipe_vnode,1039036320289938793410432185759232459286639542272},undefined},{{riak_pipe_vnode,1050454301831586472458898473514828420377701515264},undefined},{{riak_pipe_vnode,1061872283373234151507364761270424381468763488256},undefined},{{riak_pipe_vnode,1073290264914881830555831049026020342559825461248},undefined},{{riak_pipe_vnode,1084708246456529509604297336781616303650887434240},undefined},{{riak_pipe_vnode,1096126227998177188652763624537212264741949407232},undefined},{{riak_pipe_vnode,1107544209539824867701229912292808225833011380224},undefined},{{riak_pipe_vnode,1118962191081472546749696200048404186924073353216},undefined},{{riak_pipe_vnode,1130380172623120225798162487804000148015135326208},undefined},{{riak_pipe_vnode,1141798154164767904846628775559596109106197299200},undefined},{{riak_pipe_vnode,1153216135706415583895095063315192070197259272192},undefined},{{riak_pipe_vnode,1164634117248063262943561351070788031288321245184},undefined},{{riak_pipe_vnode,1176052098789710941992027638826383992379383218176},undefined},{{riak_pipe_vnode,1187470080331358621040493926581979953470445191168},undefined},{{riak_pipe_vnode,1198888061873006300088960214337575914561507164160},undefined},{{riak_pipe_vnode,1210306043414653979137426502093171875652569137152},undefined},{{riak_pipe_vnode,1221724024956301658185892789848767836743631110144},undefined},{{riak_pipe_vnode,1233142006497949337234359077604363797834693083136},undefined},{{riak_pipe_vnode,1244559988039597016282825365359959758925755056128},undefined},{{riak_pipe_vnode,1255977969581244695331291653115555720016817029120},undefined},{{riak_pipe_vnode,1267395951122892374379757940871151681107879002112},undefined},{{riak_pipe_vnode,1278813932664540053428224228626747642198940975104},undefined},{{riak_pipe_vnode,1290231914206187732476690516382343603290002948096},undefined},{{riak_pipe_vnode,1301649895747835411525156804137939564381064921088},undefined},{{riak_pipe_vnode,1313067877289483090573623091893535525472126894080},undefined},{{riak_pipe_vnode,1324485858831130769622089379649131486563188867072},undefined},{{riak_pipe_vnode,1335903840372778448670555667404727447654250840064},undefined},{{riak_pipe_vnode,1347321821914426127719021955160323408745312813056},undefined},{{riak_pipe_vnode,1358739803456073806767488242915919369836374786048},undefined},{{riak_pipe_vnode,1370157784997721485815954530671515330927436759040},undefined},{{riak_pipe_vnode,1381575766539369164864420818427111292018498732032},undefined},{{riak_pipe_vnode,1392993748081016843912887106182707253109560705024},undefined},{{riak_pipe_vnode,1404411729622664522961353393938303214200622678016},undefined},{{riak_pipe_vnode,1415829711164312202009819681693899175291684651008},undefined},{{riak_pipe_vnode,1427247692705959881058285969449495136382746624000},undefined},{{riak_pipe_vnode,1438665674247607560106752257205091097473808596992},undefined},{{riak_pipe_vnode,1450083655789255239155218544960687058564870569984},undefined},{{riak_search_vnode,0},undefined},{{riak_search_vnode,11417981541647679048466287755595961091061972992},undefined},{{riak_search_vnode,22835963083295358096932575511191922182123945984},undefined},{{riak_search_vnode,34253944624943037145398863266787883273185918976},undefined},{{riak_search_vnode,45671926166590716193865151022383844364247891968},undefined},{{riak_search_vnode,57089907708238395242331438777979805455309864960},undefined},{{riak_search_vnode,68507889249886074290797726533575766546371837952},undefined},{{riak_search_vnode,79925870791533753339264014289171727637433810944},undefined},{{riak_search_vnode,91343852333181432387730302044767688728495783936},undefined},{{riak_search_vnode,102761833874829111436196589800363649819557756928},undefined},{{riak_search_vnode,114179815416476790484662877555959610910619729920},undefined},{{riak_search_vnode,125597796958124469533129165311555572001681702912},undefined},{{riak_search_vnode,137015778499772148581595453067151533092743675904},undefined},{{riak_search_vnode,148433760041419827630061740822747494183805648896},undefined},{{riak_search_vnode,159851741583067506678528028578343455274867621888},undefined},{{riak_search_vnode,171269723124715185726994316333939416365929594880},undefined},{{riak_search_vnode,182687704666362864775460604089535377456991567872},undefined},{{riak_search_vnode,194105686208010543823926891845131338548053540864},undefined},{{riak_search_vnode,205523667749658222872393179600727299639115513856},undefined},{{riak_search_vnode,216941649291305901920859467356323260730177486848},undefined},{{riak_search_vnode,228359630832953580969325755111919221821239459840},undefined},{{riak_search_vnode,239777612374601260017792042867515182912301432832},undefined},{{riak_search_vnode,251195593916248939066258330623111144003363405824},undefined},{{riak_search_vnode,262613575457896618114724618378707105094425378816},undefined},{{riak_search_vnode,274031556999544297163190906134303066185487351808},undefined},{{riak_search_vnode,285449538541191976211657193889899027276549324800},undefined},{{riak_search_vnode,296867520082839655260123481645494988367611297792},undefined},{{riak_search_vnode,308285501624487334308589769401090949458673270784},undefined},{{riak_search_vnode,319703483166135013357056057156686910549735243776},undefined},{{riak_search_vnode,331121464707782692405522344912282871640797216768},undefined},{{riak_search_vnode,342539446249430371453988632667878832731859189760},undefined},{{riak_search_vnode,353957427791078050502454920423474793822921162752},undefined},{{riak_search_vnode,365375409332725729550921208179070754913983135744},undefined},{{riak_search_vnode,376793390874373408599387495934666716005045108736},undefined},{{riak_search_vnode,388211372416021087647853783690262677096107081728},undefined},{{riak_search_vnode,399629353957668766696320071445858638187169054720},undefined},{{riak_search_vnode,411047335499316445744786359201454599278231027712},undefined},{{riak_search_vnode,422465317040964124793252646957050560369293000704},undefined},{{riak_search_vnode,433883298582611803841718934712646521460354973696},undefined},{{riak_search_vnode,445301280124259482890185222468242482551416946688},undefined},{{riak_search_vnode,456719261665907161938651510223838443642478919680},undefined},{{riak_search_vnode,468137243207554840987117797979434404733540892672},undefined},{{riak_search_vnode,479555224749202520035584085735030365824602865664},undefined},{{riak_search_vnode,490973206290850199084050373490626326915664838656},undefined},{{riak_search_vnode,502391187832497878132516661246222288006726811648},undefined},{{riak_search_vnode,513809169374145557180982949001818249097788784640},undefined},{{riak_search_vnode,525227150915793236229449236757414210188850757632},undefined},{{riak_search_vnode,536645132457440915277915524513010171279912730624},undefined},{{riak_search_vnode,548063113999088594326381812268606132370974703616},undefined},{{riak_search_vnode,559481095540736273374848100024202093462036676608},undefined},{{riak_search_vnode,570899077082383952423314387779798054553098649600},undefined},{{riak_search_vnode,582317058624031631471780675535394015644160622592},undefined},{{riak_search_vnode,593735040165679310520246963290989976735222595584},undefined},{{riak_search_vnode,605153021707326989568713251046585937826284568576},undefined},{{riak_search_vnode,616571003248974668617179538802181898917346541568},undefined},{{riak_search_vnode,627988984790622347665645826557777860008408514560},undefined},{{riak_search_vnode,639406966332270026714112114313373821099470487552},undefined},{{riak_search_vnode,650824947873917705762578402068969782190532460544},undefined},{{riak_search_vnode,662242929415565384811044689824565743281594433536},undefined},{{riak_search_vnode,673660910957213063859510977580161704372656406528},undefined},{{riak_search_vnode,685078892498860742907977265335757665463718379520},undefined},{{riak_search_vnode,696496874040508421956443553091353626554780352512},undefined},{{riak_search_vnode,707914855582156101004909840846949587645842325504},undefined},{{riak_search_vnode,719332837123803780053376128602545548736904298496},undefined},{{riak_search_vnode,730750818665451459101842416358141509827966271488},undefined},{{riak_search_vnode,742168800207099138150308704113737470919028244480},undefined},{{riak_search_vnode,753586781748746817198774991869333432010090217472},undefined},{{riak_search_vnode,765004763290394496247241279624929393101152190464},undefined},{{riak_search_vnode,776422744832042175295707567380525354192214163456},undefined},{{riak_search_vnode,787840726373689854344173855136121315283276136448},undefined},{{riak_search_vnode,799258707915337533392640142891717276374338109440},undefined},{{riak_search_vnode,810676689456985212441106430647313237465400082432},undefined},{{riak_search_vnode,822094670998632891489572718402909198556462055424},undefined},{{riak_search_vnode,833512652540280570538039006158505159647524028416},undefined},{{riak_search_vnode,844930634081928249586505293914101120738586001408},undefined},{{riak_search_vnode,856348615623575928634971581669697081829647974400},undefined},{{riak_search_vnode,867766597165223607683437869425293042920709947392},undefined},{{riak_search_vnode,879184578706871286731904157180889004011771920384},undefined},{{riak_search_vnode,890602560248518965780370444936484965102833893376},undefined},{{riak_search_vnode,902020541790166644828836732692080926193895866368},undefined},{{riak_search_vnode,913438523331814323877303020447676887284957839360},undefined},{{riak_search_vnode,924856504873462002925769308203272848376019812352},undefined},{{riak_search_vnode,936274486415109681974235595958868809467081785344},undefined},{{riak_search_vnode,947692467956757361022701883714464770558143758336},undefined},{{riak_search_vnode,959110449498405040071168171470060731649205731328},undefined},{{riak_search_vnode,970528431040052719119634459225656692740267704320},undefined},{{riak_search_vnode,981946412581700398168100746981252653831329677312},undefined},{{riak_search_vnode,993364394123348077216567034736848614922391650304},undefined},{{riak_search_vnode,1004782375664995756265033322492444576013453623296},undefined},{{riak_search_vnode,1016200357206643435313499610248040537104515596288},undefined},{{riak_search_vnode,1027618338748291114361965898003636498195577569280},undefined},{{riak_search_vnode,1039036320289938793410432185759232459286639542272},undefined},{{riak_search_vnode,1050454301831586472458898473514828420377701515264},undefined},{{riak_search_vnode,1061872283373234151507364761270424381468763488256},undefined},{{riak_search_vnode,1073290264914881830555831049026020342559825461248},undefined},{{riak_search_vnode,1084708246456529509604297336781616303650887434240},undefined},{{riak_search_vnode,1096126227998177188652763624537212264741949407232},undefined},{{riak_search_vnode,1107544209539824867701229912292808225833011380224},undefined},{{riak_search_vnode,1118962191081472546749696200048404186924073353216},undefined},{{riak_search_vnode,1130380172623120225798162487804000148015135326208},undefined},{{riak_search_vnode,1141798154164767904846628775559596109106197299200},undefined},{{riak_search_vnode,1153216135706415583895095063315192070197259272192},undefined},{{riak_search_vnode,1164634117248063262943561351070788031288321245184},undefined},{{riak_search_vnode,1176052098789710941992027638826383992379383218176},undefined},{{riak_search_vnode,1187470080331358621040493926581979953470445191168},undefined},{{riak_search_vnode,1198888061873006300088960214337575914561507164160},undefined},{{riak_search_vnode,1210306043414653979137426502093171875652569137152},undefined},{{riak_search_vnode,1221724024956301658185892789848767836743631110144},undefined},{{riak_search_vnode,1233142006497949337234359077604363797834693083136},undefined},{{riak_search_vnode,1244559988039597016282825365359959758925755056128},undefined},{{riak_search_vnode,1255977969581244695331291653115555720016817029120},undefined},{{riak_search_vnode,1267395951122892374379757940871151681107879002112},undefined},{{riak_search_vnode,1278813932664540053428224228626747642198940975104},undefined},{{riak_search_vnode,1290231914206187732476690516382343603290002948096},undefined},{{riak_search_vnode,1301649895747835411525156804137939564381064921088},undefined},{{riak_search_vnode,1313067877289483090573623091893535525472126894080},undefined},{{riak_search_vnode,1324485858831130769622089379649131486563188867072},undefined},{{riak_search_vnode,1335903840372778448670555667404727447654250840064},undefined},{{riak_search_vnode,1347321821914426127719021955160323408745312813056},undefined},{{riak_search_vnode,1358739803456073806767488242915919369836374786048},undefined},{{riak_search_vnode,1370157784997721485815954530671515330927436759040},undefined},{{riak_search_vnode,1381575766539369164864420818427111292018498732032},undefined},{{riak_search_vnode,1392993748081016843912887106182707253109560705024},undefined},{{riak_search_vnode,1404411729622664522961353393938303214200622678016},undefined},{{riak_search_vnode,1415829711164312202009819681693899175291684651008},undefined},{{riak_search_vnode,1427247692705959881058285969449495136382746624000},undefined},{{riak_search_vnode,1438665674247607560106752257205091097473808596992},undefined},{{riak_search_vnode,1450083655789255239155218544960687058564870569984},undefined}],[]}
> ** Reason for termination ==
> ** {{badmatch,{error,{{badmatch,{error,emfile}},[{riak_kv_vnode,init,1},{riak_core_vnode,init,1},{gen_fsm,init_it,6},{proc_lib,init_p_do_apply,3}]}}},[{riak_core_vnode_manager,get_vnode,3},{riak_core_vnode_manager,handle_call,3},{gen_server,handle_msg,5},{proc_lib,init_p_do_apply,3}]}
> 2012-05-18 04:59:01 =CRASH REPORT====
>  crasher:
>    initial call: riak_core_vnode_manager:init/1
>    pid: <0.22403.463>
>    registered_name: riak_core_vnode_manager
>    exception exit: {{badmatch,{error,{{badmatch,{error,emfile}},[{riak_kv_vnode,init,1},{riak_core_vnode,init,1},{gen_fsm,init_it,6},{proc_lib,init_p_do_apply,3}]}}},[{riak_core_vnode_manager,get_vnode,3},{riak_core_vnode_manager,handle_call,3},{gen_server,handle_msg,5},{proc_lib,init_p_do_apply,3}]}
>      in function  gen_server:terminate/6
>      in call from proc_lib:init_p_do_apply/3
>    ancestors: [riak_core_sup,<0.91.0>]
>    messages: []
>    links: [<0.92.0>]
>    dictionary: []
>    trap_exit: false
>    status: running
>    heap_size: 17711
>    stack_size: 24
>    reductions: 170622
>  neighbours:
> 2012-05-18 04:59:01 =SUPERVISOR REPORT====
>     Supervisor: {local,riak_core_sup}
>     Context:    child_terminated
>     Reason:     {{badmatch,{error,{{badmatch,{error,emfile}},[{riak_kv_vnode,init,1},{riak_core_vnode,init,1},{gen_fsm,init_it,6},{proc_lib,init_p_do_apply,3}]}}},[{riak_core_vnode_manager,get_vnode,3},{riak_core_vnode_manager,handle_call,3},{gen_server,handle_msg,5},{proc_lib,init_p_do_apply,3}]}
>     Offender:   [{pid,<0.22403.463>},{name,riak_core_vnode_manager},{mfargs,{riak_core_vnode_manager,start_link,[]}},{restart_type,permanent},{shutdown,5000},{child_type,worker}]
>
> 2012-05-18 04:59:01 =CRASH REPORT====
>  crasher:
>    initial call: riak_core_vnode_proxy:init/1
>    pid: <0.21469.398>
>    registered_name: proxy_riak_kv_vnode_947692467956757361022701883714464770558143758336
>    exception exit: {{{badmatch,{error,{{badmatch,{error,emfile}},[{riak_kv_vnode,init,1},{riak_core_vnode,init,1},{gen_fsm,init_it,6},{proc_lib,init_p_do_apply,3}]}}},[{riak_core_vnode_manager,get_vnode,3},{riak_core_vnode_manager,handle_call,3},{gen_server,handle_msg,5},{proc_lib,init_p_do_apply,3}]},{gen_server,call,[riak_core_vnode_manager,{947692467956757361022701883714464770558143758336,riak_kv_vnode,get_vnode},infinity]}}
>      in function  gen_server:call/3
>      in call from riak_core_vnode_proxy:get_vnode_pid/1
>      in call from riak_core_vnode_proxy:handle_proxy/2
>      in call from riak_core_vnode_proxy:loop/2
>      in call from proc_lib:init_p_do_apply/3
>    ancestors: [riak_core_vnode_proxy_sup,riak_core_sup,<0.91.0>]
>    messages: []
>    links: [<0.20960.398>]
>    dictionary: []
>    trap_exit: false
>    status: running
>    heap_size: 377
>    stack_size: 24
>    reductions: 89
>  neighbours:
> 2012-05-18 04:59:01 =SUPERVISOR REPORT====
>     Supervisor: {local,riak_core_vnode_proxy_sup}
>     Context:    child_terminated
>     Reason:     {{{badmatch,{error,{{badmatch,{error,emfile}},[{riak_kv_vnode,init,1},{riak_core_vnode,init,1},{gen_fsm,init_it,6},{proc_lib,init_p_do_apply,3}]}}},[{riak_core_vnode_manager,get_vnode,3},{riak_core_vnode_manager,handle_call,3},{gen_server,handle_msg,5},{proc_lib,init_p_do_apply,3}]},{gen_server,call,[riak_core_vnode_manager,{947692467956757361022701883714464770558143758336,riak_kv_vnode,get_vnode},infinity]}}
>     Offender:   [{pid,<0.21469.398>},{name,{riak_kv_vnode,947692467956757361022701883714464770558143758336}},{mfargs,{riak_core_vnode_proxy,start_link,[riak_kv_vnode,947692467956757361022701883714464770558143758336]}},{restart_type,permanent},{shutdown,5000},{child_type,worker}]
>
> 2012-05-18 04:59:06 =ERROR REPORT====
> ** Generic server riak_kv_stat terminating
> ** Last message in was {'$gen_cast',{update,{put_fsm_time,732},63504536346}}
> ** When Server state == {state,{spiral,63504536346,[6,0,0,0,0,35,20,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]},{spiral,63504536346,[2,0,0,0,0,21,14,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]},61,37,{spiral,63504536341,[126,84,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]},210,{spiral,63504536346,[2,0,0,0,0,21,14,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]},37,{spiral,63504536346,[0,0,0,0,0,4,3,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]},7,{spiral,63504536340,[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]},0,{spiral,63504536346,[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]},0,12,8,{slide,63504536340,60,60,"/var/lib/riak/slide-data/3053/1337.331540.24720",{file_descriptor,prim_file,{#Port<0.2186439>,69}},63504536341},{slide,63504536340,60,60,"/var/lib/riak/slide-data/3053/1337.331540.25076",{file_descriptor,prim_file,{#Port<0.2186440>,128}},63504536341},{slide,63504536340,60,60,"/var/lib/riak/slide-data/3053/1337.331540.25785",{file_descriptor,prim_file,{#Port<0.2186444>,40}},63504536346},{slide,63504536340,60,60,"/var/lib/riak/slide-data/3053/1337.331540.26002",{file_descriptor,prim_file,{#Port<0.2186441>,177}},63504536341},{spiral,63504536340,[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]},0,0,{spiral,63504536340,[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]},0,undefined,0,0,undefined,undefined,0,0,true}
> ** Reason for termination ==
> ** {emfile,[{erlang,open_port,[{spawn,"/bin/sh -s unix:cmd 2>&1"},[stream]]},{os,start_port_srv_handle,1},{os,start_port_srv_loop,0}]}
> 2012-05-18 04:59:06 =CRASH REPORT====
>  crasher:
>    initial call: gen:init_it/7
>    pid: <0.22372.463>
>    registered_name: riak_kv_stat
>    exception exit: {emfile,[{erlang,open_port,[{spawn,"/bin/sh -s unix:cmd 2>&1"},[stream]]},{os,start_port_srv_handle,1},{os,start_port_srv_loop,0}]}
>      in function  gen_server2:terminate/6
>      in call from proc_lib:init_p_do_apply/3
>    ancestors: [riak_kv_sup,<0.323.0>]
>    messages: [{'$gen_cast',{update,{vnode_index_write,0,0},63504536346}},{'$gen_cast',{update,vnode_put,63504536346}},{'$gen_cast',{update,{vnode_index_write,1,0},63504536346}},{'$gen_cast',{update,vnode_put,63504536346}},{'$gen_cast',{update,{vnode_index_write,1,0},63504536346}},{'$gen_cast',{update,vnode_put,63504536346}},{'$gen_cast',{update,{vnode_index_write,1,0},63504536346}},{'$gen_cast',{update,vnode_put,63504536346}},{'$gen_cast',{update,{put_fsm_time,5241},63504536346}}]
>    links: [#Port<0.2186440>,#Port<0.2186444>,<0.420.0>,#Port<0.2186439>]
>    dictionary: []
>    trap_exit: true
>    status: running
>    heap_size: 6765
>    stack_size: 24
>    reductions: 23738
>  neighbours:
> 2012-05-18 04:59:06 =SUPERVISOR REPORT====
>     Supervisor: {local,riak_kv_sup}
>     Context:    child_terminated
>     Reason:     {emfile,[{erlang,open_port,[{spawn,"/bin/sh -s unix:cmd 2>&1"},[stream]]},{os,start_port_srv_handle,1},{os,start_port_srv_loop,0}]}
>     Offender:   [{pid,<0.22372.463>},{name,riak_kv_stat},{mfargs,{riak_kv_stat,start_link,[]}},{restart_type,permanent},{shutdown,5000},{child_type,worker}]
>
> 2012-05-18 04:59:06 =SUPERVISOR REPORT====
>     Supervisor: {local,riak_kv_sup}
>     Context:    start_error
>     Reason:     {emfile,[{erlang,open_port,[{spawn,"/bin/sh -s unix:cmd 2>&1"},[stream]]},{os,start_port_srv_handle,1},{os,start_port_srv_loop,0}]}
>     Offender:   [{pid,<0.22372.463>},{name,riak_kv_stat},{mfargs,{riak_kv_stat,start_link,[]}},{restart_type,permanent},{shutdown,5000},{child_type,worker}]
>
> 2012-05-18 04:59:06 =CRASH REPORT====
>  crasher:
>    initial call: gen:init_it/7
>    pid: <0.22460.463>
>    registered_name: []
>    exception exit: {emfile,[{erlang,open_port,[{spawn,"/bin/sh -s unix:cmd 2>&1"},[stream]]},{os,start_port_srv_handle,1},{os,start_port_srv_loop,0}]}
>      in function  gen_server2:init_it/6
>      in call from proc_lib:init_p_do_apply/3
>    ancestors: [riak_kv_sup,<0.323.0>]
>    messages: []
>    links: [<0.420.0>]
>    dictionary: []
>    trap_exit: true
>    status: running
>    heap_size: 610
>    stack_size: 24
>    reductions: 363
>  neighbours:
> 2012-05-18 04:59:06 =CRASH REPORT====
>  crasher:
>    initial call: gen:init_it/7
>    pid: <0.22462.463>
>    registered_name: []
>    exception exit: {emfile,[{erlang,open_port,[{spawn,"/bin/sh -s unix:cmd 2>&1"},[stream]]},{os,start_port_srv_handle,1},{os,start_port_srv_loop,0}]}
>      in function  gen_server2:init_it/6
>      in call from proc_lib:init_p_do_apply/3
>    ancestors: [riak_kv_sup,<0.323.0>]
>    messages: []
>    links: [<0.420.0>]
>    dictionary: []
>    trap_exit: true
>    status: running
>    heap_size: 610
>    stack_size: 24
>    reductions: 367
>  neighbours:
> 2012-05-18 04:59:06 =SUPERVISOR REPORT====
>     Supervisor: {local,riak_kv_sup}
>     Context:    start_error
>     Reason:     {emfile,[{erlang,open_port,[{spawn,"/bin/sh -s unix:cmd 2>&1"},[stream]]},{os,start_port_srv_handle,1},{os,start_port_srv_loop,0}]}
>     Offender:   [{pid,<0.22372.463>},{name,riak_kv_stat},{mfargs,{riak_kv_stat,start_link,[]}},{restart_type,permanent},{shutdown,5000},{child_type,worker}]
>
> 2012-05-18 04:59:06 =SUPERVISOR REPORT====
>     Supervisor: {local,riak_kv_sup}
>     Context:    start_error
>     Reason:     {emfile,[{erlang,open_port,[{spawn,"/bin/sh -s unix:cmd 2>&1"},[stream]]},{os,start_port_srv_handle,1},{os,start_port_srv_loop,0}]}
>     Offender:   [{pid,<0.22372.463>},{name,riak_kv_stat},{mfargs,{riak_kv_stat,start_link,[]}},{restart_type,permanent},{shutdown,5000},{child_type,worker}]
>
>
> Node 3:
>
> 2012-05-18 04:58:20 =ERROR REPORT====
> ** Generic server riak_core_vnode_manager terminating
> ** Last message in was {947692467956757361022701883714464770558143758336,riak_kv_vnode,get_vnode}
> ** When Server state == {state,40798507028,[{{riak_kv_vnode,0},undefined},{{riak_kv_vnode,11417981541647679048466287755595961091061972992},undefined},{{riak_kv_vnode,22835963083295358096932575511191922182123945984},undefined},{{riak_kv_vnode,34253944624943037145398863266787883273185918976},undefined},{{riak_kv_vnode,45671926166590716193865151022383844364247891968},undefined},{{riak_kv_vnode,57089907708238395242331438777979805455309864960},undefined},{{riak_kv_vnode,68507889249886074290797726533575766546371837952},undefined},{{riak_kv_vnode,79925870791533753339264014289171727637433810944},undefined},{{riak_kv_vnode,91343852333181432387730302044767688728495783936},undefined},{{riak_kv_vnode,102761833874829111436196589800363649819557756928},undefined},{{riak_kv_vnode,114179815416476790484662877555959610910619729920},undefined},{{riak_kv_vnode,125597796958124469533129165311555572001681702912},undefined},{{riak_kv_vnode,137015778499772148581595453067151533092743675904},undefined},{{riak_kv_vnode,148433760041419827630061740822747494183805648896},undefined},{{riak_kv_vnode,159851741583067506678528028578343455274867621888},undefined},{{riak_kv_vnode,171269723124715185726994316333939416365929594880},undefined},{{riak_kv_vnode,182687704666362864775460604089535377456991567872},undefined},{{riak_kv_vnode,194105686208010543823926891845131338548053540864},undefined},{{riak_kv_vnode,205523667749658222872393179600727299639115513856},undefined},{{riak_kv_vnode,216941649291305901920859467356323260730177486848},undefined},{{riak_kv_vnode,228359630832953580969325755111919221821239459840},undefined},{{riak_kv_vnode,239777612374601260017792042867515182912301432832},undefined},{{riak_kv_vnode,251195593916248939066258330623111144003363405824},undefined},{{riak_kv_vnode,262613575457896618114724618378707105094425378816},undefined},{{riak_kv_vnode,274031556999544297163190906134303066185487351808},undefined},{{riak_kv_vnode,285449538541191976211657193889899027276549324800},undefined},{{riak_kv_vnode,296867520082839655260123481645494988367611297792},undefined},{{riak_kv_vnode,308285501624487334308589769401090949458673270784},undefined},{{riak_kv_vnode,319703483166135013357056057156686910549735243776},undefined},{{riak_kv_vnode,331121464707782692405522344912282871640797216768},undefined},{{riak_kv_vnode,342539446249430371453988632667878832731859189760},undefined},{{riak_kv_vnode,353957427791078050502454920423474793822921162752},undefined},{{riak_kv_vnode,365375409332725729550921208179070754913983135744},undefined},{{riak_kv_vnode,376793390874373408599387495934666716005045108736},undefined},{{riak_kv_vnode,388211372416021087647853783690262677096107081728},undefined},{{riak_kv_vnode,399629353957668766696320071445858638187169054720},undefined},{{riak_kv_vnode,411047335499316445744786359201454599278231027712},undefined},{{riak_kv_vnode,422465317040964124793252646957050560369293000704},undefined},{{riak_kv_vnode,433883298582611803841718934712646521460354973696},undefined},{{riak_kv_vnode,445301280124259482890185222468242482551416946688},undefined},{{riak_kv_vnode,456719261665907161938651510223838443642478919680},undefined},{{riak_kv_vnode,468137243207554840987117797979434404733540892672},undefined},{{riak_kv_vnode,479555224749202520035584085735030365824602865664},undefined},{{riak_kv_vnode,490973206290850199084050373490626326915664838656},undefined},{{riak_kv_vnode,502391187832497878132516661246222288006726811648},undefined},{{riak_kv_vnode,513809169374145557180982949001818249097788784640},undefined},{{riak_kv_vnode,525227150915793236229449236757414210188850757632},undefined},{{riak_kv_vnode,536645132457440915277915524513010171279912730624},undefined},{{riak_kv_vnode,548063113999088594326381812268606132370974703616},undefined},{{riak_kv_vnode,559481095540736273374848100024202093462036676608},undefined},{{riak_kv_vnode,570899077082383952423314387779798054553098649600},undefined},{{riak_kv_vnode,582317058624031631471780675535394015644160622592},undefined},{{riak_kv_vnode,593735040165679310520246963290989976735222595584},undefined},{{riak_kv_vnode,605153021707326989568713251046585937826284568576},undefined},{{riak_kv_vnode,616571003248974668617179538802181898917346541568},undefined},{{riak_kv_vnode,627988984790622347665645826557777860008408514560},undefined},{{riak_kv_vnode,639406966332270026714112114313373821099470487552},undefined},{{riak_kv_vnode,650824947873917705762578402068969782190532460544},undefined},{{riak_kv_vnode,662242929415565384811044689824565743281594433536},undefined},{{riak_kv_vnode,673660910957213063859510977580161704372656406528},undefined},{{riak_kv_vnode,685078892498860742907977265335757665463718379520},undefined},{{riak_kv_vnode,696496874040508421956443553091353626554780352512},undefined},{{riak_kv_vnode,707914855582156101004909840846949587645842325504},undefined},{{riak_kv_vnode,719332837123803780053376128602545548736904298496},undefined},{{riak_kv_vnode,730750818665451459101842416358141509827966271488},undefined},{{riak_kv_vnode,742168800207099138150308704113737470919028244480},undefined},{{riak_kv_vnode,753586781748746817198774991869333432010090217472},undefined},{{riak_kv_vnode,765004763290394496247241279624929393101152190464},undefined},{{riak_kv_vnode,776422744832042175295707567380525354192214163456},undefined},{{riak_kv_vnode,787840726373689854344173855136121315283276136448},undefined},{{riak_kv_vnode,799258707915337533392640142891717276374338109440},undefined},{{riak_kv_vnode,810676689456985212441106430647313237465400082432},undefined},{{riak_kv_vnode,822094670998632891489572718402909198556462055424},undefined},{{riak_kv_vnode,833512652540280570538039006158505159647524028416},undefined},{{riak_kv_vnode,844930634081928249586505293914101120738586001408},undefined},{{riak_kv_vnode,856348615623575928634971581669697081829647974400},undefined},{{riak_kv_vnode,867766597165223607683437869425293042920709947392},undefined},{{riak_kv_vnode,879184578706871286731904157180889004011771920384},undefined},{{riak_kv_vnode,890602560248518965780370444936484965102833893376},undefined},{{riak_kv_vnode,902020541790166644828836732692080926193895866368},undefined},{{riak_kv_vnode,913438523331814323877303020447676887284957839360},undefined},{{riak_kv_vnode,924856504873462002925769308203272848376019812352},undefined},{{riak_kv_vnode,936274486415109681974235595958868809467081785344},undefined},{{riak_kv_vnode,947692467956757361022701883714464770558143758336},undefined},{{riak_kv_vnode,959110449498405040071168171470060731649205731328},undefined},{{riak_kv_vnode,970528431040052719119634459225656692740267704320},undefined},{{riak_kv_vnode,981946412581700398168100746981252653831329677312},undefined},{{riak_kv_vnode,993364394123348077216567034736848614922391650304},undefined},{{riak_kv_vnode,1004782375664995756265033322492444576013453623296},undefined},{{riak_kv_vnode,1016200357206643435313499610248040537104515596288},undefined},{{riak_kv_vnode,1027618338748291114361965898003636498195577569280},undefined},{{riak_kv_vnode,1039036320289938793410432185759232459286639542272},undefined},{{riak_kv_vnode,1050454301831586472458898473514828420377701515264},undefined},{{riak_kv_vnode,1061872283373234151507364761270424381468763488256},undefined},{{riak_kv_vnode,1073290264914881830555831049026020342559825461248},undefined},{{riak_kv_vnode,1084708246456529509604297336781616303650887434240},undefined},{{riak_kv_vnode,1096126227998177188652763624537212264741949407232},undefined},{{riak_kv_vnode,1107544209539824867701229912292808225833011380224},undefined},{{riak_kv_vnode,1118962191081472546749696200048404186924073353216},undefined},{{riak_kv_vnode,1130380172623120225798162487804000148015135326208},undefined},{{riak_kv_vnode,1141798154164767904846628775559596109106197299200},undefined},{{riak_kv_vnode,1153216135706415583895095063315192070197259272192},undefined},{{riak_kv_vnode,1164634117248063262943561351070788031288321245184},undefined},{{riak_kv_vnode,1176052098789710941992027638826383992379383218176},undefined},{{riak_kv_vnode,1187470080331358621040493926581979953470445191168},undefined},{{riak_kv_vnode,1198888061873006300088960214337575914561507164160},undefined},{{riak_kv_vnode,1210306043414653979137426502093171875652569137152},undefined},{{riak_kv_vnode,1221724024956301658185892789848767836743631110144},undefined},{{riak_kv_vnode,1233142006497949337234359077604363797834693083136},undefined},{{riak_kv_vnode,1244559988039597016282825365359959758925755056128},undefined},{{riak_kv_vnode,1255977969581244695331291653115555720016817029120},undefined},{{riak_kv_vnode,1267395951122892374379757940871151681107879002112},undefined},{{riak_kv_vnode,1278813932664540053428224228626747642198940975104},undefined},{{riak_kv_vnode,1290231914206187732476690516382343603290002948096},undefined},{{riak_kv_vnode,1301649895747835411525156804137939564381064921088},undefined},{{riak_kv_vnode,1313067877289483090573623091893535525472126894080},undefined},{{riak_kv_vnode,1324485858831130769622089379649131486563188867072},undefined},{{riak_kv_vnode,1335903840372778448670555667404727447654250840064},undefined},{{riak_kv_vnode,1347321821914426127719021955160323408745312813056},undefined},{{riak_kv_vnode,1358739803456073806767488242915919369836374786048},undefined},{{riak_kv_vnode,1370157784997721485815954530671515330927436759040},undefined},{{riak_kv_vnode,1381575766539369164864420818427111292018498732032},undefined},{{riak_kv_vnode,1392993748081016843912887106182707253109560705024},undefined},{{riak_kv_vnode,1404411729622664522961353393938303214200622678016},undefined},{{riak_kv_vnode,1415829711164312202009819681693899175291684651008},undefined},{{riak_kv_vnode,1427247692705959881058285969449495136382746624000},undefined},{{riak_kv_vnode,1438665674247607560106752257205091097473808596992},undefined},{{riak_kv_vnode,1450083655789255239155218544960687058564870569984},undefined},{{riak_pipe_vnode,0},undefined},{{riak_pipe_vnode,11417981541647679048466287755595961091061972992},undefined},{{riak_pipe_vnode,22835963083295358096932575511191922182123945984},undefined},{{riak_pipe_vnode,34253944624943037145398863266787883273185918976},undefined},{{riak_pipe_vnode,45671926166590716193865151022383844364247891968},undefined},{{riak_pipe_vnode,57089907708238395242331438777979805455309864960},undefined},{{riak_pipe_vnode,68507889249886074290797726533575766546371837952},undefined},{{riak_pipe_vnode,79925870791533753339264014289171727637433810944},undefined},{{riak_pipe_vnode,91343852333181432387730302044767688728495783936},undefined},{{riak_pipe_vnode,102761833874829111436196589800363649819557756928},undefined},{{riak_pipe_vnode,114179815416476790484662877555959610910619729920},undefined},{{riak_pipe_vnode,125597796958124469533129165311555572001681702912},undefined},{{riak_pipe_vnode,137015778499772148581595453067151533092743675904},undefined},{{riak_pipe_vnode,148433760041419827630061740822747494183805648896},undefined},{{riak_pipe_vnode,159851741583067506678528028578343455274867621888},undefined},{{riak_pipe_vnode,171269723124715185726994316333939416365929594880},undefined},{{riak_pipe_vnode,182687704666362864775460604089535377456991567872},undefined},{{riak_pipe_vnode,194105686208010543823926891845131338548053540864},undefined},{{riak_pipe_vnode,205523667749658222872393179600727299639115513856},undefined},{{riak_pipe_vnode,216941649291305901920859467356323260730177486848},undefined},{{riak_pipe_vnode,228359630832953580969325755111919221821239459840},undefined},{{riak_pipe_vnode,239777612374601260017792042867515182912301432832},undefined},{{riak_pipe_vnode,251195593916248939066258330623111144003363405824},undefined},{{riak_pipe_vnode,262613575457896618114724618378707105094425378816},undefined},{{riak_pipe_vnode,274031556999544297163190906134303066185487351808},undefined},{{riak_pipe_vnode,285449538541191976211657193889899027276549324800},undefined},{{riak_pipe_vnode,296867520082839655260123481645494988367611297792},undefined},{{riak_pipe_vnode,308285501624487334308589769401090949458673270784},undefined},{{riak_pipe_vnode,319703483166135013357056057156686910549735243776},undefined},{{riak_pipe_vnode,331121464707782692405522344912282871640797216768},undefined},{{riak_pipe_vnode,342539446249430371453988632667878832731859189760},undefined},{{riak_pipe_vnode,353957427791078050502454920423474793822921162752},undefined},{{riak_pipe_vnode,365375409332725729550921208179070754913983135744},undefined},{{riak_pipe_vnode,376793390874373408599387495934666716005045108736},undefined},{{riak_pipe_vnode,388211372416021087647853783690262677096107081728},undefined},{{riak_pipe_vnode,399629353957668766696320071445858638187169054720},undefined},{{riak_pipe_vnode,411047335499316445744786359201454599278231027712},undefined},{{riak_pipe_vnode,422465317040964124793252646957050560369293000704},undefined},{{riak_pipe_vnode,433883298582611803841718934712646521460354973696},undefined},{{riak_pipe_vnode,445301280124259482890185222468242482551416946688},undefined},{{riak_pipe_vnode,456719261665907161938651510223838443642478919680},undefined},{{riak_pipe_vnode,468137243207554840987117797979434404733540892672},undefined},{{riak_pipe_vnode,479555224749202520035584085735030365824602865664},undefined},{{riak_pipe_vnode,490973206290850199084050373490626326915664838656},undefined},{{riak_pipe_vnode,502391187832497878132516661246222288006726811648},undefined},{{riak_pipe_vnode,513809169374145557180982949001818249097788784640},undefined},{{riak_pipe_vnode,525227150915793236229449236757414210188850757632},undefined},{{riak_pipe_vnode,536645132457440915277915524513010171279912730624},undefined},{{riak_pipe_vnode,548063113999088594326381812268606132370974703616},undefined},{{riak_pipe_vnode,559481095540736273374848100024202093462036676608},undefined},{{riak_pipe_vnode,570899077082383952423314387779798054553098649600},undefined},{{riak_pipe_vnode,582317058624031631471780675535394015644160622592},undefined},{{riak_pipe_vnode,593735040165679310520246963290989976735222595584},undefined},{{riak_pipe_vnode,605153021707326989568713251046585937826284568576},undefined},{{riak_pipe_vnode,616571003248974668617179538802181898917346541568},undefined},{{riak_pipe_vnode,627988984790622347665645826557777860008408514560},undefined},{{riak_pipe_vnode,639406966332270026714112114313373821099470487552},undefined},{{riak_pipe_vnode,650824947873917705762578402068969782190532460544},undefined},{{riak_pipe_vnode,662242929415565384811044689824565743281594433536},undefined},{{riak_pipe_vnode,673660910957213063859510977580161704372656406528},undefined},{{riak_pipe_vnode,685078892498860742907977265335757665463718379520},undefined},{{riak_pipe_vnode,696496874040508421956443553091353626554780352512},undefined},{{riak_pipe_vnode,707914855582156101004909840846949587645842325504},undefined},{{riak_pipe_vnode,719332837123803780053376128602545548736904298496},undefined},{{riak_pipe_vnode,730750818665451459101842416358141509827966271488},undefined},{{riak_pipe_vnode,742168800207099138150308704113737470919028244480},undefined},{{riak_pipe_vnode,753586781748746817198774991869333432010090217472},undefined},{{riak_pipe_vnode,765004763290394496247241279624929393101152190464},undefined},{{riak_pipe_vnode,776422744832042175295707567380525354192214163456},undefined},{{riak_pipe_vnode,787840726373689854344173855136121315283276136448},undefined},{{riak_pipe_vnode,799258707915337533392640142891717276374338109440},undefined},{{riak_pipe_vnode,810676689456985212441106430647313237465400082432},undefined},{{riak_pipe_vnode,822094670998632891489572718402909198556462055424},undefined},{{riak_pipe_vnode,833512652540280570538039006158505159647524028416},undefined},{{riak_pipe_vnode,844930634081928249586505293914101120738586001408},undefined},{{riak_pipe_vnode,856348615623575928634971581669697081829647974400},undefined},{{riak_pipe_vnode,867766597165223607683437869425293042920709947392},undefined},{{riak_pipe_vnode,879184578706871286731904157180889004011771920384},undefined},{{riak_pipe_vnode,890602560248518965780370444936484965102833893376},undefined},{{riak_pipe_vnode,902020541790166644828836732692080926193895866368},undefined},{{riak_pipe_vnode,913438523331814323877303020447676887284957839360},undefined},{{riak_pipe_vnode,924856504873462002925769308203272848376019812352},undefined},{{riak_pipe_vnode,936274486415109681974235595958868809467081785344},undefined},{{riak_pipe_vnode,947692467956757361022701883714464770558143758336},undefined},{{riak_pipe_vnode,959110449498405040071168171470060731649205731328},undefined},{{riak_pipe_vnode,970528431040052719119634459225656692740267704320},undefined},{{riak_pipe_vnode,981946412581700398168100746981252653831329677312},undefined},{{riak_pipe_vnode,993364394123348077216567034736848614922391650304},undefined},{{riak_pipe_vnode,1004782375664995756265033322492444576013453623296},undefined},{{riak_pipe_vnode,1016200357206643435313499610248040537104515596288},undefined},{{riak_pipe_vnode,1027618338748291114361965898003636498195577569280},undefined},{{riak_pipe_vnode,1039036320289938793410432185759232459286639542272},undefined},{{riak_pipe_vnode,1050454301831586472458898473514828420377701515264},undefined},{{riak_pipe_vnode,1061872283373234151507364761270424381468763488256},undefined},{{riak_pipe_vnode,1073290264914881830555831049026020342559825461248},undefined},{{riak_pipe_vnode,1084708246456529509604297336781616303650887434240},undefined},{{riak_pipe_vnode,1096126227998177188652763624537212264741949407232},undefined},{{riak_pipe_vnode,1107544209539824867701229912292808225833011380224},undefined},{{riak_pipe_vnode,1118962191081472546749696200048404186924073353216},undefined},{{riak_pipe_vnode,1130380172623120225798162487804000148015135326208},undefined},{{riak_pipe_vnode,1141798154164767904846628775559596109106197299200},undefined},{{riak_pipe_vnode,1153216135706415583895095063315192070197259272192},undefined},{{riak_pipe_vnode,1164634117248063262943561351070788031288321245184},undefined},{{riak_pipe_vnode,1176052098789710941992027638826383992379383218176},undefined},{{riak_pipe_vnode,1187470080331358621040493926581979953470445191168},undefined},{{riak_pipe_vnode,1198888061873006300088960214337575914561507164160},undefined},{{riak_pipe_vnode,1210306043414653979137426502093171875652569137152},undefined},{{riak_pipe_vnode,1221724024956301658185892789848767836743631110144},undefined},{{riak_pipe_vnode,1233142006497949337234359077604363797834693083136},undefined},{{riak_pipe_vnode,1244559988039597016282825365359959758925755056128},undefined},{{riak_pipe_vnode,1255977969581244695331291653115555720016817029120},undefined},{{riak_pipe_vnode,1267395951122892374379757940871151681107879002112},undefined},{{riak_pipe_vnode,1278813932664540053428224228626747642198940975104},undefined},{{riak_pipe_vnode,1290231914206187732476690516382343603290002948096},undefined},{{riak_pipe_vnode,1301649895747835411525156804137939564381064921088},undefined},{{riak_pipe_vnode,1313067877289483090573623091893535525472126894080},undefined},{{riak_pipe_vnode,1324485858831130769622089379649131486563188867072},undefined},{{riak_pipe_vnode,1335903840372778448670555667404727447654250840064},undefined},{{riak_pipe_vnode,1347321821914426127719021955160323408745312813056},undefined},{{riak_pipe_vnode,1358739803456073806767488242915919369836374786048},undefined},{{riak_pipe_vnode,1370157784997721485815954530671515330927436759040},undefined},{{riak_pipe_vnode,1381575766539369164864420818427111292018498732032},undefined},{{riak_pipe_vnode,1392993748081016843912887106182707253109560705024},undefined},{{riak_pipe_vnode,1404411729622664522961353393938303214200622678016},undefined},{{riak_pipe_vnode,1415829711164312202009819681693899175291684651008},undefined},{{riak_pipe_vnode,1427247692705959881058285969449495136382746624000},undefined},{{riak_pipe_vnode,1438665674247607560106752257205091097473808596992},undefined},{{riak_pipe_vnode,1450083655789255239155218544960687058564870569984},undefined},{{riak_search_vnode,0},undefined},{{riak_search_vnode,11417981541647679048466287755595961091061972992},undefined},{{riak_search_vnode,22835963083295358096932575511191922182123945984},undefined},{{riak_search_vnode,34253944624943037145398863266787883273185918976},undefined},{{riak_search_vnode,45671926166590716193865151022383844364247891968},undefined},{{riak_search_vnode,57089907708238395242331438777979805455309864960},undefined},{{riak_search_vnode,68507889249886074290797726533575766546371837952},undefined},{{riak_search_vnode,79925870791533753339264014289171727637433810944},undefined},{{riak_search_vnode,91343852333181432387730302044767688728495783936},undefined},{{riak_search_vnode,102761833874829111436196589800363649819557756928},undefined},{{riak_search_vnode,114179815416476790484662877555959610910619729920},undefined},{{riak_search_vnode,125597796958124469533129165311555572001681702912},undefined},{{riak_search_vnode,137015778499772148581595453067151533092743675904},undefined},{{riak_search_vnode,148433760041419827630061740822747494183805648896},undefined},{{riak_search_vnode,159851741583067506678528028578343455274867621888},undefined},{{riak_search_vnode,171269723124715185726994316333939416365929594880},undefined},{{riak_search_vnode,182687704666362864775460604089535377456991567872},undefined},{{riak_search_vnode,194105686208010543823926891845131338548053540864},undefined},{{riak_search_vnode,205523667749658222872393179600727299639115513856},undefined},{{riak_search_vnode,216941649291305901920859467356323260730177486848},undefined},{{riak_search_vnode,228359630832953580969325755111919221821239459840},undefined},{{riak_search_vnode,239777612374601260017792042867515182912301432832},undefined},{{riak_search_vnode,251195593916248939066258330623111144003363405824},undefined},{{riak_search_vnode,262613575457896618114724618378707105094425378816},undefined},{{riak_search_vnode,274031556999544297163190906134303066185487351808},undefined},{{riak_search_vnode,285449538541191976211657193889899027276549324800},undefined},{{riak_search_vnode,296867520082839655260123481645494988367611297792},undefined},{{riak_search_vnode,308285501624487334308589769401090949458673270784},undefined},{{riak_search_vnode,319703483166135013357056057156686910549735243776},undefined},{{riak_search_vnode,331121464707782692405522344912282871640797216768},undefined},{{riak_search_vnode,342539446249430371453988632667878832731859189760},undefined},{{riak_search_vnode,353957427791078050502454920423474793822921162752},undefined},{{riak_search_vnode,365375409332725729550921208179070754913983135744},undefined},{{riak_search_vnode,376793390874373408599387495934666716005045108736},undefined},{{riak_search_vnode,388211372416021087647853783690262677096107081728},undefined},{{riak_search_vnode,399629353957668766696320071445858638187169054720},undefined},{{riak_search_vnode,411047335499316445744786359201454599278231027712},undefined},{{riak_search_vnode,422465317040964124793252646957050560369293000704},undefined},{{riak_search_vnode,433883298582611803841718934712646521460354973696},undefined},{{riak_search_vnode,445301280124259482890185222468242482551416946688},undefined},{{riak_search_vnode,456719261665907161938651510223838443642478919680},undefined},{{riak_search_vnode,468137243207554840987117797979434404733540892672},undefined},{{riak_search_vnode,479555224749202520035584085735030365824602865664},undefined},{{riak_search_vnode,490973206290850199084050373490626326915664838656},undefined},{{riak_search_vnode,502391187832497878132516661246222288006726811648},undefined},{{riak_search_vnode,513809169374145557180982949001818249097788784640},undefined},{{riak_search_vnode,525227150915793236229449236757414210188850757632},undefined},{{riak_search_vnode,536645132457440915277915524513010171279912730624},undefined},{{riak_search_vnode,548063113999088594326381812268606132370974703616},undefined},{{riak_search_vnode,559481095540736273374848100024202093462036676608},undefined},{{riak_search_vnode,570899077082383952423314387779798054553098649600},undefined},{{riak_search_vnode,582317058624031631471780675535394015644160622592},undefined},{{riak_search_vnode,593735040165679310520246963290989976735222595584},undefined},{{riak_search_vnode,605153021707326989568713251046585937826284568576},undefined},{{riak_search_vnode,616571003248974668617179538802181898917346541568},undefined},{{riak_search_vnode,627988984790622347665645826557777860008408514560},undefined},{{riak_search_vnode,639406966332270026714112114313373821099470487552},undefined},{{riak_search_vnode,650824947873917705762578402068969782190532460544},undefined},{{riak_search_vnode,662242929415565384811044689824565743281594433536},undefined},{{riak_search_vnode,673660910957213063859510977580161704372656406528},undefined},{{riak_search_vnode,685078892498860742907977265335757665463718379520},undefined},{{riak_search_vnode,696496874040508421956443553091353626554780352512},undefined},{{riak_search_vnode,707914855582156101004909840846949587645842325504},undefined},{{riak_search_vnode,719332837123803780053376128602545548736904298496},undefined},{{riak_search_vnode,730750818665451459101842416358141509827966271488},undefined},{{riak_search_vnode,742168800207099138150308704113737470919028244480},undefined},{{riak_search_vnode,753586781748746817198774991869333432010090217472},undefined},{{riak_search_vnode,765004763290394496247241279624929393101152190464},undefined},{{riak_search_vnode,776422744832042175295707567380525354192214163456},undefined},{{riak_search_vnode,787840726373689854344173855136121315283276136448},undefined},{{riak_search_vnode,799258707915337533392640142891717276374338109440},undefined},{{riak_search_vnode,810676689456985212441106430647313237465400082432},undefined},{{riak_search_vnode,822094670998632891489572718402909198556462055424},undefined},{{riak_search_vnode,833512652540280570538039006158505159647524028416},undefined},{{riak_search_vnode,844930634081928249586505293914101120738586001408},undefined},{{riak_search_vnode,856348615623575928634971581669697081829647974400},undefined},{{riak_search_vnode,867766597165223607683437869425293042920709947392},undefined},{{riak_search_vnode,879184578706871286731904157180889004011771920384},undefined},{{riak_search_vnode,890602560248518965780370444936484965102833893376},undefined},{{riak_search_vnode,902020541790166644828836732692080926193895866368},undefined},{{riak_search_vnode,913438523331814323877303020447676887284957839360},undefined},{{riak_search_vnode,924856504873462002925769308203272848376019812352},undefined},{{riak_search_vnode,936274486415109681974235595958868809467081785344},undefined},{{riak_search_vnode,947692467956757361022701883714464770558143758336},undefined},{{riak_search_vnode,959110449498405040071168171470060731649205731328},undefined},{{riak_search_vnode,970528431040052719119634459225656692740267704320},undefined},{{riak_search_vnode,981946412581700398168100746981252653831329677312},undefined},{{riak_search_vnode,993364394123348077216567034736848614922391650304},undefined},{{riak_search_vnode,1004782375664995756265033322492444576013453623296},undefined},{{riak_search_vnode,1016200357206643435313499610248040537104515596288},undefined},{{riak_search_vnode,1027618338748291114361965898003636498195577569280},undefined},{{riak_search_vnode,1039036320289938793410432185759232459286639542272},undefined},{{riak_search_vnode,1050454301831586472458898473514828420377701515264},undefined},{{riak_search_vnode,1061872283373234151507364761270424381468763488256},undefined},{{riak_search_vnode,1073290264914881830555831049026020342559825461248},undefined},{{riak_search_vnode,1084708246456529509604297336781616303650887434240},undefined},{{riak_search_vnode,1096126227998177188652763624537212264741949407232},undefined},{{riak_search_vnode,1107544209539824867701229912292808225833011380224},undefined},{{riak_search_vnode,1118962191081472546749696200048404186924073353216},undefined},{{riak_search_vnode,1130380172623120225798162487804000148015135326208},undefined},{{riak_search_vnode,1141798154164767904846628775559596109106197299200},undefined},{{riak_search_vnode,1153216135706415583895095063315192070197259272192},undefined},{{riak_search_vnode,1164634117248063262943561351070788031288321245184},undefined},{{riak_search_vnode,1176052098789710941992027638826383992379383218176},undefined},{{riak_search_vnode,1187470080331358621040493926581979953470445191168},undefined},{{riak_search_vnode,1198888061873006300088960214337575914561507164160},undefined},{{riak_search_vnode,1210306043414653979137426502093171875652569137152},undefined},{{riak_search_vnode,1221724024956301658185892789848767836743631110144},undefined},{{riak_search_vnode,1233142006497949337234359077604363797834693083136},undefined},{{riak_search_vnode,1244559988039597016282825365359959758925755056128},undefined},{{riak_search_vnode,1255977969581244695331291653115555720016817029120},undefined},{{riak_search_vnode,1267395951122892374379757940871151681107879002112},undefined},{{riak_search_vnode,1278813932664540053428224228626747642198940975104},undefined},{{riak_search_vnode,1290231914206187732476690516382343603290002948096},undefined},{{riak_search_vnode,1301649895747835411525156804137939564381064921088},undefined},{{riak_search_vnode,1313067877289483090573623091893535525472126894080},undefined},{{riak_search_vnode,1324485858831130769622089379649131486563188867072},undefined},{{riak_search_vnode,1335903840372778448670555667404727447654250840064},undefined},{{riak_search_vnode,1347321821914426127719021955160323408745312813056},undefined},{{riak_search_vnode,1358739803456073806767488242915919369836374786048},undefined},{{riak_search_vnode,1370157784997721485815954530671515330927436759040},undefined},{{riak_search_vnode,1381575766539369164864420818427111292018498732032},undefined},{{riak_search_vnode,1392993748081016843912887106182707253109560705024},undefined},{{riak_search_vnode,1404411729622664522961353393938303214200622678016},undefined},{{riak_search_vnode,1415829711164312202009819681693899175291684651008},undefined},{{riak_search_vnode,1427247692705959881058285969449495136382746624000},undefined},{{riak_search_vnode,1438665674247607560106752257205091097473808596992},undefined},{{riak_search_vnode,1450083655789255239155218544960687058564870569984},undefined}],[]}
> ** Reason for termination ==
> ** {{badmatch,{error,{{badmatch,{error,emfile}},[{riak_kv_vnode,init,1},{riak_core_vnode,init,1},{gen_fsm,init_it,6},{proc_lib,init_p_do_apply,3}]}}},[{riak_core_vnode_manager,get_vnode,3},{riak_core_vnode_manager,handle_call,3},{gen_server,handle_msg,5},{proc_lib,init_p_do_apply,3}]}
> 2012-05-18 04:58:20 =CRASH REPORT====
>  crasher:
>    initial call: riak_core_vnode_manager:init/1
>    pid: <0.15696.398>
>    registered_name: riak_core_vnode_manager
>    exception exit: {{badmatch,{error,{{badmatch,{error,emfile}},[{riak_kv_vnode,init,1},{riak_core_vnode,init,1},{gen_fsm,init_it,6},{proc_lib,init_p_do_apply,3}]}}},[{riak_core_vnode_manager,get_vnode,3},{riak_core_vnode_manager,handle_call,3},{gen_server,handle_msg,5},{proc_lib,init_p_do_apply,3}]}
>      in function  gen_server:terminate/6
>      in call from proc_lib:init_p_do_apply/3
>    ancestors: [riak_core_sup,<0.91.0>]
>    messages: []
>    links: [<0.92.0>]
>    dictionary: []
>    trap_exit: false
>    status: running
>    heap_size: 17711
>    stack_size: 24
>    reductions: 170595
>  neighbours:
> 2012-05-18 04:58:20 =CRASH REPORT====
>  crasher:
>    initial call: riak_core_vnode_proxy:init/1
>    pid: <0.15703.398>
>    registered_name: proxy_riak_kv_vnode_947692467956757361022701883714464770558143758336
>    exception exit: {{{badmatch,{error,{{badmatch,{error,emfile}},[{riak_kv_vnode,init,1},{riak_core_vnode,init,1},{gen_fsm,init_it,6},{proc_lib,init_p_do_apply,3}]}}},[{riak_core_vnode_manager,get_vnode,3},{riak_core_vnode_manager,handle_call,3},{gen_server,handle_msg,5},{proc_lib,init_p_do_apply,3}]},{gen_server,call,[riak_core_vnode_manager,{947692467956757361022701883714464770558143758336,riak_kv_vnode,get_vnode},infinity]}}
>      in function  gen_server:call/3
>      in call from riak_core_vnode_proxy:get_vnode_pid/1
>      in call from riak_core_vnode_proxy:handle_proxy/2
>      in call from riak_core_vnode_proxy:loop/2
>      in call from proc_lib:init_p_do_apply/3
>    ancestors: [riak_core_vnode_proxy_sup,riak_core_sup,<0.91.0>]
>    messages: []
>    links: [<0.107.0>]
>    dictionary: []
>    trap_exit: false
>    status: running
>    heap_size: 377
>    stack_size: 24
>    reductions: 89
>  neighbours:
> 2012-05-18 04:58:20 =SUPERVISOR REPORT====
>     Supervisor: {local,riak_core_sup}
>     Context:    child_terminated
>     Reason:     {{badmatch,{error,{{badmatch,{error,emfile}},[{riak_kv_vnode,init,1},{riak_core_vnode,init,1},{gen_fsm,init_it,6},{proc_lib,init_p_do_apply,3}]}}},[{riak_core_vnode_manager,get_vnode,3},{riak_core_vnode_manager,handle_call,3},{gen_server,handle_msg,5},{proc_lib,init_p_do_apply,3}]}
>     Offender:   [{pid,<0.15696.398>},{name,riak_core_vnode_manager},{mfargs,{riak_core_vnode_manager,start_link,[]}},{restart_type,permanent},{shutdown,5000},{child_type,worker}]
>
> 2012-05-18 04:58:20 =SUPERVISOR REPORT====
>     Supervisor: {local,riak_core_vnode_proxy_sup}
>     Context:    child_terminated
>     Reason:     {{{badmatch,{error,{{badmatch,{error,emfile}},[{riak_kv_vnode,init,1},{riak_core_vnode,init,1},{gen_fsm,init_it,6},{proc_lib,init_p_do_apply,3}]}}},[{riak_core_vnode_manager,get_vnode,3},{riak_core_vnode_manager,handle_call,3},{gen_server,handle_msg,5},{proc_lib,init_p_do_apply,3}]},{gen_server,call,[riak_core_vnode_manager,{947692467956757361022701883714464770558143758336,riak_kv_vnode,get_vnode},infinity]}}
>     Offender:   [{pid,<0.15703.398>},{name,{riak_kv_vnode,947692467956757361022701883714464770558143758336}},{mfargs,{riak_core_vnode_proxy,start_link,[riak_kv_vnode,947692467956757361022701883714464770558143758336]}},{restart_type,permanent},{shutdown,5000},{child_type,worker}]
>
> 2012-05-18 04:58:20 =SUPERVISOR REPORT====
>     Supervisor: {local,riak_core_vnode_proxy_sup}
>     Context:    shutdown
>     Reason:     reached_max_restart_intensity
>     Offender:   [{pid,<0.15703.398>},{name,{riak_kv_vnode,947692467956757361022701883714464770558143758336}},{mfargs,{riak_core_vnode_proxy,start_link,[riak_kv_vnode,947692467956757361022701883714464770558143758336]}},{restart_type,permanent},{shutdown,5000},{child_type,worker}]
>
> 2012-05-18 04:58:20 =ERROR REPORT====
> ** Generic server riak_kv_vnode_master terminating
> ** Last message in was {'$gen_cast',{riak_vnode_req_v1,947692467956757361022701883714464770558143758336,{fsm,undefined,<11602.10975.494>},{riak_kv_put_req_v1,{<<"_rsid_pages">>,<<"K4fUuIQqNsUHdH2rNqMzk6AKSot">>},{r_object,<<"_rsid_pages">>,<<"K4fUuIQqNsUHdH2rNqMzk6AKSot">>,[{r_content,{dict,3,16,16,8,80,48,{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},{{[],[],[],[],[],[],[],[],[],[],[[<<"X-Riak-VTag">>,53,49,103,55,99,86,83,98,79,82,83,73,110,68,56,84,109,118,55,89,71,72]],[[<<"index">>]],[],[[<<"X-Riak-Last-Modified">>|{1337,331480,246971}]],[],[]}}},{riak_idx_doc,<<"pages">>,<<"K4fUuIQqNsUHdH2rNqMzk6AKSot">>,[{<<"_type">>,<<"Page">>,[{<<"Page">>,[0]}]},{<<"classification">>,<<"0">>,[{<<"0000000000">>,[0]}]},{<<"contact_email">>,<<"david at serieaticketsonline.com<mailto:david at serieaticketsonline.com>">>,[{<<"david">>,[0]},{<<"serieaticketsonline.com<http://serieaticketsonline.com>">>,[1]}]},{<<"created_at">>,<<"2012-05-18">>,[{<<"2012-05-18">>,[0]}]},{<<"domain">>,<<"serieaticketsonline.com<http://serieaticketsonline.com>">>,[{<<"serieaticketsonline.com<http://serieaticketsonline.com>">>,[0]}]},{<<"outbound_links">>,<<"16">>,[{<<"0000000016">>,[0]}]},{<<"revaluate">>,<<"false">>,[]},{<<"search_terms">>,<<"44 999 This Day and Age Safetysuit Civil Twilight Reign of Kindo As Tall as Lions Copeland Alpha Rev Secondhand Serenade 38th Parallel Sent By Ravens Boys Like Girls Manic Drive P.O.D Ivoryline We Shot the Moon Deas Vail Neon Horse Grey Holiday Jon Mclaughlin Jonezetta Pillar Barcelona Mutemath Lydia Jonsi Matt Nathanson Rocket Summer Rapture Ruckus My Morning Jacket Doubledge Hey Monday Newworldson Above the Golden State Matt Wertz James Morrison VersaEmerge Thousand Foot Krutch The Drawing Room Manafest Jimmy Needham Waking Ashland Flyleaf Falling Up A Dream to the Moon Since October Fireflight Stereophonics The Fray Arcade Fire Mat Kearney Turn off the Stars Skillet Snow Patrol Shiny Toy Guns Lifehouse Children 18:3 Celldweller Article One HB Abandon Lost Ocean Staple Spoken Ever Stays Red This Beautiful Republic The Send Oceans Above Needtobreathe The Beatles Breaking Benjamin I See Stars Radiohead Muse AFI Alkaline Trio Pink Floyd U2 REM Blues Traveler Creedence Clearwater Revival Poets of the Fall Disciple Emarosa Evanescence Fall Out Boy Underoath The Killers Linkin Park My Chemical Romance Red Hot Chili Peppers Simon and Garfunkel Sixpence (None The Richer) System of a Down Weezer The Who Tokio Hotel Paramore Green Day The Cure Depeche Mode Black Veil Brides Korn Oomph! 3 Days Grace Escape the Fate The All-American Rejects All Time Low 30 Seconds to Mars Plain White T's Kings of Leon Silversun Pickups Nirvana Kerli David Bowie ACDC LED ZEPLIN BEATLES FOO FIGHTERS GUNS N ROSES PINK FLYOD THE JIMI HENDRIX EXPERIANCE BLACK SABBATH IRON MAIDEN THE ROLLING STONES K-ROCK AEROSMITH THE BLACK KEYS THE CLASH PEARL JAM THE ANIMALS THE DOORS BON JOVI IGGY AND THE STOOGES VAMPIRE WEEKEND DRAGON FORCE WARRANT QUEEN DEEP PURPLE THE RAMONES KISS THE POLICE COLDPLAY MOTORHEARDS black veiled brides misfits ten years trapt kid cudi neon trees silverstein velvet revolver hoobastank adema seether shinedown 3 doors down default whitesnake motley crue panic at the disco disterbed anberlin lovehatehero slipknot rise against ateyu Cream Tokyo Police Club MGMT Passion Pit Animal Collective Cocoon Death Cab for Cutie Def Leopard Dirty Projectors Elton John Keith Urban Black Eyed Peas Florence and the Machine The Postal Service Chromeo Freelance Whales Blind Pilot The Man Beach House Ghostland Observatory The Temper Trap Hall & Oates Led Zeppelin Santana Van Morrison Journey The Pet Shop Boys Lynayrd Skynard ZZ Top Sufjan Stevens The Welcome Wagon Lionel Richie Pablo Cruise Discovery Tears for Fears El Ten Eleven Prince Stevie Wonder Fleetwood Mac Bjork Tom Petty and the Heartbreakers Wolf Parade Shout Out Louds Yeah Yeah Yeahs Keane Matt & Kim She and Him Frans Ferdinand">>,[{<<"moon">>,[30,81]},{<<"journey">>,[354]},{<<"who">>,[173]},{<<"motley">>,[289]},{<<"club">>,[304]},{<<"needtobreathe">>,"z"},{<<"rolling">>,[231]},{<<"him">>,[401]},{<<"beatles">>,[124,215]},{<<"barcelona">>,")"},{<<"jimi">>,[223]},{<<"iron">>,[228]},{<<"hoobastank">>,[281]},{<<"fray">>,"W"},{<<"eyed">>,[324]},{<<"dirty">>,[317]},{<<"petty">>,[384]},{<<"staple">>,"n"},{<<"kerli">>,[209]},{<<"abandon">>,"k"},{<<"republic">>,"u"},{<<"mode">>,[182]},{<<"doubledge">>,"5"},{<<"john">>,[320]},{<<"spoken">>,"o"},{<<"bon">>,[246]},{<<"pilot">>,[337]},{<<"999">>,[0]},{<<"day">>,[2,178]},{<<"parallel">>,[18]},{<<"pillar">>,"("},{<<"santana">>,[351]},{<<"versaemerge">>,"A"},{<<"passion">>,[306]},{<<"skillet">>,"`"},{<<"whales">>,[335]},{<<"animal">>,[308]},{<<"house">>,[341]},{<<"observatory">>,[343]},{<<"ruckus">>,"2"},{<<"manafest">>,"H"},{<<"falling">>,"N"},{<<"beautiful">>,"t"},{<<"weezer">>,[171]},{<<"collective">>,[309]},{<<"cruise">>,[370]},{<<"sufjan">>,[362]},{<<"force">>,[255]},{<<"roses">>,[219]},{<<"ten">>,[271,375]},{<<"ever">>,"p"},{<<"tom">>,[383]},{<<"krutch">>,"D"},{<<"kings">>,[204]},{<<"blind">>,[336]},{<<"rem">>,[136]},{<<"clearwater">>,[140]},{<<"one">>,"j"},{<<"man">>,[339]},{<<"silverstein">>,[278]},{<<"dream">>,"O"},{<<"kearney">>,"["},{<<"jonezetta">>,"'"},{<<"cure">>,[180]},{<<"clash">>,[239]},{<<"shop">>,[357]},{<<"kid">>,[274]},{<<"stars">>,[95,128]},{<<"warrant">>,[256]},{<<"copeland">>,[12]},{<<"experiance">>,[225]},{<<"boys">>,[21,358]},{<<"stereophonics">>,"U"},{<<"trio">>,[133]},{<<"louds">>,[392]},{<<"weekend">>,[253]},{<<"animals">>,[243]},{<<"peas">>,[325]},{<<"tears">>,[372]},{<<"manic">>,[24]},{<<"down">>,[170,286]},{<<"maiden">>,[229]},{<<"misfits">>,[270]},{<<"wonder">>,[379]},{<<"summer">>,"0"},{<<"patrol">>,"b"},{<<"jonsi">>,","},{<<"stooges">>,[251]},{<<"shot">>,[28]},{<<"vail">>," "},{<<"out">>,[149,391]},{<<"simon">>,[162]},{<<"beach">>,[340]},{<<"shiny">>,"c"},{<<"default">>,[287]},{<<"oceans">>,"x"},{<<"low">>,[199]},{<<"police">>,[264,303]},{<<"sent">>,[19]},{<<"sabbath">>,[227]},{<<"cab">>,[312]},{<<"rejects">>,[196]},{<<"discovery">>,[371]},{<<"breaking">>,"}"},{<<"golden">>,";"},{<<"since">>,"R"},{<<"parade">>,[389]},{<<"pink">>,[134,220]},{<<"welcome">>,[365]},{<<"kindo">>,"\t"},{<<"death">>,[311]},{<<"postal">>,[331]},{<<"time">>,[198]},{<<"matt">>,[45,61,397]},{<<"snow">>,"a"},{<<"linkin">>,[154]},{<<"pickups">>,[207]},{<<"hendrix">>,[224]},{<<"disterbed">>,[294]},{<<"prince">>,[377]},{<<"mac">>,[381]},{<<"sixpence">>,[165]},{<<"garfunkel">>,[164]},{<<"green">>,[177]},{<<"queen">>,[257]},{<<"pablo">>,[369]},{<<"trapt">>,[273]},{<<"mutemath">>,"*"},{<<"rocket">>,"/"},{<<"underoath">>,[151]},{<<"ghostland">>,[342]},{<<"system">>,[169]},{<<"cudi">>,[275]},{<<"rise">>,[298]},{<<"led">>,[213,349]},{<<"state">>,"<"},{<<"room">>,"G"},{<<"romance">>,[157]},{<<"richer">>,[168]},{<<"rev">>,[14]},{<<"flyleaf">>,"M"},{<<"years">>,[272]},{<<"slipknot">>,[297]},{<<"cream">>,[301]},{<<"needham">>,"J"},{<<"twilight">>,[7]},{<<"article">>,"i"},{<<"evanescence">>,[147]},{<<"flyod">>,[221]},{<<"trees">>,[277]},{<<"anberlin">>,[295]},{<<"benjamin">>,"~"},{<<"service">>,[332]},{<<"thousand">>,"B"},{<<"shout">>,[390]},{<<"nathanson">>,"."},{<<"ashland">>,"L"},{<<"age">>,[4]},{<<"ocean">>,"m"},{<<"mat">>,"Z"},{<<"velvet">>,[279]},{<<"iggy">>,[248]},{<<"celldweller">>,"h"},{<<"leon">>,[205]},{<<"peppers">>,[161]},{<<"heartbreakers">>,[387]},{<<"paramore">>,[176]},{<<"frans">>,[402]},{<<"motorheards">>,[266]},{<<"ferdinand">>,[403]},{<<"monday">>,"7"},{<<"wertz">>,">"},{<<"escape">>,[190]},{<<"secondhand">>,[15]},{<<"jimmy">>,"I"},{<<"veil">>,[184]},{<<"toy">>,"d"},{<<"chromeo">>,[333]},{<<"wolf">>,[388]},{<<"james">>,"?"},{<<"top">>,[361]},{<<"turn">>,"\"},{<<"panic">>,[291]},{<<"cocoon">>,[310]},{<<"lifehouse">>,"f"},{<<"tokio">>,[174]},{<<"yeah">>,[393,394]},{<<"lovehatehero">>,[296]},{<<"kim">>,[398]},{<<"all">>,[194,197]},{<<"lydia">>,"+"},{<<"foo">>,[216]},{<<"disciple">>,[145]},{<<"vampire">>,[252]},{<<"lost">>,"l"},{<<"silversun">>,[206]},{<<"stevie">>,[378]},{<<"38th">>,[17]},{<<"horse">>,"""},{<<"deep">>,[258]},{<<"florence">>,[326]},{<<"nirvana">>,[208]},{<<"eleven">>,[376]},{<<"drive">>,[25]},{<<"zeppelin">>,[350]},{<<"emarosa">>,[146]},{<<"hall">>,[347]},{<<"bowie">>,[211]},{<<"above">>,"9y"},{<<"seconds">>,[200]},{<<"deas">>,[31]},{<<"arcade">>,"X"},{<<"plain">>,[202]},{<<"projectors">>,[318]},{<<"acdc">>,[212]},{<<"jovi">>,[247]},{<<"shinedown">>,[284]},{<<"traveler">>,[138]},{<<"alpha">>,"\r"},{<<"korn">>,[186]},{<<"alkaline">>,[132]},{<<"creedence">>,[139]},{<<"mars">>,[201]},{<<"like">>,[22]},{<<"temper">>,[345]},{<<"neon">>,[33,276]},{<<"fire">>,"Y"},{<<"cutie">>,[314]},{<<"park">>,[155]},{<<"against">>,[299]},{<<"she">>,[399]},{<<"boy">>,[150]},{<<"elton">>,[319]},{<<"wagon">>,[366]},{<<"purple">>,[259]},{<<"serenade">>,[16]},{<<"children">>,"g"},{<<"hotel">>,[175]},{<<"civil">>,[6]},{<<"jacket">>,"4"},{<<"ramones">>,[261]},{<<"oates">>,[348]},{<<"red">>,[114,158]},{<<"fears">>,[374]},{<<"revolver">>,[280]},{<<"david">>,[210]},{<<"jam">>,[241]},{<<"pit">>,[307]},{<<"keith">>,[321]},{<<"ivoryline">>,[27]},{<<"waking">>,"K"},{<<"richie">>,[368]},{<<"fall">>,[144,148]},{<<"killers">>,[153]},{<<"grace">>,[189]},{<<"skynard">>,[360]},{<<"p.o.d">>,[26]},{<<"chili">>,[160]},{<<"pearl">>,[240]},{<<"machine">>,[329]},{<<"adema">>,[282]},{<<"safetysuit">>,[5]},{<<"none">>,[166]},{<<"oomph">>,[187]},{<<"reign">>,[8]},{<<"october">>,"S"},{<<"def">>,[315]},{<<"girls">>,[23]},{<<"crue">>,[290]},{<<"grey">>,"#"},{<<"floyd">>,[135]},{<<"ravens">>,[20]},{<<"stones">>,[232]},{<<"tall">>,"\n"},{<<"van">>,[352]},{<<"drawing">>,"F"},{<<"urban">>,[322]},{<<"foot">>,"C"},{<<"veiled">>,[268]},{<<"days">>,[188]},{<<"kiss">>,[262]},{<<"stays">>,"q"},{<<"seether">>,[283]},{<<"hey">>,"6"},{<<"dragon">>,[254]},{<<"radiohead">>,[129]},{<<"lynayrd">>,[359]},{<<"brides">>,[185,269]},{<<"tokyo">>,[302]},{<<"morning">>,"3"},{<<"muse">>,[130]},{<<"pet">>,[356]},{<<"bjork">>,[382]},{<<"holiday">>,"$"},{<<"lionel">>,[367]},{<<"morrison">>,[64,353]},{<<"whitesnake">>,[288]},{<<"rock">>,[233]},{<<"disco">>,[293]},{<<"depeche">>,[181]},{<<"keane">>,[396]},{<<"fate">>,[192]},{<<"off">>,"]"},{<<"mgmt">>,[305]},{<<"blues">>,[137]},{<<"yeahs">>,[395]},{<<"newworldson">>,"8"},{<<"see">>,[127]},{<<"fleetwood">>,[380]},{<<"lions">>,[11]},{<<"white">>,[203]},{<<"coldplay">>,[265]},{<<"trap">>,[346]},{<<"revival">>,[141]},{<<"aerosmith">>,[234]},{<<"guns">>,[101,218]},{<<"american">>,[195]},{<<"ateyu">>,[300]},{<<"leopard">>,[316]},{<<"stevens">>,[363]},{<<"fighters">>,[217]},{<<"chemical">>,[156]},{<<"doors">>,[245,285]},{<<"jon">>,"%"},{<<"rapture">>,"1"},{<<"keys">>,[237]},{<<"send">>,"w"},{<<"afi">>,[131]},{<<"mclaughlin">>,"&"},{<<"freelance">>,[334]},{<<"poets">>,[142]},{<<"fireflight">>,"T"},{<<"hot">>,[159]},{<<"black">>,[183,226,236,267,323]},{<<"zeplin">>,[214]}]},{<<"updated_at">>,<<"2012-05-18">>,[{<<"2012-05-18">>,[0]}]},{<<"url">>,<<"http://www.serieaticketsonline.com/content-milan-players-looking-forward-for-second-leg-against-barcelona/">>,[{<<"barcelona">>,[11]},{<<"leg">>,"\t"},{<<"http">>,[0]},{<<"players">>,[4]},{<<"www.serieaticketsonline.com<http://www.serieaticketsonline.com>">>,[1]},{<<"second">>,[8]},{<<"content">>,[2]},{<<"looking">>,[5]},{<<"against">>,"\n"},{<<"milan">>,[3]},{<<"forward">>,[6]}]}],[],[],true}}],[{<<15,27,99,155,79,178,239,200>>,{3,63504550680}}],{dict,1,16,16,8,80,48,{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[[clean|true]],[]}}},undefined},13537270,63504550680,[]}}}
> ** When Server state == {state,undefined,undefined,riak_kv_vnode,riak_kv_legacy_vnode}
> ** Reason for termination ==
> ** {badarg,[{gen_fsm,send_event,2},{riak_core_vnode_master,handle_cast,2},{gen_server,handle_msg,5},{proc_lib,init_p_do_apply,3}]}
> 2012-05-18 04:58:20 =CRASH REPORT====
>  crasher:
>    initial call: riak_core_vnode_master:init/1
>    pid: <0.421.0>
>    registered_name: riak_kv_vnode_master
>    exception exit: {badarg,[{gen_fsm,send_event,2},{riak_core_vnode_master,handle_cast,2},{gen_server,handle_msg,5},{proc_lib,init_p_do_apply,3}]}
>      in function  gen_server:terminate/6
>      in call from proc_lib:init_p_do_apply/3
>    ancestors: [riak_kv_sup,<0.323.0>]
>    messages: []
>    links: [<0.420.0>]
>    dictionary: []
>    trap_exit: false
>    status: running
>    heap_size: 6765
>    stack_size: 24
>    reductions: 547963388
>  neighbours:
> 2012-05-18 04:58:20 =SUPERVISOR REPORT====
>     Supervisor: {local,riak_kv_sup}
>     Context:    child_terminated
>     Reason:     {badarg,[{gen_fsm,send_event,2},{riak_core_vnode_master,handle_cast,2},{gen_server,handle_msg,5},{proc_lib,init_p_do_apply,3}]}
>     Offender:   [{pid,<0.421.0>},{name,riak_kv_vnode_master},{mfargs,{riak_core_vnode_master,start_link,[riak_kv_vnode,riak_kv_legacy_vnode]}},{restart_type,permanent},{shutdown,5000},{child_type,worker}]
>
> 2012-05-18 04:58:20 =SUPERVISOR REPORT====
>     Supervisor: {local,riak_core_sup}
>     Context:    child_terminated
>     Reason:     shutdown
>     Offender:   [{pid,<0.107.0>},{name,riak_core_vnode_proxy_sup},{mfargs,{riak_core_vnode_proxy_sup,start_link,[]}},{restart_type,permanent},{shutdown,5000},{child_type,supervisor}]
>
> 2012-05-18 04:58:20 =ERROR REPORT====
> ** Generic server riak_kv_stat terminating
> ** Last message in was {'$gen_cast',{update,{get_fsm,undefined,279,1,1350},63504536300}}
> ** When Server state == {state,{spiral,63504536300,[11,0,0,0,0,0,0,0,0,0,0,18,0,0,0,3,17,0,0,0,0,0,0,0,0,0,23,0,0,0,9,12,0,0,0,14,0,0,0,0,0,3,5,0,0,0,2,6,0,0,0,0,0,0,0,0,0,30,0,0,16,27,33,0,0,0,0,0,0,0,0,0,0,0,0,0,0,8,6,0,0,14,0,0,0,0,0,7,22]},{spiral,63504536300,[6,0,0,0,0,0,0,0,0,0,0,12,0,0,0,3,10,0,0,0,0,0,0,0,0,0,16,0,0,0,6,7,0,0,0,7,0,0,0,0,0,3,2,0,0,0,2,2,0,0,0,0,0,0,0,0,0,18,0,0,11,18,22,0,0,0,0,0,0,0,0,0,0,0,0,0,0,4,6,0,0,7,0,0,0,0,0,7,12]},5887366,3601568,{spiral,63504536300,[42,0,0,0,0,0,0,0,0,0,0,42,0,0,0,42,42,0,0,0,0,0,0,0,0,0,42,0,0,0,42,84,0,0,0,84,0,0,0,0,0,42,42,0,0,0,42,42,0,0,0,0,0,0,0,0,0,42,0,0,0,84,42,0,0,0,0,0,0,0,0,42,0,0,0,0,0,84,0,0,0,84,0,0,0,0,0,42,168]},10079908,{spiral,63504536300,[6,0,0,0,0,0,0,0,0,0,0,12,0,0,0,3,10,0,0,0,0,0,0,0,0,0,16,0,0,0,6,7,0,0,0,7,0,0,0,0,0,3,2,0,0,0,2,2,0,0,0,0,0,0,0,0,0,18,0,0,11,18,22,0,0,0,0,0,0,0,0,0,0,0,0,0,0,4,6,0,0,7,0,0,0,0,0,7,12]},3601568,{spiral,63504536300,[3,0,0,0,0,0,0,0,0,0,0,3,0,0,0,0,1,0,0,0,0,0,0,0,0,0,3,0,0,0,0,3,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,4,0,0,0,3,3,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,7]},788755,{spiral,63504477129,[1,0,0,0,0,0,0,0,0,0,0,2,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,0,0,0,0,0,0,0,0,0,0,3,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,3,0]},32,{spiral,63504536300,[0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]},17236,2394509,1571512,{slide,63504471451,60,60,"/var/lib/riak/slide-data/3053/1337.266551.794838",{file_descriptor,prim_file,{#Port<0.2186351>,40}},63504536289},{slide,63504471451,60,60,"/var/lib/riak/slide-data/3053/1337.266551.794979",{file_descriptor,prim_file,{#Port<0.2186352>,128}},63504536289},{slide,63504471451,60,60,"/var/lib/riak/slide-data/3053/1337.266551.794984",{file_descriptor,prim_file,{#Port<0.2186349>,69}},63504536289},{slide,63504475275,60,60,"/var/lib/riak/slide-data/3053/1337.266551.794989",{file_descriptor,prim_file,{#Port<0.2186353>,177}},63504536289},{spiral,63504471351,[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]},0,0,{spiral,63504536285,[2,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1]},132,undefined,259259,0,undefined,undefined,0,0,true}
> ** Reason for termination ==
> ** {emfile,[{erlang,open_port,[{spawn,"/bin/sh -s unix:cmd 2>&1"},[stream]]},{os,start_port_srv_handle,1},{os,start_port_srv_loop,0}]}
> 2012-05-18 04:58:20 =CRASH REPORT====
>  crasher:
>    initial call: gen:init_it/7
>    pid: <0.426.0>
>    registered_name: riak_kv_stat
>    exception exit: {emfile,[{erlang,open_port,[{spawn,"/bin/sh -s unix:cmd 2>&1"},[stream]]},{os,start_port_srv_handle,1},{os,start_port_srv_loop,0}]}
>      in function  gen_server2:terminate/6
>      in call from proc_lib:init_p_do_apply/3
>    ancestors: [riak_kv_sup,<0.323.0>]
>    messages: [{'$gen_cast',{update,vnode_get,63504536300}},{'$gen_cast',{update,vnode_get,63504536300}},{'$gen_cast',{update,vnode_get,63504536300}},{'$gen_cast',{update,vnode_get,63504536300}},{'$gen_cast',{update,{get_fsm,undefined,337,1,1350},63504536300}},{'$gen_cast',{update,{vnode_index_write,0,0},63504536300}},{'$gen_cast',{update,vnode_put,63504536300}},{'$gen_cast',{update,{vnode_index_write,0,0},63504536300}},{'$gen_cast',{update,vnode_put,63504536300}},{'$gen_cast',{update,{put_fsm_time,603},63504536300}},{'$gen_cast',{update,{vnode_index_write,0,0},63504536300}},{'$gen_cast',{update,vnode_put,63504536300}},{'$gen_cast',{update,{vnode_index_write,0,0},63504536300}},{'$gen_cast',{update,vnode_put,63504536300}},{'$gen_cast',{update,{vnode_index_write,0,0},63504536300}},{'$gen_cast',{update,vnode_put,63504536300}},{'$gen_cast',{update,{put_fsm_time,13883},63504536300}},{'$gen_cast',{update,{vnode_index_write,0,0},63504536300}},{'$gen_cast',{update,vnode_put,63504536300}}]
>    links: [#Port<0.2186353>,<0.420.0>,#Port<0.2186351>,#Port<0.2186352>]
>    dictionary: []
>    trap_exit: true
>    status: running
>    heap_size: 6765
>    stack_size: 24
>    reductions: 1873973117
>  neighbours:
> 2012-05-18 04:58:20 =SUPERVISOR REPORT====
>     Supervisor: {local,riak_kv_sup}
>     Context:    child_terminated
>     Reason:     {emfile,[{erlang,open_port,[{spawn,"/bin/sh -s unix:cmd 2>&1"},[stream]]},{os,start_port_srv_handle,1},{os,start_port_srv_loop,0}]}
>     Offender:   [{pid,<0.426.0>},{name,riak_kv_stat},{mfargs,{riak_kv_stat,start_link,[]}},{restart_type,permanent},{shutdown,5000},{child_type,worker}]
>
>
> App Config:
>
> %% -*- mode: erlang;erlang-indent-level: 4;indent-tabs-mode: nil -*-
> %% ex: ft=erlang ts=4 sw=4 et
> [
> %% Riak Core config
> {riak_core, [
>              %% Default location of ringstate
>              {ring_state_dir, "/var/lib/riak/ring"},
>
>     %% The number of partitions to divide the hash space into
>     %% The default is 64. Set to a size to allow for growth as
>     %% this cannot be changed easily after a cluster is set up.
>     %% Each node should have at least 10 partitions assigned to it.
>     {ring_creation_size, 128},
>
>              %% http is a list of IP addresses and TCP ports that the Riak
>              %% HTTP interface will bind.
>              {http, [ {"10.38.24.196", 8098 } ]},
>
>              %% https is a list of IP addresses and TCP ports that the Riak
>              %% HTTPS interface will bind.
>              {https, [{"50.28.61.141", 8069 }]},
>
>              %% Default cert and key locations for https can be overridden
>              %% with the ssl config variable, for example:
>              {ssl, [
>                     {certfile, "/etc/riak/poseidon_cert.pem"},
>                     {keyfile, "/etc/riak/poseidon_key.pem"}
>                    ]},
>
>              %% riak_handoff_port is the TCP port that Riak uses for
>              %% intra-cluster data handoff.
>              {handoff_port, 8099 },
>
>              %% To encrypt riak_core intra-cluster data handoff traffic,
>              %% uncomment the following line and edit its path to an
>              %% appropriate certfile and keyfile.  (This example uses a
>              %% single file with both items concatenated together.)
>              %{handoff_ssl_options, [{certfile, "/tmp/erlserver.pem"}]},
>
>              %% Platform-specific installation paths (substituted by rebar)
>              {platform_bin_dir, "/usr/sbin"},
>              {platform_data_dir, "/var/lib/riak"},
>              {platform_etc_dir, "/etc/riak"},
>              {platform_lib_dir, "/usr/lib/riak"},
>              {platform_log_dir, "/var/log/riak"}
>             ]},
>
> %% Riak KV config
> {riak_kv, [
>            %% Storage_backend specifies the Erlang module defining the storage
>            %% mechanism that will be used on this node.
>            {storage_backend, riak_kv_eleveldb_backend},
>
>            %% pb_ip is the IP address that the Riak Protocol Buffers interface
>            %% will bind to.  If this is undefined, the interface will not run.
>            {pb_ip,   "127.0.0.1" },
>
>            %% pb_port is the TCP port that the Riak Protocol Buffers interface
>            %% will bind to
>            {pb_port, 8087 },
>
>            %% pb_backlog is the maximum length to which the queue of pending
>            %% connections may grow. If set, it must be an integer >= 0.
>            %% By default the value is 5. If you anticipate a huge number of
>            %% connections being initialised *simultaneously*, set this number
>            %% higher.
>            %% {pb_backlog, 64},
>
>            %% raw_name is the first part of all URLS used by the Riak raw HTTP
>            %% interface.  See riak_web.erl and raw_http_resource.erl for
>            %% details.
>            %{raw_name, "riak"},
>
>            %% mapred_name is URL used to submit map/reduce requests to Riak.
>            {mapred_name, "mapred"},
>
>            %% mapred_system indicates which version of the MapReduce
>            %% system should be used: 'pipe' means riak_pipe will
>            %% power MapReduce queries, while 'legacy' means that luke
>            %% will be used
>            {mapred_system, pipe},
>
>            %% mapred_2i_pipe indicates whether secondary-index
>            %% MapReduce inputs are queued in parallel via their own
>            %% pipe ('true'), or serially via a helper process
>            %% ('false' or undefined).  Set to 'false' or leave
>            %% undefined during a rolling upgrade from 1.0.
>            {mapred_2i_pipe, true},
>
>            %% directory used to store a transient queue for pending
>            %% map tasks
>            %% Only valid when mapred_system == legacy
>            %% {mapred_queue_dir, "/var/lib/riak/mr_queue" },
>
>            %% Each of the following entries control how many Javascript
>            %% virtual machines are available for executing map, reduce,
>            %% pre- and post-commit hook functions.
>            {map_js_vm_count, 8 },
>            {reduce_js_vm_count, 6 },
>            {hook_js_vm_count, 2 },
>
>            %% Number of items the mapper will fetch in one request.
>            %% Larger values can impact read/write performance for
>            %% non-MapReduce requests.
>            %% Only valid when mapred_system == legacy
>            %% {mapper_batch_size, 5},
>
>            %% js_max_vm_mem is the maximum amount of memory, in megabytes,
>            %% allocated to the Javascript VMs. If unset, the default is
>            %% 8MB.
>            {js_max_vm_mem, 8},
>
>            %% js_thread_stack is the maximum amount of thread stack, in megabyes,
>            %% allocate to the Javascript VMs. If unset, the default is 16MB.
>            %% NOTE: This is not the same as the C thread stack.
>            {js_thread_stack, 16},
>
>            %% Number of objects held in the MapReduce cache. These will be
>            %% ejected when the cache runs out of room or the bucket/key
>            %% pair for that entry changes
>            %% Only valid when mapred_system == legacy
>            %% {map_cache_size, 10000},
>
>            %% js_source_dir should point to a directory containing Javascript
>            %% source files which will be loaded by Riak when it initializes
>            %% Javascript VMs.
>            {js_source_dir, "/tmp/js_source"},
>
>            %% http_url_encoding determines how Riak treats URL encoded
>            %% buckets, keys, and links over the REST API. When set to 'on'
>            %% Riak always decodes encoded values sent as URLs and Headers.
>            %% Otherwise, Riak defaults to compatibility mode where links
>            %% are decoded, but buckets and keys are not. The compatibility
>            %% mode will be removed in a future release.
>            {http_url_encoding, on},
>
>            %% riak_stat enables the use of the "riak-admin status" command to
>            %% retrieve information the Riak node for performance and debugging needs
>            {riak_kv_stat, true},
>
>            %% When using riak_kv_stat, use the legacy routines for tracking
>            {legacy_stats, true},
>
>            %% Switch to vnode-based vclocks rather than client ids.  This
>            %% significantly reduces the number of vclock entries.
>            %% Only set true if *all* nodes in the cluster are upgraded to 1.0
>            {vnode_vclocks, true},
>
>            %% This option enables compatability of bucket and key listing
>            %% with 0.14 and earlier versions. Once a rolling upgrade to
>            %% a version > 0.14 is completed for a cluster, this should be
>            %% set to false for improved performance for bucket and key
>            %% listing operations.
>            {legacy_keylisting, false},
>
>            %% This option toggles compatibility of keylisting with 1.0
>            %% and earlier versions.  Once a rolling upgrade to a version
>            %% > 1.0 is completed for a cluster, this should be set to
>            %% true for better control of memory usage during key listing
>            %% operations
>            {listkeys_backpressure, true}
>           ]},
>
> %% Riak Search Config
> {riak_search, [
>                %% To enable Search functionality set this 'true'.
>                {enabled, true}
>               ]},
>
> %% Merge Index Config
> {merge_index, [
>                %% The root dir to store search merge_index data
>                {data_root, "/var/lib/riak/merge_index"},
>
>                %% The root dir to store secondary index merge_index data
>                {data_root_2i, "/var/lib/riak/merge_index_2i"},
>
>                %% Size, in bytes, of the in-memory buffer.  When this
>                %% threshold has been reached the data is transformed
>                %% into a segment file which resides on disk.
>                {buffer_rollover_size, 1048576},
>
>                %% Overtime the segment files need to be compacted.
>                %% This is the maximum number of segments that will be
>                %% compacted at once.  A lower value will lead to
>                %% quicker but more frequent compactions.
>                {max_compact_segments, 20}
>               ]},
>
> %% Bitcask Config
> {bitcask, [
>             {data_root, "/var/lib/riak/bitcask"}
>           ]},
>
> %% eLevelDB Config
> {eleveldb, [
>             {data_root, "/var/lib/riak/leveldb"}
>            ]},
>
> %% Lager Config
> {lager, [
>            %% What handlers to install with what arguments
>            %% The defaults for the logfiles are to rotate the files when
>            %% they reach 10Mb or at midnight, whichever comes first, and keep
>            %% the last 5 rotations. See the lager README for a description of
>            %% the time rotation format:
>            %% https://github.com/basho/lager/blob/master/README.org
>            %%
>            %% If you wish to disable rotation, you can either set the size to 0
>            %% and the rotation time to "", or instead specify a 2-tuple that only
>            %% consists of {Logfile, Level}.
>            {handlers, [
>                {lager_console_backend, info},
>                {lager_file_backend, [
>                    {"/var/log/riak/error.log", error, 10485760, "$D0", 5},
>                    {"/var/log/riak/console.log", info, 10485760, "$D0", 5}
>                ]}
>            ]},
>
>            %% Whether to write a crash log, and where.
>            %% Commented/omitted/undefined means no crash logger.
>            {crash_log, "/var/log/riak/crash.log"},
>
>            %% Maximum size in bytes of events in the crash log - defaults to 65536
>            {crash_log_msg_size, 65536},
>
>            %% Maximum size of the crash log in bytes, before its rotated, set
>            %% to 0 to disable rotation - default is 0
>            {crash_log_size, 10485760},
>
>            %% What time to rotate the crash log - default is no time
>            %% rotation. See the lager README for a description of this format:
>            %% https://github.com/basho/lager/blob/master/README.org
>            {crash_log_date, "$D0"},
>
>            %% Number of rotated crash logs to keep, 0 means keep only the
>            %% current one - default is 0
>            {crash_log_count, 5},
>
>            %% Whether to redirect error_logger messages into lager - defaults to true
>            {error_logger_redirect, true}
>        ]},
>
> %% riak_sysmon config
> {riak_sysmon, [
>         %% To disable forwarding events of a particular type, use a
>         %% limit of 0.
>         {process_limit, 30},
>         {port_limit, 2},
>
>         %% Finding reasonable limits for a given workload is a matter
>         %% of experimentation.
>         {gc_ms_limit, 100},
>         {heap_word_limit, 40111000},
>
>         %% Configure the following items to 'false' to disable logging
>         %% of that event type.
>         {busy_port, true},
>         {busy_dist_port, true}
>        ]},
>
> %% SASL config
> {sasl, [
>         {sasl_error_logger, false}
>        ]},
>
> %% riak_control config
> {riak_control, [
>                %% Set to false to disable the admin panel.
>                {enabled, true},
>
>                %% Authentication style used for access to the admin
>                %% panel. Valid styles are 'userlist' <TODO>.
>                {auth, userlist},
>
>                %% If auth is set to 'userlist' then this is the
>                %% list of usernames and passwords for access to the
>                %% admin panel.
>                {userlist, [{"Aquaman", "At1ant1s"}
>                           ]},
>
>                %% The admin panel is broken up into multiple
>                %% components, each of which is enabled or disabled
>                %% by one of these settings.
>                {admin, true}
>                ]}
> ].
>
> Hopefully this is helpful. Thanks Dave.
>
> -Gustav
>
> On Tue, May 22, 2012 at 6:35 AM, Dave Smith <dizzyd at basho.com<mailto:dizzyd at basho.com>> wrote:
> Hi Gustav,
>
> Can you share the logs or any other information from the server that crashes _first_? It might be good to also include app.config, just so we can double-check that no other settings have changed.
>
> Thanks,
>
> D.
>
> --
> Dave Smith
> VP, Engineering
> Basho Technologies, Inc.
> dizzyd at basho.com<mailto:dizzyd at basho.com>
>
>
>
> On Friday, May 18, 2012 at 10:03 AM, Gustav Gennrich wrote:
>
>> Hi,
>>
>> Below is our setup to better diagnose:
>>
>> Ring creation size: 128
>> Backend: LevelDB
>>
>> All other settings in app.config are using the default values
>>
>> We have 3 servers (nodes) each of which have 8GB of ram.
>>
>> Question 1: We are attempting to load data into Riak using CSVs that we generated from data previously stored is our MySQL database. We are using Ripple to create the documents and associations (links). The problem we are having is Riak continue to grow in memory consumption until it finally crashed while loading this data. We are only able to add around a couple hundred thousand rows from the CSV's before this happens. Riak then responds with 500 errors and eventually all of the nodes crash and the nodes need to be restarted.
>>
>> Question 2 (unrelated): I am attempting to use some of the already defined js functions in ripple.js in a mapreduce, however, I keep getting the following error:
>>
>> Riak::MapReduceError: {"phase":0,"error":"[{<<\"lineno\">>,1},{<<\"message\">>,<<\"ReferenceError: Ripple is not defined\">>},{<<\"source\">>,<<\"unknown\">>}]"
>>
>> In ripple.yml I have the following (***** represent hidden values):
>>
>> # Configure Riak connections for the Ripple library.
>> development:
>> # http_port: 8091
>> # pb_port: 8087
>> # host: 127.0.0.1
>> basic_auth: *****
>> http_port: 80
>> pb_port: 8087
>> host: *****
>> http_backend: Excon
>> solr: "/solr"
>> js_source_dir: <%= Rails.root + "app/mapreduce" %>
>>
>> # The test environment has additional keys for configuring the
>> # Riak::TestServer for your test/spec suite:
>> #
>> # * bin_dir specifies the path to the "riak" script that you use to
>> # start Riak (just the directory)
>> # * js_source_dir specifies where your custom Javascript functions for
>> # MapReduce should be loaded from. Usually app/mapreduce.
>> test:
>> http_port: 9000
>> pb_port: 9002
>> host: localhost
>>
>> source: /usr/local/bin # Default for Homebrew.
>> js_source_dir: <%= Rails.root + "/app/mapreduce" %>
>>
>> production:
>> nodes:
>> - host: *****
>> http_port: 8098
>> pb_port: 8087
>> - host: *****
>> http_port: 8098
>> pb_port: 8087
>> - host: *****
>> http_port: 8098
>> pb_port: 8087
>> http_backend: Excon
>> solr: "/solr"
>> js_source_dir: <%= Rails.root + "app/mapreduce" %>
>>
>>
>> I don't see why I am getting this error when I have defined where the js_source_dir is.
>>
>> Any help would be greatly appreciated!
>>
>> Thanks,
>>
>> Gustav
>> _______________________________________________
>> riak-users mailing list
>> riak-users at lists.basho.com<mailto:riak-users at lists.basho.com> (mailto:riak-users at lists.basho.com<mailto:riak-users at lists.basho.com>)
>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
>
>
>
> _______________________________________________
> riak-users mailing list
> riak-users at lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com




More information about the riak-users mailing list