1. Riak Crashing on Data Load from CSV and 2. How to get Ripple to load js functions

Gustav Gennrich ggennrich at gmail.com
Fri May 25 14:18:08 EDT 2012


Hi,

I allowed the data loading to run a couple days to see if things changed
after increasing the ulimit to 5120 and this is what I noticed:

- The memory consumption increases at a much lower rate than before
- The memory consumption is still never released, though

It appears that I still have the same problem but its just happening at a
slower rate due to the ulimit change. Any other ideas why Riak is not
releasing memory during data load?

Thanks,

Gustav

On Tue, May 22, 2012 at 10:26 AM, Gustav Gennrich <ggennrich at gmail.com>wrote:

> Hi,
>
> Thanks for the replies. The 'ulimit -n' prints out 1024. Executing 'lsof |
> wc -l' outputs 2884 so I am already way over the limit set by the system. I
> am going to increase the ulimit value to 4096. However, based on my
> understanding this value needs to be set in system conf files as setting it
> via the 'ulimit' command will only apply it to the current user for the
> current session which isn't helpful. Let me make these changes and see if
> this resolves things.
>
> On Tue, May 22, 2012 at 10:02 AM, Kresten Krab Thorup <krab at trifork.com>wrote:
>
>> It looks like you have run out of file descriptors. The crash is an
>> emfile.
>>
>> What does 'ulimit -n' print?
>>
>> Kresten
>>
>> On 22/05/2012, at 18.24, "Gustav Gennrich" <ggennrich at gmail.com> wrote:
>>
>> > Hi Dave,
>> >
>> > This is some output from the crash.log file on the nodes. app.config
>> file is at the end:
>> >
>> > Node 1 (Hosting Riak Control):
>> >
>> > 2012-05-18 04:58:25 =ERROR REPORT====
>> >
>> {mochiweb_socket_server,310,{acceptor_error,{error,accept_failed}}}2012-05-18
>> 04:58:25 =ERROR REPORT====
>> >
>> {mochiweb_socket_server,310,{acceptor_error,{error,accept_failed}}}2012-05-18
>> 04:58:25 =ERROR REPORT====
>> >
>> {mochiweb_socket_server,310,{acceptor_error,{error,accept_failed}}}2012-05-18
>> 04:58:25 =ERROR REPORT====
>> >
>> {mochiweb_socket_server,310,{acceptor_error,{error,accept_failed}}}2012-05-18
>> 04:58:26 =ERROR REPORT====
>> >
>> {mochiweb_socket_server,310,{acceptor_error,{error,accept_failed}}}2012-05-18
>> 04:58:26 =ERROR REPORT====
>> >
>> {mochiweb_socket_server,310,{acceptor_error,{error,accept_failed}}}2012-05-18
>> 04:58:26 =ERROR REPORT====
>> >
>> {mochiweb_socket_server,310,{acceptor_error,{error,accept_failed}}}2012-05-18
>> 04:58:26 =ERROR REPORT====
>> >
>> {mochiweb_socket_server,310,{acceptor_error,{error,accept_failed}}}2012-05-18
>> 04:58:26 =ERROR REPORT====
>> >
>> {mochiweb_socket_server,310,{acceptor_error,{error,accept_failed}}}2012-05-18
>> 04:58:26 =ERROR REPORT====
>> >
>> {mochiweb_socket_server,310,{acceptor_error,{error,accept_failed}}}2012-05-18
>> 04:58:26 =ERROR REPORT====
>> >
>> {mochiweb_socket_server,310,{acceptor_error,{error,accept_failed}}}2012-05-18
>> 04:58:26 =ERROR REPORT====
>> >
>> {mochiweb_socket_server,310,{acceptor_error,{error,accept_failed}}}2012-05-18
>> 04:58:26 =ERROR REPORT====
>> >
>> {mochiweb_socket_server,310,{acceptor_error,{error,accept_failed}}}2012-05-18
>> 04:58:26 =ERROR REPORT====
>> >
>> {mochiweb_socket_server,310,{acceptor_error,{error,accept_failed}}}2012-05-18
>> 04:58:27 =ERROR REPORT====
>> >
>> {mochiweb_socket_server,310,{acceptor_error,{error,accept_failed}}}2012-05-18
>> 04:59:00 =ERROR REPORT====
>> > ** Generic server riak_kv_stat terminating
>> > ** Last message in was
>> {'$gen_cast',{update,{get_fsm,undefined,1158,1,4559},63504536340}}
>> > ** When Server state ==
>> {state,{spiral,63504536340,[1,0,0,0,0,0,0,0,0,0,0,0,0,0,14,0,0,0,0,0,0,0,0,0,0,0,8,0,0,11,0,0,0,0,0,14,0,0,0,17,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]},{spiral,63504536326,[7,0,0,0,0,0,0,0,0,0,0,0,4,0,0,7,0,0,0,0,0,9,0,0,0,11,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]},65,38,{spiral,63504536326,[84,0,0,0,0,0,0,0,0,0,0,0,84,0,0,84,0,0,0,0,0,126,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]},378,{spiral,63504536326,[7,0,0,0,0,0,0,0,0,0,0,0,4,0,0,7,0,0,0,0,0,9,0,0,0,11,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]},38,{spiral,63504536326,[1,0,0,0,0,0,0,0,0,0,0,0,1,0,0,1,0,0,0,0,0,4,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]},7,{spiral,63504536301,[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]},0,{spiral,63504536326,[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]},1,29,18,{slide,63504536301,60,60,"/var/lib/riak/slide-data/3053/1337.331501.51152",{file_descriptor,prim_file,{#Port<0.2186418>,128}},63504536326},{slide,63504536301,60,60,"/var/lib/riak/slide-data/3053/1337.331501.51714",{file_descriptor,prim_file,{#Port<0.2186419>,177}},63504536326},{slide,63504536301,60,60,"/var/lib/riak/slide-data/3053/1337.331501.52049",{file_descriptor,prim_file,{#Port<0.2186417>,69}},63504536326},{slide,63504536301,60,60,"/var/lib/riak/slide-data/3053/1337.331501.52247",{file_descriptor,prim_file,{#Port<0.2186420>,40}},63504536326},{spiral,63504536301,[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]},0,0,{spiral,63504536301,[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]},0,undefined,0,0,undefined,undefined,0,0,true}
>> > ** Reason for termination ==
>> > ** {emfile,[{erlang,open_port,[{spawn,"/bin/sh -s unix:cmd
>> 2>&1"},[stream]]},{os,start_port_srv_handle,1},{os,start_port_srv_loop,0}]}
>> > 2012-05-18 04:59:00 =CRASH REPORT====
>> >  crasher:
>> >    initial call: gen:init_it/7
>> >    pid: <0.31225.398>
>> >    registered_name: riak_kv_stat
>> >    exception exit: {emfile,[{erlang,open_port,[{spawn,"/bin/sh -s
>> unix:cmd
>> 2>&1"},[stream]]},{os,start_port_srv_handle,1},{os,start_port_srv_loop,0}]}
>> >      in function  gen_server2:terminate/6
>> >      in call from proc_lib:init_p_do_apply/3
>> >    ancestors: [riak_kv_sup,<0.323.0>]
>> >    messages: []
>> >    links: [#Port<0.2186419>,#Port<0.2186420>,<0.420.0>,#Port<0.2186418>]
>> >    dictionary: []
>> >    trap_exit: true
>> >    status: running
>> >    heap_size: 377
>> >    stack_size: 24
>> >    reductions: 41467
>> >  neighbours:
>> > 2012-05-18 04:59:00 =SUPERVISOR REPORT====
>> >     Supervisor: {local,riak_kv_sup}
>> >     Context:    child_terminated
>> >     Reason:     {emfile,[{erlang,open_port,[{spawn,"/bin/sh -s unix:cmd
>> 2>&1"},[stream]]},{os,start_port_srv_handle,1},{os,start_port_srv_loop,0}]}
>> >     Offender:
>> [{pid,<0.31225.398>},{name,riak_kv_stat},{mfargs,{riak_kv_stat,start_link,[]}},{restart_type,permanent},{shutdown,5000},{child_type,worker}]
>> >
>> > 2012-05-18 04:59:00 =CRASH REPORT====
>> >  crasher:
>> >    initial call: riak_core_vnode:init/1
>> >    pid: <0.22390.463>
>> >    registered_name: []
>> >    exception exit:
>> {{badmatch,{error,emfile}},[{riak_kv_vnode,init,1},{riak_core_vnode,init,1},{gen_fsm,init_it,6},{proc_lib,init_p_do_apply,3}]}
>> >      in function  gen_fsm:init_it/6
>> >      in call from proc_lib:init_p_do_apply/3
>> >    ancestors: [riak_core_vnode_sup,riak_core_sup,<0.91.0>]
>> >    messages: []
>> >    links: [<0.95.0>]
>> >    dictionary: []
>> >    trap_exit: true
>> >    status: running
>> >    heap_size: 987
>> >    stack_size: 24
>> >    reductions: 2734
>> >  neighbours:
>> > 2012-05-18 04:59:00 =SUPERVISOR REPORT====
>> >     Supervisor: {local,riak_core_vnode_sup}
>> >     Context:    child_terminated
>> >     Reason:
>> {{badmatch,{error,emfile}},[{riak_kv_vnode,init,1},{riak_core_vnode,init,1},{gen_fsm,init_it,6},{proc_lib,init_p_do_apply,3}]}
>> >     Offender:
>> [{pid,<0.22390.463>},{name,undefined},{mfargs,{riak_core_vnode,start_link,undefined}},{restart_type,temporary},{shutdown,300000},{child_type,worker}]
>> >
>> > 2012-05-18 04:59:00 =ERROR REPORT====
>> > ** Generic server riak_core_vnode_manager terminating
>> > ** Last message in was
>> {137015778499772148581595453067151533092743675904,riak_kv_vnode,get_vnode}
>> > ** When Server state ==
>> {state,40798511124,[{{riak_kv_vnode,0},undefined},{{riak_kv_vnode,11417981541647679048466287755595961091061972992},undefined},{{riak_kv_vnode,22835963083295358096932575511191922182123945984},undefined},{{riak_kv_vnode,34253944624943037145398863266787883273185918976},undefined},{{riak_kv_vnode,45671926166590716193865151022383844364247891968},undefined},{{riak_kv_vnode,57089907708238395242331438777979805455309864960},undefined},{{riak_kv_vnode,68507889249886074290797726533575766546371837952},undefined},{{riak_kv_vnode,79925870791533753339264014289171727637433810944},undefined},{{riak_kv_vnode,91343852333181432387730302044767688728495783936},undefined},{{riak_kv_vnode,102761833874829111436196589800363649819557756928},undefined},{{riak_kv_vnode,114179815416476790484662877555959610910619729920},undefined},{{riak_kv_vnode,125597796958124469533129165311555572001681702912},undefined},{{riak_kv_vnode,137015778499772148581595453067151533092743675904},undefined},{{riak_kv_vnode,148433760041419827630061740822747494183805648896},undefined},{{riak_kv_vnode,159851741583067506678528028578343455274867621888},undefined},{{riak_kv_vnode,171269723124715185726994316333939416365929594880},undefined},{{riak_kv_vnode,182687704666362864775460604089535377456991567872},undefined},{{riak_kv_vnode,194105686208010543823926891845131338548053540864},undefined},{{riak_kv_vnode,205523667749658222872393179600727299639115513856},undefined},{{riak_kv_vnode,216941649291305901920859467356323260730177486848},undefined},{{riak_kv_vnode,228359630832953580969325755111919221821239459840},undefined},{{riak_kv_vnode,239777612374601260017792042867515182912301432832},undefined},{{riak_kv_vnode,251195593916248939066258330623111144003363405824},undefined},{{riak_kv_vnode,262613575457896618114724618378707105094425378816},undefined},{{riak_kv_vnode,274031556999544297163190906134303066185487351808},undefined},{{riak_kv_vnode,285449538541191976211657193889899027276549324800},undefined},{{riak_kv_vnode,296867520082839655260123481645494988367611297792},undefined},{{riak_kv_vnode,308285501624487334308589769401090949458673270784},undefined},{{riak_kv_vnode,319703483166135013357056057156686910549735243776},undefined},{{riak_kv_vnode,331121464707782692405522344912282871640797216768},undefined},{{riak_kv_vnode,342539446249430371453988632667878832731859189760},undefined},{{riak_kv_vnode,353957427791078050502454920423474793822921162752},undefined},{{riak_kv_vnode,365375409332725729550921208179070754913983135744},undefined},{{riak_kv_vnode,376793390874373408599387495934666716005045108736},undefined},{{riak_kv_vnode,388211372416021087647853783690262677096107081728},undefined},{{riak_kv_vnode,399629353957668766696320071445858638187169054720},undefined},{{riak_kv_vnode,411047335499316445744786359201454599278231027712},undefined},{{riak_kv_vnode,422465317040964124793252646957050560369293000704},undefined},{{riak_kv_vnode,433883298582611803841718934712646521460354973696},undefined},{{riak_kv_vnode,445301280124259482890185222468242482551416946688},undefined},{{riak_kv_vnode,456719261665907161938651510223838443642478919680},undefined},{{riak_kv_vnode,468137243207554840987117797979434404733540892672},undefined},{{riak_kv_vnode,479555224749202520035584085735030365824602865664},undefined},{{riak_kv_vnode,490973206290850199084050373490626326915664838656},undefined},{{riak_kv_vnode,502391187832497878132516661246222288006726811648},undefined},{{riak_kv_vnode,513809169374145557180982949001818249097788784640},undefined},{{riak_kv_vnode,525227150915793236229449236757414210188850757632},undefined},{{riak_kv_vnode,536645132457440915277915524513010171279912730624},undefined},{{riak_kv_vnode,548063113999088594326381812268606132370974703616},undefined},{{riak_kv_vnode,559481095540736273374848100024202093462036676608},undefined},{{riak_kv_vnode,570899077082383952423314387779798054553098649600},undefined},{{riak_kv_vnode,582317058624031631471780675535394015644160622592},undefined},{{riak_kv_vnode,593735040165679310520246963290989976735222595584},undefined},{{riak_kv_vnode,605153021707326989568713251046585937826284568576},undefined},{{riak_kv_vnode,616571003248974668617179538802181898917346541568},undefined},{{riak_kv_vnode,627988984790622347665645826557777860008408514560},undefined},{{riak_kv_vnode,639406966332270026714112114313373821099470487552},undefined},{{riak_kv_vnode,650824947873917705762578402068969782190532460544},undefined},{{riak_kv_vnode,662242929415565384811044689824565743281594433536},undefined},{{riak_kv_vnode,673660910957213063859510977580161704372656406528},undefined},{{riak_kv_vnode,685078892498860742907977265335757665463718379520},undefined},{{riak_kv_vnode,696496874040508421956443553091353626554780352512},undefined},{{riak_kv_vnode,707914855582156101004909840846949587645842325504},undefined},{{riak_kv_vnode,719332837123803780053376128602545548736904298496},undefined},{{riak_kv_vnode,730750818665451459101842416358141509827966271488},undefined},{{riak_kv_vnode,742168800207099138150308704113737470919028244480},undefined},{{riak_kv_vnode,753586781748746817198774991869333432010090217472},undefined},{{riak_kv_vnode,765004763290394496247241279624929393101152190464},undefined},{{riak_kv_vnode,776422744832042175295707567380525354192214163456},undefined},{{riak_kv_vnode,787840726373689854344173855136121315283276136448},undefined},{{riak_kv_vnode,799258707915337533392640142891717276374338109440},undefined},{{riak_kv_vnode,810676689456985212441106430647313237465400082432},undefined},{{riak_kv_vnode,822094670998632891489572718402909198556462055424},undefined},{{riak_kv_vnode,833512652540280570538039006158505159647524028416},undefined},{{riak_kv_vnode,844930634081928249586505293914101120738586001408},undefined},{{riak_kv_vnode,856348615623575928634971581669697081829647974400},undefined},{{riak_kv_vnode,867766597165223607683437869425293042920709947392},undefined},{{riak_kv_vnode,879184578706871286731904157180889004011771920384},undefined},{{riak_kv_vnode,890602560248518965780370444936484965102833893376},undefined},{{riak_kv_vnode,902020541790166644828836732692080926193895866368},undefined},{{riak_kv_vnode,913438523331814323877303020447676887284957839360},undefined},{{riak_kv_vnode,924856504873462002925769308203272848376019812352},undefined},{{riak_kv_vnode,936274486415109681974235595958868809467081785344},undefined},{{riak_kv_vnode,947692467956757361022701883714464770558143758336},undefined},{{riak_kv_vnode,959110449498405040071168171470060731649205731328},undefined},{{riak_kv_vnode,970528431040052719119634459225656692740267704320},undefined},{{riak_kv_vnode,981946412581700398168100746981252653831329677312},undefined},{{riak_kv_vnode,993364394123348077216567034736848614922391650304},undefined},{{riak_kv_vnode,1004782375664995756265033322492444576013453623296},undefined},{{riak_kv_vnode,1016200357206643435313499610248040537104515596288},undefined},{{riak_kv_vnode,1027618338748291114361965898003636498195577569280},undefined},{{riak_kv_vnode,1039036320289938793410432185759232459286639542272},undefined},{{riak_kv_vnode,1050454301831586472458898473514828420377701515264},undefined},{{riak_kv_vnode,1061872283373234151507364761270424381468763488256},undefined},{{riak_kv_vnode,1073290264914881830555831049026020342559825461248},undefined},{{riak_kv_vnode,1084708246456529509604297336781616303650887434240},undefined},{{riak_kv_vnode,1096126227998177188652763624537212264741949407232},undefined},{{riak_kv_vnode,1107544209539824867701229912292808225833011380224},undefined},{{riak_kv_vnode,1118962191081472546749696200048404186924073353216},undefined},{{riak_kv_vnode,1130380172623120225798162487804000148015135326208},undefined},{{riak_kv_vnode,1141798154164767904846628775559596109106197299200},undefined},{{riak_kv_vnode,1153216135706415583895095063315192070197259272192},undefined},{{riak_kv_vnode,1164634117248063262943561351070788031288321245184},undefined},{{riak_kv_vnode,1176052098789710941992027638826383992379383218176},undefined},{{riak_kv_vnode,1187470080331358621040493926581979953470445191168},undefined},{{riak_kv_vnode,1198888061873006300088960214337575914561507164160},undefined},{{riak_kv_vnode,1210306043414653979137426502093171875652569137152},undefined},{{riak_kv_vnode,1221724024956301658185892789848767836743631110144},undefined},{{riak_kv_vnode,1233142006497949337234359077604363797834693083136},undefined},{{riak_kv_vnode,1244559988039597016282825365359959758925755056128},undefined},{{riak_kv_vnode,1255977969581244695331291653115555720016817029120},undefined},{{riak_kv_vnode,1267395951122892374379757940871151681107879002112},undefined},{{riak_kv_vnode,1278813932664540053428224228626747642198940975104},undefined},{{riak_kv_vnode,1290231914206187732476690516382343603290002948096},undefined},{{riak_kv_vnode,1301649895747835411525156804137939564381064921088},undefined},{{riak_kv_vnode,1313067877289483090573623091893535525472126894080},undefined},{{riak_kv_vnode,1324485858831130769622089379649131486563188867072},undefined},{{riak_kv_vnode,1335903840372778448670555667404727447654250840064},undefined},{{riak_kv_vnode,1347321821914426127719021955160323408745312813056},undefined},{{riak_kv_vnode,1358739803456073806767488242915919369836374786048},undefined},{{riak_kv_vnode,1370157784997721485815954530671515330927436759040},undefined},{{riak_kv_vnode,1381575766539369164864420818427111292018498732032},undefined},{{riak_kv_vnode,1392993748081016843912887106182707253109560705024},undefined},{{riak_kv_vnode,1404411729622664522961353393938303214200622678016},undefined},{{riak_kv_vnode,1415829711164312202009819681693899175291684651008},undefined},{{riak_kv_vnode,1427247692705959881058285969449495136382746624000},undefined},{{riak_kv_vnode,1438665674247607560106752257205091097473808596992},undefined},{{riak_kv_vnode,1450083655789255239155218544960687058564870569984},undefined},{{riak_pipe_vnode,0},undefined},{{riak_pipe_vnode,11417981541647679048466287755595961091061972992},undefined},{{riak_pipe_vnode,22835963083295358096932575511191922182123945984},undefined},{{riak_pipe_vnode,34253944624943037145398863266787883273185918976},undefined},{{riak_pipe_vnode,45671926166590716193865151022383844364247891968},undefined},{{riak_pipe_vnode,57089907708238395242331438777979805455309864960},undefined},{{riak_pipe_vnode,68507889249886074290797726533575766546371837952},undefined},{{riak_pipe_vnode,79925870791533753339264014289171727637433810944},undefined},{{riak_pipe_vnode,91343852333181432387730302044767688728495783936},undefined},{{riak_pipe_vnode,102761833874829111436196589800363649819557756928},undefined},{{riak_pipe_vnode,114179815416476790484662877555959610910619729920},undefined},{{riak_pipe_vnode,125597796958124469533129165311555572001681702912},undefined},{{riak_pipe_vnode,137015778499772148581595453067151533092743675904},undefined},{{riak_pipe_vnode,148433760041419827630061740822747494183805648896},undefined},{{riak_pipe_vnode,159851741583067506678528028578343455274867621888},undefined},{{riak_pipe_vnode,171269723124715185726994316333939416365929594880},undefined},{{riak_pipe_vnode,182687704666362864775460604089535377456991567872},undefined},{{riak_pipe_vnode,194105686208010543823926891845131338548053540864},undefined},{{riak_pipe_vnode,205523667749658222872393179600727299639115513856},undefined},{{riak_pipe_vnode,216941649291305901920859467356323260730177486848},undefined},{{riak_pipe_vnode,228359630832953580969325755111919221821239459840},undefined},{{riak_pipe_vnode,239777612374601260017792042867515182912301432832},undefined},{{riak_pipe_vnode,251195593916248939066258330623111144003363405824},undefined},{{riak_pipe_vnode,262613575457896618114724618378707105094425378816},undefined},{{riak_pipe_vnode,274031556999544297163190906134303066185487351808},undefined},{{riak_pipe_vnode,285449538541191976211657193889899027276549324800},undefined},{{riak_pipe_vnode,296867520082839655260123481645494988367611297792},undefined},{{riak_pipe_vnode,308285501624487334308589769401090949458673270784},undefined},{{riak_pipe_vnode,319703483166135013357056057156686910549735243776},undefined},{{riak_pipe_vnode,331121464707782692405522344912282871640797216768},undefined},{{riak_pipe_vnode,342539446249430371453988632667878832731859189760},undefined},{{riak_pipe_vnode,353957427791078050502454920423474793822921162752},undefined},{{riak_pipe_vnode,365375409332725729550921208179070754913983135744},undefined},{{riak_pipe_vnode,376793390874373408599387495934666716005045108736},undefined},{{riak_pipe_vnode,388211372416021087647853783690262677096107081728},undefined},{{riak_pipe_vnode,399629353957668766696320071445858638187169054720},undefined},{{riak_pipe_vnode,411047335499316445744786359201454599278231027712},undefined},{{riak_pipe_vnode,422465317040964124793252646957050560369293000704},undefined},{{riak_pipe_vnode,433883298582611803841718934712646521460354973696},undefined},{{riak_pipe_vnode,445301280124259482890185222468242482551416946688},undefined},{{riak_pipe_vnode,456719261665907161938651510223838443642478919680},undefined},{{riak_pipe_vnode,468137243207554840987117797979434404733540892672},undefined},{{riak_pipe_vnode,479555224749202520035584085735030365824602865664},undefined},{{riak_pipe_vnode,490973206290850199084050373490626326915664838656},undefined},{{riak_pipe_vnode,502391187832497878132516661246222288006726811648},undefined},{{riak_pipe_vnode,513809169374145557180982949001818249097788784640},undefined},{{riak_pipe_vnode,525227150915793236229449236757414210188850757632},undefined},{{riak_pipe_vnode,536645132457440915277915524513010171279912730624},undefined},{{riak_pipe_vnode,548063113999088594326381812268606132370974703616},undefined},{{riak_pipe_vnode,559481095540736273374848100024202093462036676608},undefined},{{riak_pipe_vnode,570899077082383952423314387779798054553098649600},undefined},{{riak_pipe_vnode,582317058624031631471780675535394015644160622592},undefined},{{riak_pipe_vnode,593735040165679310520246963290989976735222595584},undefined},{{riak_pipe_vnode,605153021707326989568713251046585937826284568576},undefined},{{riak_pipe_vnode,616571003248974668617179538802181898917346541568},undefined},{{riak_pipe_vnode,627988984790622347665645826557777860008408514560},undefined},{{riak_pipe_vnode,639406966332270026714112114313373821099470487552},undefined},{{riak_pipe_vnode,650824947873917705762578402068969782190532460544},undefined},{{riak_pipe_vnode,662242929415565384811044689824565743281594433536},undefined},{{riak_pipe_vnode,673660910957213063859510977580161704372656406528},undefined},{{riak_pipe_vnode,685078892498860742907977265335757665463718379520},undefined},{{riak_pipe_vnode,696496874040508421956443553091353626554780352512},undefined},{{riak_pipe_vnode,707914855582156101004909840846949587645842325504},undefined},{{riak_pipe_vnode,719332837123803780053376128602545548736904298496},undefined},{{riak_pipe_vnode,730750818665451459101842416358141509827966271488},undefined},{{riak_pipe_vnode,742168800207099138150308704113737470919028244480},undefined},{{riak_pipe_vnode,753586781748746817198774991869333432010090217472},undefined},{{riak_pipe_vnode,765004763290394496247241279624929393101152190464},undefined},{{riak_pipe_vnode,776422744832042175295707567380525354192214163456},undefined},{{riak_pipe_vnode,787840726373689854344173855136121315283276136448},undefined},{{riak_pipe_vnode,799258707915337533392640142891717276374338109440},undefined},{{riak_pipe_vnode,810676689456985212441106430647313237465400082432},undefined},{{riak_pipe_vnode,822094670998632891489572718402909198556462055424},undefined},{{riak_pipe_vnode,833512652540280570538039006158505159647524028416},undefined},{{riak_pipe_vnode,844930634081928249586505293914101120738586001408},undefined},{{riak_pipe_vnode,856348615623575928634971581669697081829647974400},undefined},{{riak_pipe_vnode,867766597165223607683437869425293042920709947392},undefined},{{riak_pipe_vnode,879184578706871286731904157180889004011771920384},undefined},{{riak_pipe_vnode,890602560248518965780370444936484965102833893376},undefined},{{riak_pipe_vnode,902020541790166644828836732692080926193895866368},undefined},{{riak_pipe_vnode,913438523331814323877303020447676887284957839360},undefined},{{riak_pipe_vnode,924856504873462002925769308203272848376019812352},undefined},{{riak_pipe_vnode,936274486415109681974235595958868809467081785344},undefined},{{riak_pipe_vnode,947692467956757361022701883714464770558143758336},undefined},{{riak_pipe_vnode,959110449498405040071168171470060731649205731328},undefined},{{riak_pipe_vnode,970528431040052719119634459225656692740267704320},undefined},{{riak_pipe_vnode,981946412581700398168100746981252653831329677312},undefined},{{riak_pipe_vnode,993364394123348077216567034736848614922391650304},undefined},{{riak_pipe_vnode,1004782375664995756265033322492444576013453623296},undefined},{{riak_pipe_vnode,1016200357206643435313499610248040537104515596288},undefined},{{riak_pipe_vnode,1027618338748291114361965898003636498195577569280},undefined},{{riak_pipe_vnode,1039036320289938793410432185759232459286639542272},undefined},{{riak_pipe_vnode,1050454301831586472458898473514828420377701515264},undefined},{{riak_pipe_vnode,1061872283373234151507364761270424381468763488256},undefined},{{riak_pipe_vnode,1073290264914881830555831049026020342559825461248},undefined},{{riak_pipe_vnode,1084708246456529509604297336781616303650887434240},undefined},{{riak_pipe_vnode,1096126227998177188652763624537212264741949407232},undefined},{{riak_pipe_vnode,1107544209539824867701229912292808225833011380224},undefined},{{riak_pipe_vnode,1118962191081472546749696200048404186924073353216},undefined},{{riak_pipe_vnode,1130380172623120225798162487804000148015135326208},undefined},{{riak_pipe_vnode,1141798154164767904846628775559596109106197299200},undefined},{{riak_pipe_vnode,1153216135706415583895095063315192070197259272192},undefined},{{riak_pipe_vnode,1164634117248063262943561351070788031288321245184},undefined},{{riak_pipe_vnode,1176052098789710941992027638826383992379383218176},undefined},{{riak_pipe_vnode,1187470080331358621040493926581979953470445191168},undefined},{{riak_pipe_vnode,1198888061873006300088960214337575914561507164160},undefined},{{riak_pipe_vnode,1210306043414653979137426502093171875652569137152},undefined},{{riak_pipe_vnode,1221724024956301658185892789848767836743631110144},undefined},{{riak_pipe_vnode,1233142006497949337234359077604363797834693083136},undefined},{{riak_pipe_vnode,1244559988039597016282825365359959758925755056128},undefined},{{riak_pipe_vnode,1255977969581244695331291653115555720016817029120},undefined},{{riak_pipe_vnode,1267395951122892374379757940871151681107879002112},undefined},{{riak_pipe_vnode,1278813932664540053428224228626747642198940975104},undefined},{{riak_pipe_vnode,1290231914206187732476690516382343603290002948096},undefined},{{riak_pipe_vnode,1301649895747835411525156804137939564381064921088},undefined},{{riak_pipe_vnode,1313067877289483090573623091893535525472126894080},undefined},{{riak_pipe_vnode,1324485858831130769622089379649131486563188867072},undefined},{{riak_pipe_vnode,1335903840372778448670555667404727447654250840064},undefined},{{riak_pipe_vnode,1347321821914426127719021955160323408745312813056},undefined},{{riak_pipe_vnode,1358739803456073806767488242915919369836374786048},undefined},{{riak_pipe_vnode,1370157784997721485815954530671515330927436759040},undefined},{{riak_pipe_vnode,1381575766539369164864420818427111292018498732032},undefined},{{riak_pipe_vnode,1392993748081016843912887106182707253109560705024},undefined},{{riak_pipe_vnode,1404411729622664522961353393938303214200622678016},undefined},{{riak_pipe_vnode,1415829711164312202009819681693899175291684651008},undefined},{{riak_pipe_vnode,1427247692705959881058285969449495136382746624000},undefined},{{riak_pipe_vnode,1438665674247607560106752257205091097473808596992},undefined},{{riak_pipe_vnode,1450083655789255239155218544960687058564870569984},undefined},{{riak_search_vnode,0},undefined},{{riak_search_vnode,11417981541647679048466287755595961091061972992},undefined},{{riak_search_vnode,22835963083295358096932575511191922182123945984},undefined},{{riak_search_vnode,34253944624943037145398863266787883273185918976},undefined},{{riak_search_vnode,45671926166590716193865151022383844364247891968},undefined},{{riak_search_vnode,57089907708238395242331438777979805455309864960},undefined},{{riak_search_vnode,68507889249886074290797726533575766546371837952},undefined},{{riak_search_vnode,79925870791533753339264014289171727637433810944},undefined},{{riak_search_vnode,91343852333181432387730302044767688728495783936},undefined},{{riak_search_vnode,102761833874829111436196589800363649819557756928},undefined},{{riak_search_vnode,114179815416476790484662877555959610910619729920},undefined},{{riak_search_vnode,125597796958124469533129165311555572001681702912},undefined},{{riak_search_vnode,137015778499772148581595453067151533092743675904},undefined},{{riak_search_vnode,148433760041419827630061740822747494183805648896},undefined},{{riak_search_vnode,159851741583067506678528028578343455274867621888},undefined},{{riak_search_vnode,171269723124715185726994316333939416365929594880},undefined},{{riak_search_vnode,182687704666362864775460604089535377456991567872},undefined},{{riak_search_vnode,194105686208010543823926891845131338548053540864},undefined},{{riak_search_vnode,205523667749658222872393179600727299639115513856},undefined},{{riak_search_vnode,216941649291305901920859467356323260730177486848},undefined},{{riak_search_vnode,228359630832953580969325755111919221821239459840},undefined},{{riak_search_vnode,239777612374601260017792042867515182912301432832},undefined},{{riak_search_vnode,251195593916248939066258330623111144003363405824},undefined},{{riak_search_vnode,262613575457896618114724618378707105094425378816},undefined},{{riak_search_vnode,274031556999544297163190906134303066185487351808},undefined},{{riak_search_vnode,285449538541191976211657193889899027276549324800},undefined},{{riak_search_vnode,296867520082839655260123481645494988367611297792},undefined},{{riak_search_vnode,308285501624487334308589769401090949458673270784},undefined},{{riak_search_vnode,319703483166135013357056057156686910549735243776},undefined},{{riak_search_vnode,331121464707782692405522344912282871640797216768},undefined},{{riak_search_vnode,342539446249430371453988632667878832731859189760},undefined},{{riak_search_vnode,353957427791078050502454920423474793822921162752},undefined},{{riak_search_vnode,365375409332725729550921208179070754913983135744},undefined},{{riak_search_vnode,376793390874373408599387495934666716005045108736},undefined},{{riak_search_vnode,388211372416021087647853783690262677096107081728},undefined},{{riak_search_vnode,399629353957668766696320071445858638187169054720},undefined},{{riak_search_vnode,411047335499316445744786359201454599278231027712},undefined},{{riak_search_vnode,422465317040964124793252646957050560369293000704},undefined},{{riak_search_vnode,433883298582611803841718934712646521460354973696},undefined},{{riak_search_vnode,445301280124259482890185222468242482551416946688},undefined},{{riak_search_vnode,456719261665907161938651510223838443642478919680},undefined},{{riak_search_vnode,468137243207554840987117797979434404733540892672},undefined},{{riak_search_vnode,479555224749202520035584085735030365824602865664},undefined},{{riak_search_vnode,490973206290850199084050373490626326915664838656},undefined},{{riak_search_vnode,502391187832497878132516661246222288006726811648},undefined},{{riak_search_vnode,513809169374145557180982949001818249097788784640},undefined},{{riak_search_vnode,525227150915793236229449236757414210188850757632},undefined},{{riak_search_vnode,536645132457440915277915524513010171279912730624},undefined},{{riak_search_vnode,548063113999088594326381812268606132370974703616},undefined},{{riak_search_vnode,559481095540736273374848100024202093462036676608},undefined},{{riak_search_vnode,570899077082383952423314387779798054553098649600},undefined},{{riak_search_vnode,582317058624031631471780675535394015644160622592},undefined},{{riak_search_vnode,593735040165679310520246963290989976735222595584},undefined},{{riak_search_vnode,605153021707326989568713251046585937826284568576},undefined},{{riak_search_vnode,616571003248974668617179538802181898917346541568},undefined},{{riak_search_vnode,627988984790622347665645826557777860008408514560},undefined},{{riak_search_vnode,639406966332270026714112114313373821099470487552},undefined},{{riak_search_vnode,650824947873917705762578402068969782190532460544},undefined},{{riak_search_vnode,662242929415565384811044689824565743281594433536},undefined},{{riak_search_vnode,673660910957213063859510977580161704372656406528},undefined},{{riak_search_vnode,685078892498860742907977265335757665463718379520},undefined},{{riak_search_vnode,696496874040508421956443553091353626554780352512},undefined},{{riak_search_vnode,707914855582156101004909840846949587645842325504},undefined},{{riak_search_vnode,719332837123803780053376128602545548736904298496},undefined},{{riak_search_vnode,730750818665451459101842416358141509827966271488},undefined},{{riak_search_vnode,742168800207099138150308704113737470919028244480},undefined},{{riak_search_vnode,753586781748746817198774991869333432010090217472},undefined},{{riak_search_vnode,765004763290394496247241279624929393101152190464},undefined},{{riak_search_vnode,776422744832042175295707567380525354192214163456},undefined},{{riak_search_vnode,787840726373689854344173855136121315283276136448},undefined},{{riak_search_vnode,799258707915337533392640142891717276374338109440},undefined},{{riak_search_vnode,810676689456985212441106430647313237465400082432},undefined},{{riak_search_vnode,822094670998632891489572718402909198556462055424},undefined},{{riak_search_vnode,833512652540280570538039006158505159647524028416},undefined},{{riak_search_vnode,844930634081928249586505293914101120738586001408},undefined},{{riak_search_vnode,856348615623575928634971581669697081829647974400},undefined},{{riak_search_vnode,867766597165223607683437869425293042920709947392},undefined},{{riak_search_vnode,879184578706871286731904157180889004011771920384},undefined},{{riak_search_vnode,890602560248518965780370444936484965102833893376},undefined},{{riak_search_vnode,902020541790166644828836732692080926193895866368},undefined},{{riak_search_vnode,913438523331814323877303020447676887284957839360},undefined},{{riak_search_vnode,924856504873462002925769308203272848376019812352},undefined},{{riak_search_vnode,936274486415109681974235595958868809467081785344},undefined},{{riak_search_vnode,947692467956757361022701883714464770558143758336},undefined},{{riak_search_vnode,959110449498405040071168171470060731649205731328},undefined},{{riak_search_vnode,970528431040052719119634459225656692740267704320},undefined},{{riak_search_vnode,981946412581700398168100746981252653831329677312},undefined},{{riak_search_vnode,993364394123348077216567034736848614922391650304},undefined},{{riak_search_vnode,1004782375664995756265033322492444576013453623296},undefined},{{riak_search_vnode,1016200357206643435313499610248040537104515596288},undefined},{{riak_search_vnode,1027618338748291114361965898003636498195577569280},undefined},{{riak_search_vnode,1039036320289938793410432185759232459286639542272},undefined},{{riak_search_vnode,1050454301831586472458898473514828420377701515264},undefined},{{riak_search_vnode,1061872283373234151507364761270424381468763488256},undefined},{{riak_search_vnode,1073290264914881830555831049026020342559825461248},undefined},{{riak_search_vnode,1084708246456529509604297336781616303650887434240},undefined},{{riak_search_vnode,1096126227998177188652763624537212264741949407232},undefined},{{riak_search_vnode,1107544209539824867701229912292808225833011380224},undefined},{{riak_search_vnode,1118962191081472546749696200048404186924073353216},undefined},{{riak_search_vnode,1130380172623120225798162487804000148015135326208},undefined},{{riak_search_vnode,1141798154164767904846628775559596109106197299200},undefined},{{riak_search_vnode,1153216135706415583895095063315192070197259272192},undefined},{{riak_search_vnode,1164634117248063262943561351070788031288321245184},undefined},{{riak_search_vnode,1176052098789710941992027638826383992379383218176},undefined},{{riak_search_vnode,1187470080331358621040493926581979953470445191168},undefined},{{riak_search_vnode,1198888061873006300088960214337575914561507164160},undefined},{{riak_search_vnode,1210306043414653979137426502093171875652569137152},undefined},{{riak_search_vnode,1221724024956301658185892789848767836743631110144},undefined},{{riak_search_vnode,1233142006497949337234359077604363797834693083136},undefined},{{riak_search_vnode,1244559988039597016282825365359959758925755056128},undefined},{{riak_search_vnode,1255977969581244695331291653115555720016817029120},undefined},{{riak_search_vnode,1267395951122892374379757940871151681107879002112},undefined},{{riak_search_vnode,1278813932664540053428224228626747642198940975104},undefined},{{riak_search_vnode,1290231914206187732476690516382343603290002948096},undefined},{{riak_search_vnode,1301649895747835411525156804137939564381064921088},undefined},{{riak_search_vnode,1313067877289483090573623091893535525472126894080},undefined},{{riak_search_vnode,1324485858831130769622089379649131486563188867072},undefined},{{riak_search_vnode,1335903840372778448670555667404727447654250840064},undefined},{{riak_search_vnode,1347321821914426127719021955160323408745312813056},undefined},{{riak_search_vnode,1358739803456073806767488242915919369836374786048},undefined},{{riak_search_vnode,1370157784997721485815954530671515330927436759040},undefined},{{riak_search_vnode,1381575766539369164864420818427111292018498732032},undefined},{{riak_search_vnode,1392993748081016843912887106182707253109560705024},undefined},{{riak_search_vnode,1404411729622664522961353393938303214200622678016},undefined},{{riak_search_vnode,1415829711164312202009819681693899175291684651008},undefined},{{riak_search_vnode,1427247692705959881058285969449495136382746624000},undefined},{{riak_search_vnode,1438665674247607560106752257205091097473808596992},undefined},{{riak_search_vnode,1450083655789255239155218544960687058564870569984},undefined}],[]}
>> > ** Reason for termination ==
>> > **
>> {{badmatch,{error,{{badmatch,{error,emfile}},[{riak_kv_vnode,init,1},{riak_core_vnode,init,1},{gen_fsm,init_it,6},{proc_lib,init_p_do_apply,3}]}}},[{riak_core_vnode_manager,get_vnode,3},{riak_core_vnode_manager,handle_call,3},{gen_server,handle_msg,5},{proc_lib,init_p_do_apply,3}]}
>> > 2012-05-18 04:59:00 =CRASH REPORT====
>> >  crasher:
>> >    initial call: riak_core_vnode_manager:init/1
>> >    pid: <0.18742.398>
>> >    registered_name: riak_core_vnode_manager
>> >    exception exit:
>> {{badmatch,{error,{{badmatch,{error,emfile}},[{riak_kv_vnode,init,1},{riak_core_vnode,init,1},{gen_fsm,init_it,6},{proc_lib,init_p_do_apply,3}]}}},[{riak_core_vnode_manager,get_vnode,3},{riak_core_vnode_manager,handle_call,3},{gen_server,handle_msg,5},{proc_lib,init_p_do_apply,3}]}
>> >      in function  gen_server:terminate/6
>> >      in call from proc_lib:init_p_do_apply/3
>> >    ancestors: [riak_core_sup,<0.91.0>]
>> >    messages: []
>> >    links: [<0.92.0>]
>> >    dictionary: []
>> >    trap_exit: false
>> >    status: running
>> >    heap_size: 17711
>> >    stack_size: 24
>> >    reductions: 546975
>> >  neighbours:
>> > 2012-05-18 04:59:00 =SUPERVISOR REPORT====
>> >     Supervisor: {local,riak_core_sup}
>> >     Context:    child_terminated
>> >     Reason:
>> {{badmatch,{error,{{badmatch,{error,emfile}},[{riak_kv_vnode,init,1},{riak_core_vnode,init,1},{gen_fsm,init_it,6},{proc_lib,init_p_do_apply,3}]}}},[{riak_core_vnode_manager,get_vnode,3},{riak_core_vnode_manager,handle_call,3},{gen_server,handle_msg,5},{proc_lib,init_p_do_apply,3}]}
>> >     Offender:
>> [{pid,<0.18742.398>},{name,riak_core_vnode_manager},{mfargs,{riak_core_vnode_manager,start_link,[]}},{restart_type,permanent},{shutdown,5000},{child_type,worker}]
>> >
>> > 2012-05-18 04:59:00 =CRASH REPORT====
>> >  crasher:
>> >    initial call: riak_core_vnode_proxy:init/1
>> >    pid: <0.21306.398>
>> >    registered_name:
>> proxy_riak_kv_vnode_137015778499772148581595453067151533092743675904
>> >    exception exit:
>> {{{badmatch,{error,{{badmatch,{error,emfile}},[{riak_kv_vnode,init,1},{riak_core_vnode,init,1},{gen_fsm,init_it,6},{proc_lib,init_p_do_apply,3}]}}},[{riak_core_vnode_manager,get_vnode,3},{riak_core_vnode_manager,handle_call,3},{gen_server,handle_msg,5},{proc_lib,init_p_do_apply,3}]},{gen_server,call,[riak_core_vnode_manager,{137015778499772148581595453067151533092743675904,riak_kv_vnode,get_vnode},infinity]}}
>> >      in function  gen_server:call/3
>> >      in call from riak_core_vnode_proxy:get_vnode_pid/1
>> >      in call from riak_core_vnode_proxy:handle_proxy/2
>> >      in call from riak_core_vnode_proxy:loop/2
>> >      in call from proc_lib:init_p_do_apply/3
>> >    ancestors: [riak_core_vnode_proxy_sup,riak_core_sup,<0.91.0>]
>> >    messages: []
>> >    links: [<0.20960.398>]
>> >    dictionary: []
>> >    trap_exit: false
>> >    status: running
>> >    heap_size: 377
>> >    stack_size: 24
>> >    reductions: 89
>> >  neighbours:
>> >
>> > Node 2:
>> >
>> > 2012-05-18 04:49:22 =ERROR REPORT====
>> > webmachine error: path="/buckets/contacts/index/email_bin/
>> event at panchabuta.com<mailto:event at panchabuta.com>"
>> >
>> {error,{error,{case_clause,{error,timeout,[[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[<<"N8XpooDmyVDCLPqGiAhjDVsXNqC">>],[],[],[],[],[],[]]}},[{riak_kv_wm_index,produce_index_results,2},{webmachine_resource,resource_call,3},{webmachine_resource,do,3},{webmachine_decision_core,resource_call,1},{webmachine_decision_core,decision,1},{webmachine_decision_core,handle_request,2},{webmachine_mochiweb,loop,1},{mochiweb_http,parse_headers,5}]}}
>> > 2012-05-18 04:49:34 =ERROR REPORT====
>> > ** Node 'riak at 10.38.24.196<mailto:riak at 10.38.24.196>' not responding **
>> > ** Removing (timedout) connection **
>> > 2012-05-18 04:49:34 =SUPERVISOR REPORT====
>> >     Supervisor: {local,riak_kv_put_fsm_sup}
>> >     Context:    child_terminated
>> >     Reason:     {{nodedown,'riak at 10.38.24.196<mailto:riak at 10.38.24.196
>> >'},{gen_server,call,[{riak_search_vnode_master,'riak at 10.38.24.196
>> <mailto:riak at 10.38.24.196
>> >'},{riak_vnode_req_v1,1004782375664995756265033322492444576013453623296,{server,undefined,undefined},{delete_v1,[{<<"contacts">>,<<"page_keys">>,<<"njxqbdh1xgelrafoxahmlxjsuy7">>,<<"KDad4R2BVEdZXf0msX5OoTCr3lq">>,[{p,[0]}],1337330961353632}]}},infinity]}}
>> >     Offender:
>> [{pid,<0.2228.397>},{name,undefined},{mfargs,{riak_kv_put_fsm,start_link,undefined}},{restart_type,temporary},{shutdown,5000},{child_type,worker}]
>> >
>> > 2012-05-18 04:49:41 =ERROR REPORT====
>> > webmachine error: path="/buckets/contacts/index/email_bin/
>> inspector at bchouseinspections.ca<mailto:inspector at bchouseinspections.ca>"
>> >
>> {error,{error,{case_clause,{error,timeout,[[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]]}},[{riak_kv_wm_index,produce_index_results,2},{webmachine_resource,resource_call,3},{webmachine_resource,do,3},{webmachine_decision_core,resource_call,1},{webmachine_decision_core,decision,1},{webmachine_decision_core,handle_request,2},{webmachine_mochiweb,loop,1},{mochiweb_http,parse_headers,5}]}}
>> > 2012-05-18 04:50:23 =ERROR REPORT====
>> > webmachine error: path="/buckets/contacts/index/email_bin/
>> event at panchabuta.com<mailto:event at panchabuta.com>"
>> >
>> {error,{error,{case_clause,{error,timeout,[[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[<<"N8XpooDmyVDCLPqGiAhjDVsXNqC">>],[],[],[]]}},[{riak_kv_wm_index,produce_index_results,2},{webmachine_resource,resource_call,3},{webmachine_resource,do,3},{webmachine_decision_core,resource_call,1},{webmachine_decision_core,decision,1},{webmachine_decision_core,handle_request,2},{webmachine_mochiweb,loop,1},{mochiweb_http,parse_headers,5}]}}
>> > 2012-05-18 04:50:49 =ERROR REPORT====
>> > ** Node 'riak at 10.38.24.196<mailto:riak at 10.38.24.196>' not responding **
>> > ** Removing (timedout) connection **
>> > 2012-05-18 04:52:49 =ERROR REPORT====
>> > ** Node 'riak at 10.38.24.196<mailto:riak at 10.38.24.196>' not responding **
>> > ** Removing (timedout) connection **
>> > 2012-05-18 04:52:49 =SUPERVISOR REPORT====
>> >     Supervisor: {local,riak_kv_put_fsm_sup}
>> >     Context:    child_terminated
>> >     Reason:     {{nodedown,'riak at 10.38.24.196<mailto:riak at 10.38.24.196
>> >'},{gen_server,call,[{riak_search_vnode_master,'riak at 10.38.24.196
>> <mailto:riak at 10.38.24.196
>> >'},{riak_vnode_req_v1,1050454301831586472458898473514828420377701515264,{server,undefined,undefined},{delete_v1,[{<<"sites">>,<<"updated_at">>,<<"2012-05-18">>,<<"fa3FeLb9UYhf0K1VdmR7CMdGCV">>,[{p,[0]}],1337331089082754}]}},infinity]}}
>> >     Offender:
>> [{pid,<0.3077.397>},{name,undefined},{mfargs,{riak_kv_put_fsm,start_link,undefined}},{restart_type,temporary},{shutdown,5000},{child_type,worker}]
>> >
>> > 2012-05-18 04:52:49 =SUPERVISOR REPORT====
>> >     Supervisor: {local,riak_kv_put_fsm_sup}
>> >     Context:    child_terminated
>> >     Reason:     {{nodedown,'riak at 10.38.24.196<mailto:riak at 10.38.24.196
>> >'},{gen_server,call,[{riak_search_vnode_master,'riak at 10.38.24.196
>> <mailto:riak at 10.38.24.196
>> >'},{riak_vnode_req_v1,45671926166590716193865151022383844364247891968,{server,undefined,undefined},{delete_v1,[{<<"sites">>,<<"page_keys">>,<<"m9ksizkajltxgqeck2dix0y6vzs">>,<<"5Cd4M1awqLwGsRKJi2QkVB79h3g">>,[{p,[23]}],1337331105707883}]}},infinity]}}
>> >     Offender:
>> [{pid,<0.3102.397>},{name,undefined},{mfargs,{riak_kv_put_fsm,start_link,undefined}},{restart_type,temporary},{shutdown,5000},{child_type,worker}]
>> >
>> > 2012-05-18 04:52:49 =SUPERVISOR REPORT====
>> >     Supervisor: {local,riak_kv_put_fsm_sup}
>> >     Context:    child_terminated
>> >     Reason:     {{nodedown,'riak at 10.38.24.196<mailto:riak at 10.38.24.196
>> >'},{gen_server,call,[{riak_search_vnode_master,'riak at 10.38.24.196
>> <mailto:riak at 10.38.24.196
>> >'},{riak_vnode_req_v1,1084708246456529509604297336781616303650887434240,{server,undefined,undefined},{index_v1,[{<<"sites">>,<<"num_contacts">>,<<"0000000000">>,<<"Z6bUs5DmmdRggHv1XjXZe30a1Zz">>,[{p,[0]}],1337331091385105}]}},infinity]}}
>> >     Offender:
>> [{pid,<0.3083.397>},{name,undefined},{mfargs,{riak_kv_put_fsm,start_link,undefined}},{restart_type,temporary},{shutdown,5000},{child_type,worker}]
>> >
>> > ....
>> >
>> > 2012-05-18 04:59:01 =ERROR REPORT====
>> > ** Generic server riak_core_vnode_manager terminating
>> > ** Last message in was
>> {947692467956757361022701883714464770558143758336,riak_kv_vnode,get_vnode}
>> > ** When Server state ==
>> {state,40799215793,[{{riak_kv_vnode,0},undefined},{{riak_kv_vnode,11417981541647679048466287755595961091061972992},undefined},{{riak_kv_vnode,22835963083295358096932575511191922182123945984},undefined},{{riak_kv_vnode,34253944624943037145398863266787883273185918976},undefined},{{riak_kv_vnode,45671926166590716193865151022383844364247891968},undefined},{{riak_kv_vnode,57089907708238395242331438777979805455309864960},undefined},{{riak_kv_vnode,68507889249886074290797726533575766546371837952},undefined},{{riak_kv_vnode,79925870791533753339264014289171727637433810944},undefined},{{riak_kv_vnode,91343852333181432387730302044767688728495783936},undefined},{{riak_kv_vnode,102761833874829111436196589800363649819557756928},undefined},{{riak_kv_vnode,114179815416476790484662877555959610910619729920},undefined},{{riak_kv_vnode,125597796958124469533129165311555572001681702912},undefined},{{riak_kv_vnode,137015778499772148581595453067151533092743675904},undefined},{{riak_kv_vnode,148433760041419827630061740822747494183805648896},undefined},{{riak_kv_vnode,159851741583067506678528028578343455274867621888},undefined},{{riak_kv_vnode,171269723124715185726994316333939416365929594880},undefined},{{riak_kv_vnode,182687704666362864775460604089535377456991567872},undefined},{{riak_kv_vnode,194105686208010543823926891845131338548053540864},undefined},{{riak_kv_vnode,205523667749658222872393179600727299639115513856},undefined},{{riak_kv_vnode,216941649291305901920859467356323260730177486848},undefined},{{riak_kv_vnode,228359630832953580969325755111919221821239459840},undefined},{{riak_kv_vnode,239777612374601260017792042867515182912301432832},undefined},{{riak_kv_vnode,251195593916248939066258330623111144003363405824},undefined},{{riak_kv_vnode,262613575457896618114724618378707105094425378816},undefined},{{riak_kv_vnode,274031556999544297163190906134303066185487351808},undefined},{{riak_kv_vnode,285449538541191976211657193889899027276549324800},undefined},{{riak_kv_vnode,296867520082839655260123481645494988367611297792},undefined},{{riak_kv_vnode,308285501624487334308589769401090949458673270784},undefined},{{riak_kv_vnode,319703483166135013357056057156686910549735243776},undefined},{{riak_kv_vnode,331121464707782692405522344912282871640797216768},undefined},{{riak_kv_vnode,342539446249430371453988632667878832731859189760},undefined},{{riak_kv_vnode,353957427791078050502454920423474793822921162752},undefined},{{riak_kv_vnode,365375409332725729550921208179070754913983135744},undefined},{{riak_kv_vnode,376793390874373408599387495934666716005045108736},undefined},{{riak_kv_vnode,388211372416021087647853783690262677096107081728},undefined},{{riak_kv_vnode,399629353957668766696320071445858638187169054720},undefined},{{riak_kv_vnode,411047335499316445744786359201454599278231027712},undefined},{{riak_kv_vnode,422465317040964124793252646957050560369293000704},undefined},{{riak_kv_vnode,433883298582611803841718934712646521460354973696},undefined},{{riak_kv_vnode,445301280124259482890185222468242482551416946688},undefined},{{riak_kv_vnode,456719261665907161938651510223838443642478919680},undefined},{{riak_kv_vnode,468137243207554840987117797979434404733540892672},undefined},{{riak_kv_vnode,479555224749202520035584085735030365824602865664},undefined},{{riak_kv_vnode,490973206290850199084050373490626326915664838656},undefined},{{riak_kv_vnode,502391187832497878132516661246222288006726811648},undefined},{{riak_kv_vnode,513809169374145557180982949001818249097788784640},undefined},{{riak_kv_vnode,525227150915793236229449236757414210188850757632},undefined},{{riak_kv_vnode,536645132457440915277915524513010171279912730624},undefined},{{riak_kv_vnode,548063113999088594326381812268606132370974703616},undefined},{{riak_kv_vnode,559481095540736273374848100024202093462036676608},undefined},{{riak_kv_vnode,570899077082383952423314387779798054553098649600},undefined},{{riak_kv_vnode,582317058624031631471780675535394015644160622592},undefined},{{riak_kv_vnode,593735040165679310520246963290989976735222595584},undefined},{{riak_kv_vnode,605153021707326989568713251046585937826284568576},undefined},{{riak_kv_vnode,616571003248974668617179538802181898917346541568},undefined},{{riak_kv_vnode,627988984790622347665645826557777860008408514560},undefined},{{riak_kv_vnode,639406966332270026714112114313373821099470487552},undefined},{{riak_kv_vnode,650824947873917705762578402068969782190532460544},undefined},{{riak_kv_vnode,662242929415565384811044689824565743281594433536},undefined},{{riak_kv_vnode,673660910957213063859510977580161704372656406528},undefined},{{riak_kv_vnode,685078892498860742907977265335757665463718379520},undefined},{{riak_kv_vnode,696496874040508421956443553091353626554780352512},undefined},{{riak_kv_vnode,707914855582156101004909840846949587645842325504},undefined},{{riak_kv_vnode,719332837123803780053376128602545548736904298496},undefined},{{riak_kv_vnode,730750818665451459101842416358141509827966271488},undefined},{{riak_kv_vnode,742168800207099138150308704113737470919028244480},undefined},{{riak_kv_vnode,753586781748746817198774991869333432010090217472},undefined},{{riak_kv_vnode,765004763290394496247241279624929393101152190464},undefined},{{riak_kv_vnode,776422744832042175295707567380525354192214163456},undefined},{{riak_kv_vnode,787840726373689854344173855136121315283276136448},undefined},{{riak_kv_vnode,799258707915337533392640142891717276374338109440},undefined},{{riak_kv_vnode,810676689456985212441106430647313237465400082432},undefined},{{riak_kv_vnode,822094670998632891489572718402909198556462055424},undefined},{{riak_kv_vnode,833512652540280570538039006158505159647524028416},undefined},{{riak_kv_vnode,844930634081928249586505293914101120738586001408},undefined},{{riak_kv_vnode,856348615623575928634971581669697081829647974400},undefined},{{riak_kv_vnode,867766597165223607683437869425293042920709947392},undefined},{{riak_kv_vnode,879184578706871286731904157180889004011771920384},undefined},{{riak_kv_vnode,890602560248518965780370444936484965102833893376},undefined},{{riak_kv_vnode,902020541790166644828836732692080926193895866368},undefined},{{riak_kv_vnode,913438523331814323877303020447676887284957839360},undefined},{{riak_kv_vnode,924856504873462002925769308203272848376019812352},undefined},{{riak_kv_vnode,936274486415109681974235595958868809467081785344},undefined},{{riak_kv_vnode,947692467956757361022701883714464770558143758336},undefined},{{riak_kv_vnode,959110449498405040071168171470060731649205731328},undefined},{{riak_kv_vnode,970528431040052719119634459225656692740267704320},undefined},{{riak_kv_vnode,981946412581700398168100746981252653831329677312},undefined},{{riak_kv_vnode,993364394123348077216567034736848614922391650304},undefined},{{riak_kv_vnode,1004782375664995756265033322492444576013453623296},undefined},{{riak_kv_vnode,1016200357206643435313499610248040537104515596288},undefined},{{riak_kv_vnode,1027618338748291114361965898003636498195577569280},undefined},{{riak_kv_vnode,1039036320289938793410432185759232459286639542272},undefined},{{riak_kv_vnode,1050454301831586472458898473514828420377701515264},undefined},{{riak_kv_vnode,1061872283373234151507364761270424381468763488256},undefined},{{riak_kv_vnode,1073290264914881830555831049026020342559825461248},undefined},{{riak_kv_vnode,1084708246456529509604297336781616303650887434240},undefined},{{riak_kv_vnode,1096126227998177188652763624537212264741949407232},undefined},{{riak_kv_vnode,1107544209539824867701229912292808225833011380224},undefined},{{riak_kv_vnode,1118962191081472546749696200048404186924073353216},undefined},{{riak_kv_vnode,1130380172623120225798162487804000148015135326208},undefined},{{riak_kv_vnode,1141798154164767904846628775559596109106197299200},undefined},{{riak_kv_vnode,1153216135706415583895095063315192070197259272192},undefined},{{riak_kv_vnode,1164634117248063262943561351070788031288321245184},undefined},{{riak_kv_vnode,1176052098789710941992027638826383992379383218176},undefined},{{riak_kv_vnode,1187470080331358621040493926581979953470445191168},undefined},{{riak_kv_vnode,1198888061873006300088960214337575914561507164160},undefined},{{riak_kv_vnode,1210306043414653979137426502093171875652569137152},undefined},{{riak_kv_vnode,1221724024956301658185892789848767836743631110144},undefined},{{riak_kv_vnode,1233142006497949337234359077604363797834693083136},undefined},{{riak_kv_vnode,1244559988039597016282825365359959758925755056128},undefined},{{riak_kv_vnode,1255977969581244695331291653115555720016817029120},undefined},{{riak_kv_vnode,1267395951122892374379757940871151681107879002112},undefined},{{riak_kv_vnode,1278813932664540053428224228626747642198940975104},undefined},{{riak_kv_vnode,1290231914206187732476690516382343603290002948096},undefined},{{riak_kv_vnode,1301649895747835411525156804137939564381064921088},undefined},{{riak_kv_vnode,1313067877289483090573623091893535525472126894080},undefined},{{riak_kv_vnode,1324485858831130769622089379649131486563188867072},undefined},{{riak_kv_vnode,1335903840372778448670555667404727447654250840064},undefined},{{riak_kv_vnode,1347321821914426127719021955160323408745312813056},undefined},{{riak_kv_vnode,1358739803456073806767488242915919369836374786048},undefined},{{riak_kv_vnode,1370157784997721485815954530671515330927436759040},undefined},{{riak_kv_vnode,1381575766539369164864420818427111292018498732032},undefined},{{riak_kv_vnode,1392993748081016843912887106182707253109560705024},undefined},{{riak_kv_vnode,1404411729622664522961353393938303214200622678016},undefined},{{riak_kv_vnode,1415829711164312202009819681693899175291684651008},undefined},{{riak_kv_vnode,1427247692705959881058285969449495136382746624000},undefined},{{riak_kv_vnode,1438665674247607560106752257205091097473808596992},undefined},{{riak_kv_vnode,1450083655789255239155218544960687058564870569984},undefined},{{riak_pipe_vnode,0},undefined},{{riak_pipe_vnode,11417981541647679048466287755595961091061972992},undefined},{{riak_pipe_vnode,22835963083295358096932575511191922182123945984},undefined},{{riak_pipe_vnode,34253944624943037145398863266787883273185918976},undefined},{{riak_pipe_vnode,45671926166590716193865151022383844364247891968},undefined},{{riak_pipe_vnode,57089907708238395242331438777979805455309864960},undefined},{{riak_pipe_vnode,68507889249886074290797726533575766546371837952},undefined},{{riak_pipe_vnode,79925870791533753339264014289171727637433810944},undefined},{{riak_pipe_vnode,91343852333181432387730302044767688728495783936},undefined},{{riak_pipe_vnode,102761833874829111436196589800363649819557756928},undefined},{{riak_pipe_vnode,114179815416476790484662877555959610910619729920},undefined},{{riak_pipe_vnode,125597796958124469533129165311555572001681702912},undefined},{{riak_pipe_vnode,137015778499772148581595453067151533092743675904},undefined},{{riak_pipe_vnode,148433760041419827630061740822747494183805648896},undefined},{{riak_pipe_vnode,159851741583067506678528028578343455274867621888},undefined},{{riak_pipe_vnode,171269723124715185726994316333939416365929594880},undefined},{{riak_pipe_vnode,182687704666362864775460604089535377456991567872},undefined},{{riak_pipe_vnode,194105686208010543823926891845131338548053540864},undefined},{{riak_pipe_vnode,205523667749658222872393179600727299639115513856},undefined},{{riak_pipe_vnode,216941649291305901920859467356323260730177486848},undefined},{{riak_pipe_vnode,228359630832953580969325755111919221821239459840},undefined},{{riak_pipe_vnode,239777612374601260017792042867515182912301432832},undefined},{{riak_pipe_vnode,251195593916248939066258330623111144003363405824},undefined},{{riak_pipe_vnode,262613575457896618114724618378707105094425378816},undefined},{{riak_pipe_vnode,274031556999544297163190906134303066185487351808},undefined},{{riak_pipe_vnode,285449538541191976211657193889899027276549324800},undefined},{{riak_pipe_vnode,296867520082839655260123481645494988367611297792},undefined},{{riak_pipe_vnode,308285501624487334308589769401090949458673270784},undefined},{{riak_pipe_vnode,319703483166135013357056057156686910549735243776},undefined},{{riak_pipe_vnode,331121464707782692405522344912282871640797216768},undefined},{{riak_pipe_vnode,342539446249430371453988632667878832731859189760},undefined},{{riak_pipe_vnode,353957427791078050502454920423474793822921162752},undefined},{{riak_pipe_vnode,365375409332725729550921208179070754913983135744},undefined},{{riak_pipe_vnode,376793390874373408599387495934666716005045108736},undefined},{{riak_pipe_vnode,388211372416021087647853783690262677096107081728},undefined},{{riak_pipe_vnode,399629353957668766696320071445858638187169054720},undefined},{{riak_pipe_vnode,411047335499316445744786359201454599278231027712},undefined},{{riak_pipe_vnode,422465317040964124793252646957050560369293000704},undefined},{{riak_pipe_vnode,433883298582611803841718934712646521460354973696},undefined},{{riak_pipe_vnode,445301280124259482890185222468242482551416946688},undefined},{{riak_pipe_vnode,456719261665907161938651510223838443642478919680},undefined},{{riak_pipe_vnode,468137243207554840987117797979434404733540892672},undefined},{{riak_pipe_vnode,479555224749202520035584085735030365824602865664},undefined},{{riak_pipe_vnode,490973206290850199084050373490626326915664838656},undefined},{{riak_pipe_vnode,502391187832497878132516661246222288006726811648},undefined},{{riak_pipe_vnode,513809169374145557180982949001818249097788784640},undefined},{{riak_pipe_vnode,525227150915793236229449236757414210188850757632},undefined},{{riak_pipe_vnode,536645132457440915277915524513010171279912730624},undefined},{{riak_pipe_vnode,548063113999088594326381812268606132370974703616},undefined},{{riak_pipe_vnode,559481095540736273374848100024202093462036676608},undefined},{{riak_pipe_vnode,570899077082383952423314387779798054553098649600},undefined},{{riak_pipe_vnode,582317058624031631471780675535394015644160622592},undefined},{{riak_pipe_vnode,593735040165679310520246963290989976735222595584},undefined},{{riak_pipe_vnode,605153021707326989568713251046585937826284568576},undefined},{{riak_pipe_vnode,616571003248974668617179538802181898917346541568},undefined},{{riak_pipe_vnode,627988984790622347665645826557777860008408514560},undefined},{{riak_pipe_vnode,639406966332270026714112114313373821099470487552},undefined},{{riak_pipe_vnode,650824947873917705762578402068969782190532460544},undefined},{{riak_pipe_vnode,662242929415565384811044689824565743281594433536},undefined},{{riak_pipe_vnode,673660910957213063859510977580161704372656406528},undefined},{{riak_pipe_vnode,685078892498860742907977265335757665463718379520},undefined},{{riak_pipe_vnode,696496874040508421956443553091353626554780352512},undefined},{{riak_pipe_vnode,707914855582156101004909840846949587645842325504},undefined},{{riak_pipe_vnode,719332837123803780053376128602545548736904298496},undefined},{{riak_pipe_vnode,730750818665451459101842416358141509827966271488},undefined},{{riak_pipe_vnode,742168800207099138150308704113737470919028244480},undefined},{{riak_pipe_vnode,753586781748746817198774991869333432010090217472},undefined},{{riak_pipe_vnode,765004763290394496247241279624929393101152190464},undefined},{{riak_pipe_vnode,776422744832042175295707567380525354192214163456},undefined},{{riak_pipe_vnode,787840726373689854344173855136121315283276136448},undefined},{{riak_pipe_vnode,799258707915337533392640142891717276374338109440},undefined},{{riak_pipe_vnode,810676689456985212441106430647313237465400082432},undefined},{{riak_pipe_vnode,822094670998632891489572718402909198556462055424},undefined},{{riak_pipe_vnode,833512652540280570538039006158505159647524028416},undefined},{{riak_pipe_vnode,844930634081928249586505293914101120738586001408},undefined},{{riak_pipe_vnode,856348615623575928634971581669697081829647974400},undefined},{{riak_pipe_vnode,867766597165223607683437869425293042920709947392},undefined},{{riak_pipe_vnode,879184578706871286731904157180889004011771920384},undefined},{{riak_pipe_vnode,890602560248518965780370444936484965102833893376},undefined},{{riak_pipe_vnode,902020541790166644828836732692080926193895866368},undefined},{{riak_pipe_vnode,913438523331814323877303020447676887284957839360},undefined},{{riak_pipe_vnode,924856504873462002925769308203272848376019812352},undefined},{{riak_pipe_vnode,936274486415109681974235595958868809467081785344},undefined},{{riak_pipe_vnode,947692467956757361022701883714464770558143758336},undefined},{{riak_pipe_vnode,959110449498405040071168171470060731649205731328},undefined},{{riak_pipe_vnode,970528431040052719119634459225656692740267704320},undefined},{{riak_pipe_vnode,981946412581700398168100746981252653831329677312},undefined},{{riak_pipe_vnode,993364394123348077216567034736848614922391650304},undefined},{{riak_pipe_vnode,1004782375664995756265033322492444576013453623296},undefined},{{riak_pipe_vnode,1016200357206643435313499610248040537104515596288},undefined},{{riak_pipe_vnode,1027618338748291114361965898003636498195577569280},undefined},{{riak_pipe_vnode,1039036320289938793410432185759232459286639542272},undefined},{{riak_pipe_vnode,1050454301831586472458898473514828420377701515264},undefined},{{riak_pipe_vnode,1061872283373234151507364761270424381468763488256},undefined},{{riak_pipe_vnode,1073290264914881830555831049026020342559825461248},undefined},{{riak_pipe_vnode,1084708246456529509604297336781616303650887434240},undefined},{{riak_pipe_vnode,1096126227998177188652763624537212264741949407232},undefined},{{riak_pipe_vnode,1107544209539824867701229912292808225833011380224},undefined},{{riak_pipe_vnode,1118962191081472546749696200048404186924073353216},undefined},{{riak_pipe_vnode,1130380172623120225798162487804000148015135326208},undefined},{{riak_pipe_vnode,1141798154164767904846628775559596109106197299200},undefined},{{riak_pipe_vnode,1153216135706415583895095063315192070197259272192},undefined},{{riak_pipe_vnode,1164634117248063262943561351070788031288321245184},undefined},{{riak_pipe_vnode,1176052098789710941992027638826383992379383218176},undefined},{{riak_pipe_vnode,1187470080331358621040493926581979953470445191168},undefined},{{riak_pipe_vnode,1198888061873006300088960214337575914561507164160},undefined},{{riak_pipe_vnode,1210306043414653979137426502093171875652569137152},undefined},{{riak_pipe_vnode,1221724024956301658185892789848767836743631110144},undefined},{{riak_pipe_vnode,1233142006497949337234359077604363797834693083136},undefined},{{riak_pipe_vnode,1244559988039597016282825365359959758925755056128},undefined},{{riak_pipe_vnode,1255977969581244695331291653115555720016817029120},undefined},{{riak_pipe_vnode,1267395951122892374379757940871151681107879002112},undefined},{{riak_pipe_vnode,1278813932664540053428224228626747642198940975104},undefined},{{riak_pipe_vnode,1290231914206187732476690516382343603290002948096},undefined},{{riak_pipe_vnode,1301649895747835411525156804137939564381064921088},undefined},{{riak_pipe_vnode,1313067877289483090573623091893535525472126894080},undefined},{{riak_pipe_vnode,1324485858831130769622089379649131486563188867072},undefined},{{riak_pipe_vnode,1335903840372778448670555667404727447654250840064},undefined},{{riak_pipe_vnode,1347321821914426127719021955160323408745312813056},undefined},{{riak_pipe_vnode,1358739803456073806767488242915919369836374786048},undefined},{{riak_pipe_vnode,1370157784997721485815954530671515330927436759040},undefined},{{riak_pipe_vnode,1381575766539369164864420818427111292018498732032},undefined},{{riak_pipe_vnode,1392993748081016843912887106182707253109560705024},undefined},{{riak_pipe_vnode,1404411729622664522961353393938303214200622678016},undefined},{{riak_pipe_vnode,1415829711164312202009819681693899175291684651008},undefined},{{riak_pipe_vnode,1427247692705959881058285969449495136382746624000},undefined},{{riak_pipe_vnode,1438665674247607560106752257205091097473808596992},undefined},{{riak_pipe_vnode,1450083655789255239155218544960687058564870569984},undefined},{{riak_search_vnode,0},undefined},{{riak_search_vnode,11417981541647679048466287755595961091061972992},undefined},{{riak_search_vnode,22835963083295358096932575511191922182123945984},undefined},{{riak_search_vnode,34253944624943037145398863266787883273185918976},undefined},{{riak_search_vnode,45671926166590716193865151022383844364247891968},undefined},{{riak_search_vnode,57089907708238395242331438777979805455309864960},undefined},{{riak_search_vnode,68507889249886074290797726533575766546371837952},undefined},{{riak_search_vnode,79925870791533753339264014289171727637433810944},undefined},{{riak_search_vnode,91343852333181432387730302044767688728495783936},undefined},{{riak_search_vnode,102761833874829111436196589800363649819557756928},undefined},{{riak_search_vnode,114179815416476790484662877555959610910619729920},undefined},{{riak_search_vnode,125597796958124469533129165311555572001681702912},undefined},{{riak_search_vnode,137015778499772148581595453067151533092743675904},undefined},{{riak_search_vnode,148433760041419827630061740822747494183805648896},undefined},{{riak_search_vnode,159851741583067506678528028578343455274867621888},undefined},{{riak_search_vnode,171269723124715185726994316333939416365929594880},undefined},{{riak_search_vnode,182687704666362864775460604089535377456991567872},undefined},{{riak_search_vnode,194105686208010543823926891845131338548053540864},undefined},{{riak_search_vnode,205523667749658222872393179600727299639115513856},undefined},{{riak_search_vnode,216941649291305901920859467356323260730177486848},undefined},{{riak_search_vnode,228359630832953580969325755111919221821239459840},undefined},{{riak_search_vnode,239777612374601260017792042867515182912301432832},undefined},{{riak_search_vnode,251195593916248939066258330623111144003363405824},undefined},{{riak_search_vnode,262613575457896618114724618378707105094425378816},undefined},{{riak_search_vnode,274031556999544297163190906134303066185487351808},undefined},{{riak_search_vnode,285449538541191976211657193889899027276549324800},undefined},{{riak_search_vnode,296867520082839655260123481645494988367611297792},undefined},{{riak_search_vnode,308285501624487334308589769401090949458673270784},undefined},{{riak_search_vnode,319703483166135013357056057156686910549735243776},undefined},{{riak_search_vnode,331121464707782692405522344912282871640797216768},undefined},{{riak_search_vnode,342539446249430371453988632667878832731859189760},undefined},{{riak_search_vnode,353957427791078050502454920423474793822921162752},undefined},{{riak_search_vnode,365375409332725729550921208179070754913983135744},undefined},{{riak_search_vnode,376793390874373408599387495934666716005045108736},undefined},{{riak_search_vnode,388211372416021087647853783690262677096107081728},undefined},{{riak_search_vnode,399629353957668766696320071445858638187169054720},undefined},{{riak_search_vnode,411047335499316445744786359201454599278231027712},undefined},{{riak_search_vnode,422465317040964124793252646957050560369293000704},undefined},{{riak_search_vnode,433883298582611803841718934712646521460354973696},undefined},{{riak_search_vnode,445301280124259482890185222468242482551416946688},undefined},{{riak_search_vnode,456719261665907161938651510223838443642478919680},undefined},{{riak_search_vnode,468137243207554840987117797979434404733540892672},undefined},{{riak_search_vnode,479555224749202520035584085735030365824602865664},undefined},{{riak_search_vnode,490973206290850199084050373490626326915664838656},undefined},{{riak_search_vnode,502391187832497878132516661246222288006726811648},undefined},{{riak_search_vnode,513809169374145557180982949001818249097788784640},undefined},{{riak_search_vnode,525227150915793236229449236757414210188850757632},undefined},{{riak_search_vnode,536645132457440915277915524513010171279912730624},undefined},{{riak_search_vnode,548063113999088594326381812268606132370974703616},undefined},{{riak_search_vnode,559481095540736273374848100024202093462036676608},undefined},{{riak_search_vnode,570899077082383952423314387779798054553098649600},undefined},{{riak_search_vnode,582317058624031631471780675535394015644160622592},undefined},{{riak_search_vnode,593735040165679310520246963290989976735222595584},undefined},{{riak_search_vnode,605153021707326989568713251046585937826284568576},undefined},{{riak_search_vnode,616571003248974668617179538802181898917346541568},undefined},{{riak_search_vnode,627988984790622347665645826557777860008408514560},undefined},{{riak_search_vnode,639406966332270026714112114313373821099470487552},undefined},{{riak_search_vnode,650824947873917705762578402068969782190532460544},undefined},{{riak_search_vnode,662242929415565384811044689824565743281594433536},undefined},{{riak_search_vnode,673660910957213063859510977580161704372656406528},undefined},{{riak_search_vnode,685078892498860742907977265335757665463718379520},undefined},{{riak_search_vnode,696496874040508421956443553091353626554780352512},undefined},{{riak_search_vnode,707914855582156101004909840846949587645842325504},undefined},{{riak_search_vnode,719332837123803780053376128602545548736904298496},undefined},{{riak_search_vnode,730750818665451459101842416358141509827966271488},undefined},{{riak_search_vnode,742168800207099138150308704113737470919028244480},undefined},{{riak_search_vnode,753586781748746817198774991869333432010090217472},undefined},{{riak_search_vnode,765004763290394496247241279624929393101152190464},undefined},{{riak_search_vnode,776422744832042175295707567380525354192214163456},undefined},{{riak_search_vnode,787840726373689854344173855136121315283276136448},undefined},{{riak_search_vnode,799258707915337533392640142891717276374338109440},undefined},{{riak_search_vnode,810676689456985212441106430647313237465400082432},undefined},{{riak_search_vnode,822094670998632891489572718402909198556462055424},undefined},{{riak_search_vnode,833512652540280570538039006158505159647524028416},undefined},{{riak_search_vnode,844930634081928249586505293914101120738586001408},undefined},{{riak_search_vnode,856348615623575928634971581669697081829647974400},undefined},{{riak_search_vnode,867766597165223607683437869425293042920709947392},undefined},{{riak_search_vnode,879184578706871286731904157180889004011771920384},undefined},{{riak_search_vnode,890602560248518965780370444936484965102833893376},undefined},{{riak_search_vnode,902020541790166644828836732692080926193895866368},undefined},{{riak_search_vnode,913438523331814323877303020447676887284957839360},undefined},{{riak_search_vnode,924856504873462002925769308203272848376019812352},undefined},{{riak_search_vnode,936274486415109681974235595958868809467081785344},undefined},{{riak_search_vnode,947692467956757361022701883714464770558143758336},undefined},{{riak_search_vnode,959110449498405040071168171470060731649205731328},undefined},{{riak_search_vnode,970528431040052719119634459225656692740267704320},undefined},{{riak_search_vnode,981946412581700398168100746981252653831329677312},undefined},{{riak_search_vnode,993364394123348077216567034736848614922391650304},undefined},{{riak_search_vnode,1004782375664995756265033322492444576013453623296},undefined},{{riak_search_vnode,1016200357206643435313499610248040537104515596288},undefined},{{riak_search_vnode,1027618338748291114361965898003636498195577569280},undefined},{{riak_search_vnode,1039036320289938793410432185759232459286639542272},undefined},{{riak_search_vnode,1050454301831586472458898473514828420377701515264},undefined},{{riak_search_vnode,1061872283373234151507364761270424381468763488256},undefined},{{riak_search_vnode,1073290264914881830555831049026020342559825461248},undefined},{{riak_search_vnode,1084708246456529509604297336781616303650887434240},undefined},{{riak_search_vnode,1096126227998177188652763624537212264741949407232},undefined},{{riak_search_vnode,1107544209539824867701229912292808225833011380224},undefined},{{riak_search_vnode,1118962191081472546749696200048404186924073353216},undefined},{{riak_search_vnode,1130380172623120225798162487804000148015135326208},undefined},{{riak_search_vnode,1141798154164767904846628775559596109106197299200},undefined},{{riak_search_vnode,1153216135706415583895095063315192070197259272192},undefined},{{riak_search_vnode,1164634117248063262943561351070788031288321245184},undefined},{{riak_search_vnode,1176052098789710941992027638826383992379383218176},undefined},{{riak_search_vnode,1187470080331358621040493926581979953470445191168},undefined},{{riak_search_vnode,1198888061873006300088960214337575914561507164160},undefined},{{riak_search_vnode,1210306043414653979137426502093171875652569137152},undefined},{{riak_search_vnode,1221724024956301658185892789848767836743631110144},undefined},{{riak_search_vnode,1233142006497949337234359077604363797834693083136},undefined},{{riak_search_vnode,1244559988039597016282825365359959758925755056128},undefined},{{riak_search_vnode,1255977969581244695331291653115555720016817029120},undefined},{{riak_search_vnode,1267395951122892374379757940871151681107879002112},undefined},{{riak_search_vnode,1278813932664540053428224228626747642198940975104},undefined},{{riak_search_vnode,1290231914206187732476690516382343603290002948096},undefined},{{riak_search_vnode,1301649895747835411525156804137939564381064921088},undefined},{{riak_search_vnode,1313067877289483090573623091893535525472126894080},undefined},{{riak_search_vnode,1324485858831130769622089379649131486563188867072},undefined},{{riak_search_vnode,1335903840372778448670555667404727447654250840064},undefined},{{riak_search_vnode,1347321821914426127719021955160323408745312813056},undefined},{{riak_search_vnode,1358739803456073806767488242915919369836374786048},undefined},{{riak_search_vnode,1370157784997721485815954530671515330927436759040},undefined},{{riak_search_vnode,1381575766539369164864420818427111292018498732032},undefined},{{riak_search_vnode,1392993748081016843912887106182707253109560705024},undefined},{{riak_search_vnode,1404411729622664522961353393938303214200622678016},undefined},{{riak_search_vnode,1415829711164312202009819681693899175291684651008},undefined},{{riak_search_vnode,1427247692705959881058285969449495136382746624000},undefined},{{riak_search_vnode,1438665674247607560106752257205091097473808596992},undefined},{{riak_search_vnode,1450083655789255239155218544960687058564870569984},undefined}],[]}
>> > ** Reason for termination ==
>> > **
>> {{badmatch,{error,{{badmatch,{error,emfile}},[{riak_kv_vnode,init,1},{riak_core_vnode,init,1},{gen_fsm,init_it,6},{proc_lib,init_p_do_apply,3}]}}},[{riak_core_vnode_manager,get_vnode,3},{riak_core_vnode_manager,handle_call,3},{gen_server,handle_msg,5},{proc_lib,init_p_do_apply,3}]}
>> > 2012-05-18 04:59:01 =CRASH REPORT====
>> >  crasher:
>> >    initial call: riak_core_vnode_manager:init/1
>> >    pid: <0.22403.463>
>> >    registered_name: riak_core_vnode_manager
>> >    exception exit:
>> {{badmatch,{error,{{badmatch,{error,emfile}},[{riak_kv_vnode,init,1},{riak_core_vnode,init,1},{gen_fsm,init_it,6},{proc_lib,init_p_do_apply,3}]}}},[{riak_core_vnode_manager,get_vnode,3},{riak_core_vnode_manager,handle_call,3},{gen_server,handle_msg,5},{proc_lib,init_p_do_apply,3}]}
>> >      in function  gen_server:terminate/6
>> >      in call from proc_lib:init_p_do_apply/3
>> >    ancestors: [riak_core_sup,<0.91.0>]
>> >    messages: []
>> >    links: [<0.92.0>]
>> >    dictionary: []
>> >    trap_exit: false
>> >    status: running
>> >    heap_size: 17711
>> >    stack_size: 24
>> >    reductions: 170622
>> >  neighbours:
>> > 2012-05-18 04:59:01 =SUPERVISOR REPORT====
>> >     Supervisor: {local,riak_core_sup}
>> >     Context:    child_terminated
>> >     Reason:
>> {{badmatch,{error,{{badmatch,{error,emfile}},[{riak_kv_vnode,init,1},{riak_core_vnode,init,1},{gen_fsm,init_it,6},{proc_lib,init_p_do_apply,3}]}}},[{riak_core_vnode_manager,get_vnode,3},{riak_core_vnode_manager,handle_call,3},{gen_server,handle_msg,5},{proc_lib,init_p_do_apply,3}]}
>> >     Offender:
>> [{pid,<0.22403.463>},{name,riak_core_vnode_manager},{mfargs,{riak_core_vnode_manager,start_link,[]}},{restart_type,permanent},{shutdown,5000},{child_type,worker}]
>> >
>> > 2012-05-18 04:59:01 =CRASH REPORT====
>> >  crasher:
>> >    initial call: riak_core_vnode_proxy:init/1
>> >    pid: <0.21469.398>
>> >    registered_name:
>> proxy_riak_kv_vnode_947692467956757361022701883714464770558143758336
>> >    exception exit:
>> {{{badmatch,{error,{{badmatch,{error,emfile}},[{riak_kv_vnode,init,1},{riak_core_vnode,init,1},{gen_fsm,init_it,6},{proc_lib,init_p_do_apply,3}]}}},[{riak_core_vnode_manager,get_vnode,3},{riak_core_vnode_manager,handle_call,3},{gen_server,handle_msg,5},{proc_lib,init_p_do_apply,3}]},{gen_server,call,[riak_core_vnode_manager,{947692467956757361022701883714464770558143758336,riak_kv_vnode,get_vnode},infinity]}}
>> >      in function  gen_server:call/3
>> >      in call from riak_core_vnode_proxy:get_vnode_pid/1
>> >      in call from riak_core_vnode_proxy:handle_proxy/2
>> >      in call from riak_core_vnode_proxy:loop/2
>> >      in call from proc_lib:init_p_do_apply/3
>> >    ancestors: [riak_core_vnode_proxy_sup,riak_core_sup,<0.91.0>]
>> >    messages: []
>> >    links: [<0.20960.398>]
>> >    dictionary: []
>> >    trap_exit: false
>> >    status: running
>> >    heap_size: 377
>> >    stack_size: 24
>> >    reductions: 89
>> >  neighbours:
>> > 2012-05-18 04:59:01 =SUPERVISOR REPORT====
>> >     Supervisor: {local,riak_core_vnode_proxy_sup}
>> >     Context:    child_terminated
>> >     Reason:
>> {{{badmatch,{error,{{badmatch,{error,emfile}},[{riak_kv_vnode,init,1},{riak_core_vnode,init,1},{gen_fsm,init_it,6},{proc_lib,init_p_do_apply,3}]}}},[{riak_core_vnode_manager,get_vnode,3},{riak_core_vnode_manager,handle_call,3},{gen_server,handle_msg,5},{proc_lib,init_p_do_apply,3}]},{gen_server,call,[riak_core_vnode_manager,{947692467956757361022701883714464770558143758336,riak_kv_vnode,get_vnode},infinity]}}
>> >     Offender:
>> [{pid,<0.21469.398>},{name,{riak_kv_vnode,947692467956757361022701883714464770558143758336}},{mfargs,{riak_core_vnode_proxy,start_link,[riak_kv_vnode,947692467956757361022701883714464770558143758336]}},{restart_type,permanent},{shutdown,5000},{child_type,worker}]
>> >
>> > 2012-05-18 04:59:06 =ERROR REPORT====
>> > ** Generic server riak_kv_stat terminating
>> > ** Last message in was
>> {'$gen_cast',{update,{put_fsm_time,732},63504536346}}
>> > ** When Server state ==
>> {state,{spiral,63504536346,[6,0,0,0,0,35,20,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]},{spiral,63504536346,[2,0,0,0,0,21,14,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]},61,37,{spiral,63504536341,[126,84,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]},210,{spiral,63504536346,[2,0,0,0,0,21,14,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]},37,{spiral,63504536346,[0,0,0,0,0,4,3,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]},7,{spiral,63504536340,[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]},0,{spiral,63504536346,[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]},0,12,8,{slide,63504536340,60,60,"/var/lib/riak/slide-data/3053/1337.331540.24720",{file_descriptor,prim_file,{#Port<0.2186439>,69}},63504536341},{slide,63504536340,60,60,"/var/lib/riak/slide-data/3053/1337.331540.25076",{file_descriptor,prim_file,{#Port<0.2186440>,128}},63504536341},{slide,63504536340,60,60,"/var/lib/riak/slide-data/3053/1337.331540.25785",{file_descriptor,prim_file,{#Port<0.2186444>,40}},63504536346},{slide,63504536340,60,60,"/var/lib/riak/slide-data/3053/1337.331540.26002",{file_descriptor,prim_file,{#Port<0.2186441>,177}},63504536341},{spiral,63504536340,[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]},0,0,{spiral,63504536340,[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]},0,undefined,0,0,undefined,undefined,0,0,true}
>> > ** Reason for termination ==
>> > ** {emfile,[{erlang,open_port,[{spawn,"/bin/sh -s unix:cmd
>> 2>&1"},[stream]]},{os,start_port_srv_handle,1},{os,start_port_srv_loop,0}]}
>> > 2012-05-18 04:59:06 =CRASH REPORT====
>> >  crasher:
>> >    initial call: gen:init_it/7
>> >    pid: <0.22372.463>
>> >    registered_name: riak_kv_stat
>> >    exception exit: {emfile,[{erlang,open_port,[{spawn,"/bin/sh -s
>> unix:cmd
>> 2>&1"},[stream]]},{os,start_port_srv_handle,1},{os,start_port_srv_loop,0}]}
>> >      in function  gen_server2:terminate/6
>> >      in call from proc_lib:init_p_do_apply/3
>> >    ancestors: [riak_kv_sup,<0.323.0>]
>> >    messages:
>> [{'$gen_cast',{update,{vnode_index_write,0,0},63504536346}},{'$gen_cast',{update,vnode_put,63504536346}},{'$gen_cast',{update,{vnode_index_write,1,0},63504536346}},{'$gen_cast',{update,vnode_put,63504536346}},{'$gen_cast',{update,{vnode_index_write,1,0},63504536346}},{'$gen_cast',{update,vnode_put,63504536346}},{'$gen_cast',{update,{vnode_index_write,1,0},63504536346}},{'$gen_cast',{update,vnode_put,63504536346}},{'$gen_cast',{update,{put_fsm_time,5241},63504536346}}]
>> >    links: [#Port<0.2186440>,#Port<0.2186444>,<0.420.0>,#Port<0.2186439>]
>> >    dictionary: []
>> >    trap_exit: true
>> >    status: running
>> >    heap_size: 6765
>> >    stack_size: 24
>> >    reductions: 23738
>> >  neighbours:
>> > 2012-05-18 04:59:06 =SUPERVISOR REPORT====
>> >     Supervisor: {local,riak_kv_sup}
>> >     Context:    child_terminated
>> >     Reason:     {emfile,[{erlang,open_port,[{spawn,"/bin/sh -s unix:cmd
>> 2>&1"},[stream]]},{os,start_port_srv_handle,1},{os,start_port_srv_loop,0}]}
>> >     Offender:
>> [{pid,<0.22372.463>},{name,riak_kv_stat},{mfargs,{riak_kv_stat,start_link,[]}},{restart_type,permanent},{shutdown,5000},{child_type,worker}]
>> >
>> > 2012-05-18 04:59:06 =SUPERVISOR REPORT====
>> >     Supervisor: {local,riak_kv_sup}
>> >     Context:    start_error
>> >     Reason:     {emfile,[{erlang,open_port,[{spawn,"/bin/sh -s unix:cmd
>> 2>&1"},[stream]]},{os,start_port_srv_handle,1},{os,start_port_srv_loop,0}]}
>> >     Offender:
>> [{pid,<0.22372.463>},{name,riak_kv_stat},{mfargs,{riak_kv_stat,start_link,[]}},{restart_type,permanent},{shutdown,5000},{child_type,worker}]
>> >
>> > 2012-05-18 04:59:06 =CRASH REPORT====
>> >  crasher:
>> >    initial call: gen:init_it/7
>> >    pid: <0.22460.463>
>> >    registered_name: []
>> >    exception exit: {emfile,[{erlang,open_port,[{spawn,"/bin/sh -s
>> unix:cmd
>> 2>&1"},[stream]]},{os,start_port_srv_handle,1},{os,start_port_srv_loop,0}]}
>> >      in function  gen_server2:init_it/6
>> >      in call from proc_lib:init_p_do_apply/3
>> >    ancestors: [riak_kv_sup,<0.323.0>]
>> >    messages: []
>> >    links: [<0.420.0>]
>> >    dictionary: []
>> >    trap_exit: true
>> >    status: running
>> >    heap_size: 610
>> >    stack_size: 24
>> >    reductions: 363
>> >  neighbours:
>> > 2012-05-18 04:59:06 =CRASH REPORT====
>> >  crasher:
>> >    initial call: gen:init_it/7
>> >    pid: <0.22462.463>
>> >    registered_name: []
>> >    exception exit: {emfile,[{erlang,open_port,[{spawn,"/bin/sh -s
>> unix:cmd
>> 2>&1"},[stream]]},{os,start_port_srv_handle,1},{os,start_port_srv_loop,0}]}
>> >      in function  gen_server2:init_it/6
>> >      in call from proc_lib:init_p_do_apply/3
>> >    ancestors: [riak_kv_sup,<0.323.0>]
>> >    messages: []
>> >    links: [<0.420.0>]
>> >    dictionary: []
>> >    trap_exit: true
>> >    status: running
>> >    heap_size: 610
>> >    stack_size: 24
>> >    reductions: 367
>> >  neighbours:
>> > 2012-05-18 04:59:06 =SUPERVISOR REPORT====
>> >     Supervisor: {local,riak_kv_sup}
>> >     Context:    start_error
>> >     Reason:     {emfile,[{erlang,open_port,[{spawn,"/bin/sh -s unix:cmd
>> 2>&1"},[stream]]},{os,start_port_srv_handle,1},{os,start_port_srv_loop,0}]}
>> >     Offender:
>> [{pid,<0.22372.463>},{name,riak_kv_stat},{mfargs,{riak_kv_stat,start_link,[]}},{restart_type,permanent},{shutdown,5000},{child_type,worker}]
>> >
>> > 2012-05-18 04:59:06 =SUPERVISOR REPORT====
>> >     Supervisor: {local,riak_kv_sup}
>> >     Context:    start_error
>> >     Reason:     {emfile,[{erlang,open_port,[{spawn,"/bin/sh -s unix:cmd
>> 2>&1"},[stream]]},{os,start_port_srv_handle,1},{os,start_port_srv_loop,0}]}
>> >     Offender:
>> [{pid,<0.22372.463>},{name,riak_kv_stat},{mfargs,{riak_kv_stat,start_link,[]}},{restart_type,permanent},{shutdown,5000},{child_type,worker}]
>> >
>> >
>> > Node 3:
>> >
>> > 2012-05-18 04:58:20 =ERROR REPORT====
>> > ** Generic server riak_core_vnode_manager terminating
>> > ** Last message in was
>> {947692467956757361022701883714464770558143758336,riak_kv_vnode,get_vnode}
>> > ** When Server state ==
>> {state,40798507028,[{{riak_kv_vnode,0},undefined},{{riak_kv_vnode,11417981541647679048466287755595961091061972992},undefined},{{riak_kv_vnode,22835963083295358096932575511191922182123945984},undefined},{{riak_kv_vnode,34253944624943037145398863266787883273185918976},undefined},{{riak_kv_vnode,45671926166590716193865151022383844364247891968},undefined},{{riak_kv_vnode,57089907708238395242331438777979805455309864960},undefined},{{riak_kv_vnode,68507889249886074290797726533575766546371837952},undefined},{{riak_kv_vnode,79925870791533753339264014289171727637433810944},undefined},{{riak_kv_vnode,91343852333181432387730302044767688728495783936},undefined},{{riak_kv_vnode,102761833874829111436196589800363649819557756928},undefined},{{riak_kv_vnode,114179815416476790484662877555959610910619729920},undefined},{{riak_kv_vnode,125597796958124469533129165311555572001681702912},undefined},{{riak_kv_vnode,137015778499772148581595453067151533092743675904},undefined},{{riak_kv_vnode,148433760041419827630061740822747494183805648896},undefined},{{riak_kv_vnode,159851741583067506678528028578343455274867621888},undefined},{{riak_kv_vnode,171269723124715185726994316333939416365929594880},undefined},{{riak_kv_vnode,182687704666362864775460604089535377456991567872},undefined},{{riak_kv_vnode,194105686208010543823926891845131338548053540864},undefined},{{riak_kv_vnode,205523667749658222872393179600727299639115513856},undefined},{{riak_kv_vnode,216941649291305901920859467356323260730177486848},undefined},{{riak_kv_vnode,228359630832953580969325755111919221821239459840},undefined},{{riak_kv_vnode,239777612374601260017792042867515182912301432832},undefined},{{riak_kv_vnode,251195593916248939066258330623111144003363405824},undefined},{{riak_kv_vnode,262613575457896618114724618378707105094425378816},undefined},{{riak_kv_vnode,274031556999544297163190906134303066185487351808},undefined},{{riak_kv_vnode,285449538541191976211657193889899027276549324800},undefined},{{riak_kv_vnode,296867520082839655260123481645494988367611297792},undefined},{{riak_kv_vnode,308285501624487334308589769401090949458673270784},undefined},{{riak_kv_vnode,319703483166135013357056057156686910549735243776},undefined},{{riak_kv_vnode,331121464707782692405522344912282871640797216768},undefined},{{riak_kv_vnode,342539446249430371453988632667878832731859189760},undefined},{{riak_kv_vnode,353957427791078050502454920423474793822921162752},undefined},{{riak_kv_vnode,365375409332725729550921208179070754913983135744},undefined},{{riak_kv_vnode,376793390874373408599387495934666716005045108736},undefined},{{riak_kv_vnode,388211372416021087647853783690262677096107081728},undefined},{{riak_kv_vnode,399629353957668766696320071445858638187169054720},undefined},{{riak_kv_vnode,411047335499316445744786359201454599278231027712},undefined},{{riak_kv_vnode,422465317040964124793252646957050560369293000704},undefined},{{riak_kv_vnode,433883298582611803841718934712646521460354973696},undefined},{{riak_kv_vnode,445301280124259482890185222468242482551416946688},undefined},{{riak_kv_vnode,456719261665907161938651510223838443642478919680},undefined},{{riak_kv_vnode,468137243207554840987117797979434404733540892672},undefined},{{riak_kv_vnode,479555224749202520035584085735030365824602865664},undefined},{{riak_kv_vnode,490973206290850199084050373490626326915664838656},undefined},{{riak_kv_vnode,502391187832497878132516661246222288006726811648},undefined},{{riak_kv_vnode,513809169374145557180982949001818249097788784640},undefined},{{riak_kv_vnode,525227150915793236229449236757414210188850757632},undefined},{{riak_kv_vnode,536645132457440915277915524513010171279912730624},undefined},{{riak_kv_vnode,548063113999088594326381812268606132370974703616},undefined},{{riak_kv_vnode,559481095540736273374848100024202093462036676608},undefined},{{riak_kv_vnode,570899077082383952423314387779798054553098649600},undefined},{{riak_kv_vnode,582317058624031631471780675535394015644160622592},undefined},{{riak_kv_vnode,593735040165679310520246963290989976735222595584},undefined},{{riak_kv_vnode,605153021707326989568713251046585937826284568576},undefined},{{riak_kv_vnode,616571003248974668617179538802181898917346541568},undefined},{{riak_kv_vnode,627988984790622347665645826557777860008408514560},undefined},{{riak_kv_vnode,639406966332270026714112114313373821099470487552},undefined},{{riak_kv_vnode,650824947873917705762578402068969782190532460544},undefined},{{riak_kv_vnode,662242929415565384811044689824565743281594433536},undefined},{{riak_kv_vnode,673660910957213063859510977580161704372656406528},undefined},{{riak_kv_vnode,685078892498860742907977265335757665463718379520},undefined},{{riak_kv_vnode,696496874040508421956443553091353626554780352512},undefined},{{riak_kv_vnode,707914855582156101004909840846949587645842325504},undefined},{{riak_kv_vnode,719332837123803780053376128602545548736904298496},undefined},{{riak_kv_vnode,730750818665451459101842416358141509827966271488},undefined},{{riak_kv_vnode,742168800207099138150308704113737470919028244480},undefined},{{riak_kv_vnode,753586781748746817198774991869333432010090217472},undefined},{{riak_kv_vnode,765004763290394496247241279624929393101152190464},undefined},{{riak_kv_vnode,776422744832042175295707567380525354192214163456},undefined},{{riak_kv_vnode,787840726373689854344173855136121315283276136448},undefined},{{riak_kv_vnode,799258707915337533392640142891717276374338109440},undefined},{{riak_kv_vnode,810676689456985212441106430647313237465400082432},undefined},{{riak_kv_vnode,822094670998632891489572718402909198556462055424},undefined},{{riak_kv_vnode,833512652540280570538039006158505159647524028416},undefined},{{riak_kv_vnode,844930634081928249586505293914101120738586001408},undefined},{{riak_kv_vnode,856348615623575928634971581669697081829647974400},undefined},{{riak_kv_vnode,867766597165223607683437869425293042920709947392},undefined},{{riak_kv_vnode,879184578706871286731904157180889004011771920384},undefined},{{riak_kv_vnode,890602560248518965780370444936484965102833893376},undefined},{{riak_kv_vnode,902020541790166644828836732692080926193895866368},undefined},{{riak_kv_vnode,913438523331814323877303020447676887284957839360},undefined},{{riak_kv_vnode,924856504873462002925769308203272848376019812352},undefined},{{riak_kv_vnode,936274486415109681974235595958868809467081785344},undefined},{{riak_kv_vnode,947692467956757361022701883714464770558143758336},undefined},{{riak_kv_vnode,959110449498405040071168171470060731649205731328},undefined},{{riak_kv_vnode,970528431040052719119634459225656692740267704320},undefined},{{riak_kv_vnode,981946412581700398168100746981252653831329677312},undefined},{{riak_kv_vnode,993364394123348077216567034736848614922391650304},undefined},{{riak_kv_vnode,1004782375664995756265033322492444576013453623296},undefined},{{riak_kv_vnode,1016200357206643435313499610248040537104515596288},undefined},{{riak_kv_vnode,1027618338748291114361965898003636498195577569280},undefined},{{riak_kv_vnode,1039036320289938793410432185759232459286639542272},undefined},{{riak_kv_vnode,1050454301831586472458898473514828420377701515264},undefined},{{riak_kv_vnode,1061872283373234151507364761270424381468763488256},undefined},{{riak_kv_vnode,1073290264914881830555831049026020342559825461248},undefined},{{riak_kv_vnode,1084708246456529509604297336781616303650887434240},undefined},{{riak_kv_vnode,1096126227998177188652763624537212264741949407232},undefined},{{riak_kv_vnode,1107544209539824867701229912292808225833011380224},undefined},{{riak_kv_vnode,1118962191081472546749696200048404186924073353216},undefined},{{riak_kv_vnode,1130380172623120225798162487804000148015135326208},undefined},{{riak_kv_vnode,1141798154164767904846628775559596109106197299200},undefined},{{riak_kv_vnode,1153216135706415583895095063315192070197259272192},undefined},{{riak_kv_vnode,1164634117248063262943561351070788031288321245184},undefined},{{riak_kv_vnode,1176052098789710941992027638826383992379383218176},undefined},{{riak_kv_vnode,1187470080331358621040493926581979953470445191168},undefined},{{riak_kv_vnode,1198888061873006300088960214337575914561507164160},undefined},{{riak_kv_vnode,1210306043414653979137426502093171875652569137152},undefined},{{riak_kv_vnode,1221724024956301658185892789848767836743631110144},undefined},{{riak_kv_vnode,1233142006497949337234359077604363797834693083136},undefined},{{riak_kv_vnode,1244559988039597016282825365359959758925755056128},undefined},{{riak_kv_vnode,1255977969581244695331291653115555720016817029120},undefined},{{riak_kv_vnode,1267395951122892374379757940871151681107879002112},undefined},{{riak_kv_vnode,1278813932664540053428224228626747642198940975104},undefined},{{riak_kv_vnode,1290231914206187732476690516382343603290002948096},undefined},{{riak_kv_vnode,1301649895747835411525156804137939564381064921088},undefined},{{riak_kv_vnode,1313067877289483090573623091893535525472126894080},undefined},{{riak_kv_vnode,1324485858831130769622089379649131486563188867072},undefined},{{riak_kv_vnode,1335903840372778448670555667404727447654250840064},undefined},{{riak_kv_vnode,1347321821914426127719021955160323408745312813056},undefined},{{riak_kv_vnode,1358739803456073806767488242915919369836374786048},undefined},{{riak_kv_vnode,1370157784997721485815954530671515330927436759040},undefined},{{riak_kv_vnode,1381575766539369164864420818427111292018498732032},undefined},{{riak_kv_vnode,1392993748081016843912887106182707253109560705024},undefined},{{riak_kv_vnode,1404411729622664522961353393938303214200622678016},undefined},{{riak_kv_vnode,1415829711164312202009819681693899175291684651008},undefined},{{riak_kv_vnode,1427247692705959881058285969449495136382746624000},undefined},{{riak_kv_vnode,1438665674247607560106752257205091097473808596992},undefined},{{riak_kv_vnode,1450083655789255239155218544960687058564870569984},undefined},{{riak_pipe_vnode,0},undefined},{{riak_pipe_vnode,11417981541647679048466287755595961091061972992},undefined},{{riak_pipe_vnode,22835963083295358096932575511191922182123945984},undefined},{{riak_pipe_vnode,34253944624943037145398863266787883273185918976},undefined},{{riak_pipe_vnode,45671926166590716193865151022383844364247891968},undefined},{{riak_pipe_vnode,57089907708238395242331438777979805455309864960},undefined},{{riak_pipe_vnode,68507889249886074290797726533575766546371837952},undefined},{{riak_pipe_vnode,79925870791533753339264014289171727637433810944},undefined},{{riak_pipe_vnode,91343852333181432387730302044767688728495783936},undefined},{{riak_pipe_vnode,102761833874829111436196589800363649819557756928},undefined},{{riak_pipe_vnode,114179815416476790484662877555959610910619729920},undefined},{{riak_pipe_vnode,125597796958124469533129165311555572001681702912},undefined},{{riak_pipe_vnode,137015778499772148581595453067151533092743675904},undefined},{{riak_pipe_vnode,148433760041419827630061740822747494183805648896},undefined},{{riak_pipe_vnode,159851741583067506678528028578343455274867621888},undefined},{{riak_pipe_vnode,171269723124715185726994316333939416365929594880},undefined},{{riak_pipe_vnode,182687704666362864775460604089535377456991567872},undefined},{{riak_pipe_vnode,194105686208010543823926891845131338548053540864},undefined},{{riak_pipe_vnode,205523667749658222872393179600727299639115513856},undefined},{{riak_pipe_vnode,216941649291305901920859467356323260730177486848},undefined},{{riak_pipe_vnode,228359630832953580969325755111919221821239459840},undefined},{{riak_pipe_vnode,239777612374601260017792042867515182912301432832},undefined},{{riak_pipe_vnode,251195593916248939066258330623111144003363405824},undefined},{{riak_pipe_vnode,262613575457896618114724618378707105094425378816},undefined},{{riak_pipe_vnode,274031556999544297163190906134303066185487351808},undefined},{{riak_pipe_vnode,285449538541191976211657193889899027276549324800},undefined},{{riak_pipe_vnode,296867520082839655260123481645494988367611297792},undefined},{{riak_pipe_vnode,308285501624487334308589769401090949458673270784},undefined},{{riak_pipe_vnode,319703483166135013357056057156686910549735243776},undefined},{{riak_pipe_vnode,331121464707782692405522344912282871640797216768},undefined},{{riak_pipe_vnode,342539446249430371453988632667878832731859189760},undefined},{{riak_pipe_vnode,353957427791078050502454920423474793822921162752},undefined},{{riak_pipe_vnode,365375409332725729550921208179070754913983135744},undefined},{{riak_pipe_vnode,376793390874373408599387495934666716005045108736},undefined},{{riak_pipe_vnode,388211372416021087647853783690262677096107081728},undefined},{{riak_pipe_vnode,399629353957668766696320071445858638187169054720},undefined},{{riak_pipe_vnode,411047335499316445744786359201454599278231027712},undefined},{{riak_pipe_vnode,422465317040964124793252646957050560369293000704},undefined},{{riak_pipe_vnode,433883298582611803841718934712646521460354973696},undefined},{{riak_pipe_vnode,445301280124259482890185222468242482551416946688},undefined},{{riak_pipe_vnode,456719261665907161938651510223838443642478919680},undefined},{{riak_pipe_vnode,468137243207554840987117797979434404733540892672},undefined},{{riak_pipe_vnode,479555224749202520035584085735030365824602865664},undefined},{{riak_pipe_vnode,490973206290850199084050373490626326915664838656},undefined},{{riak_pipe_vnode,502391187832497878132516661246222288006726811648},undefined},{{riak_pipe_vnode,513809169374145557180982949001818249097788784640},undefined},{{riak_pipe_vnode,525227150915793236229449236757414210188850757632},undefined},{{riak_pipe_vnode,536645132457440915277915524513010171279912730624},undefined},{{riak_pipe_vnode,548063113999088594326381812268606132370974703616},undefined},{{riak_pipe_vnode,559481095540736273374848100024202093462036676608},undefined},{{riak_pipe_vnode,570899077082383952423314387779798054553098649600},undefined},{{riak_pipe_vnode,582317058624031631471780675535394015644160622592},undefined},{{riak_pipe_vnode,593735040165679310520246963290989976735222595584},undefined},{{riak_pipe_vnode,605153021707326989568713251046585937826284568576},undefined},{{riak_pipe_vnode,616571003248974668617179538802181898917346541568},undefined},{{riak_pipe_vnode,627988984790622347665645826557777860008408514560},undefined},{{riak_pipe_vnode,639406966332270026714112114313373821099470487552},undefined},{{riak_pipe_vnode,650824947873917705762578402068969782190532460544},undefined},{{riak_pipe_vnode,662242929415565384811044689824565743281594433536},undefined},{{riak_pipe_vnode,673660910957213063859510977580161704372656406528},undefined},{{riak_pipe_vnode,685078892498860742907977265335757665463718379520},undefined},{{riak_pipe_vnode,696496874040508421956443553091353626554780352512},undefined},{{riak_pipe_vnode,707914855582156101004909840846949587645842325504},undefined},{{riak_pipe_vnode,719332837123803780053376128602545548736904298496},undefined},{{riak_pipe_vnode,730750818665451459101842416358141509827966271488},undefined},{{riak_pipe_vnode,742168800207099138150308704113737470919028244480},undefined},{{riak_pipe_vnode,753586781748746817198774991869333432010090217472},undefined},{{riak_pipe_vnode,765004763290394496247241279624929393101152190464},undefined},{{riak_pipe_vnode,776422744832042175295707567380525354192214163456},undefined},{{riak_pipe_vnode,787840726373689854344173855136121315283276136448},undefined},{{riak_pipe_vnode,799258707915337533392640142891717276374338109440},undefined},{{riak_pipe_vnode,810676689456985212441106430647313237465400082432},undefined},{{riak_pipe_vnode,822094670998632891489572718402909198556462055424},undefined},{{riak_pipe_vnode,833512652540280570538039006158505159647524028416},undefined},{{riak_pipe_vnode,844930634081928249586505293914101120738586001408},undefined},{{riak_pipe_vnode,856348615623575928634971581669697081829647974400},undefined},{{riak_pipe_vnode,867766597165223607683437869425293042920709947392},undefined},{{riak_pipe_vnode,879184578706871286731904157180889004011771920384},undefined},{{riak_pipe_vnode,890602560248518965780370444936484965102833893376},undefined},{{riak_pipe_vnode,902020541790166644828836732692080926193895866368},undefined},{{riak_pipe_vnode,913438523331814323877303020447676887284957839360},undefined},{{riak_pipe_vnode,924856504873462002925769308203272848376019812352},undefined},{{riak_pipe_vnode,936274486415109681974235595958868809467081785344},undefined},{{riak_pipe_vnode,947692467956757361022701883714464770558143758336},undefined},{{riak_pipe_vnode,959110449498405040071168171470060731649205731328},undefined},{{riak_pipe_vnode,970528431040052719119634459225656692740267704320},undefined},{{riak_pipe_vnode,981946412581700398168100746981252653831329677312},undefined},{{riak_pipe_vnode,993364394123348077216567034736848614922391650304},undefined},{{riak_pipe_vnode,1004782375664995756265033322492444576013453623296},undefined},{{riak_pipe_vnode,1016200357206643435313499610248040537104515596288},undefined},{{riak_pipe_vnode,1027618338748291114361965898003636498195577569280},undefined},{{riak_pipe_vnode,1039036320289938793410432185759232459286639542272},undefined},{{riak_pipe_vnode,1050454301831586472458898473514828420377701515264},undefined},{{riak_pipe_vnode,1061872283373234151507364761270424381468763488256},undefined},{{riak_pipe_vnode,1073290264914881830555831049026020342559825461248},undefined},{{riak_pipe_vnode,1084708246456529509604297336781616303650887434240},undefined},{{riak_pipe_vnode,1096126227998177188652763624537212264741949407232},undefined},{{riak_pipe_vnode,1107544209539824867701229912292808225833011380224},undefined},{{riak_pipe_vnode,1118962191081472546749696200048404186924073353216},undefined},{{riak_pipe_vnode,1130380172623120225798162487804000148015135326208},undefined},{{riak_pipe_vnode,1141798154164767904846628775559596109106197299200},undefined},{{riak_pipe_vnode,1153216135706415583895095063315192070197259272192},undefined},{{riak_pipe_vnode,1164634117248063262943561351070788031288321245184},undefined},{{riak_pipe_vnode,1176052098789710941992027638826383992379383218176},undefined},{{riak_pipe_vnode,1187470080331358621040493926581979953470445191168},undefined},{{riak_pipe_vnode,1198888061873006300088960214337575914561507164160},undefined},{{riak_pipe_vnode,1210306043414653979137426502093171875652569137152},undefined},{{riak_pipe_vnode,1221724024956301658185892789848767836743631110144},undefined},{{riak_pipe_vnode,1233142006497949337234359077604363797834693083136},undefined},{{riak_pipe_vnode,1244559988039597016282825365359959758925755056128},undefined},{{riak_pipe_vnode,1255977969581244695331291653115555720016817029120},undefined},{{riak_pipe_vnode,1267395951122892374379757940871151681107879002112},undefined},{{riak_pipe_vnode,1278813932664540053428224228626747642198940975104},undefined},{{riak_pipe_vnode,1290231914206187732476690516382343603290002948096},undefined},{{riak_pipe_vnode,1301649895747835411525156804137939564381064921088},undefined},{{riak_pipe_vnode,1313067877289483090573623091893535525472126894080},undefined},{{riak_pipe_vnode,1324485858831130769622089379649131486563188867072},undefined},{{riak_pipe_vnode,1335903840372778448670555667404727447654250840064},undefined},{{riak_pipe_vnode,1347321821914426127719021955160323408745312813056},undefined},{{riak_pipe_vnode,1358739803456073806767488242915919369836374786048},undefined},{{riak_pipe_vnode,1370157784997721485815954530671515330927436759040},undefined},{{riak_pipe_vnode,1381575766539369164864420818427111292018498732032},undefined},{{riak_pipe_vnode,1392993748081016843912887106182707253109560705024},undefined},{{riak_pipe_vnode,1404411729622664522961353393938303214200622678016},undefined},{{riak_pipe_vnode,1415829711164312202009819681693899175291684651008},undefined},{{riak_pipe_vnode,1427247692705959881058285969449495136382746624000},undefined},{{riak_pipe_vnode,1438665674247607560106752257205091097473808596992},undefined},{{riak_pipe_vnode,1450083655789255239155218544960687058564870569984},undefined},{{riak_search_vnode,0},undefined},{{riak_search_vnode,11417981541647679048466287755595961091061972992},undefined},{{riak_search_vnode,22835963083295358096932575511191922182123945984},undefined},{{riak_search_vnode,34253944624943037145398863266787883273185918976},undefined},{{riak_search_vnode,45671926166590716193865151022383844364247891968},undefined},{{riak_search_vnode,57089907708238395242331438777979805455309864960},undefined},{{riak_search_vnode,68507889249886074290797726533575766546371837952},undefined},{{riak_search_vnode,79925870791533753339264014289171727637433810944},undefined},{{riak_search_vnode,91343852333181432387730302044767688728495783936},undefined},{{riak_search_vnode,102761833874829111436196589800363649819557756928},undefined},{{riak_search_vnode,114179815416476790484662877555959610910619729920},undefined},{{riak_search_vnode,125597796958124469533129165311555572001681702912},undefined},{{riak_search_vnode,137015778499772148581595453067151533092743675904},undefined},{{riak_search_vnode,148433760041419827630061740822747494183805648896},undefined},{{riak_search_vnode,159851741583067506678528028578343455274867621888},undefined},{{riak_search_vnode,171269723124715185726994316333939416365929594880},undefined},{{riak_search_vnode,182687704666362864775460604089535377456991567872},undefined},{{riak_search_vnode,194105686208010543823926891845131338548053540864},undefined},{{riak_search_vnode,205523667749658222872393179600727299639115513856},undefined},{{riak_search_vnode,216941649291305901920859467356323260730177486848},undefined},{{riak_search_vnode,228359630832953580969325755111919221821239459840},undefined},{{riak_search_vnode,239777612374601260017792042867515182912301432832},undefined},{{riak_search_vnode,251195593916248939066258330623111144003363405824},undefined},{{riak_search_vnode,262613575457896618114724618378707105094425378816},undefined},{{riak_search_vnode,274031556999544297163190906134303066185487351808},undefined},{{riak_search_vnode,285449538541191976211657193889899027276549324800},undefined},{{riak_search_vnode,296867520082839655260123481645494988367611297792},undefined},{{riak_search_vnode,308285501624487334308589769401090949458673270784},undefined},{{riak_search_vnode,319703483166135013357056057156686910549735243776},undefined},{{riak_search_vnode,331121464707782692405522344912282871640797216768},undefined},{{riak_search_vnode,342539446249430371453988632667878832731859189760},undefined},{{riak_search_vnode,353957427791078050502454920423474793822921162752},undefined},{{riak_search_vnode,365375409332725729550921208179070754913983135744},undefined},{{riak_search_vnode,376793390874373408599387495934666716005045108736},undefined},{{riak_search_vnode,388211372416021087647853783690262677096107081728},undefined},{{riak_search_vnode,399629353957668766696320071445858638187169054720},undefined},{{riak_search_vnode,411047335499316445744786359201454599278231027712},undefined},{{riak_search_vnode,422465317040964124793252646957050560369293000704},undefined},{{riak_search_vnode,433883298582611803841718934712646521460354973696},undefined},{{riak_search_vnode,445301280124259482890185222468242482551416946688},undefined},{{riak_search_vnode,456719261665907161938651510223838443642478919680},undefined},{{riak_search_vnode,468137243207554840987117797979434404733540892672},undefined},{{riak_search_vnode,479555224749202520035584085735030365824602865664},undefined},{{riak_search_vnode,490973206290850199084050373490626326915664838656},undefined},{{riak_search_vnode,502391187832497878132516661246222288006726811648},undefined},{{riak_search_vnode,513809169374145557180982949001818249097788784640},undefined},{{riak_search_vnode,525227150915793236229449236757414210188850757632},undefined},{{riak_search_vnode,536645132457440915277915524513010171279912730624},undefined},{{riak_search_vnode,548063113999088594326381812268606132370974703616},undefined},{{riak_search_vnode,559481095540736273374848100024202093462036676608},undefined},{{riak_search_vnode,570899077082383952423314387779798054553098649600},undefined},{{riak_search_vnode,582317058624031631471780675535394015644160622592},undefined},{{riak_search_vnode,593735040165679310520246963290989976735222595584},undefined},{{riak_search_vnode,605153021707326989568713251046585937826284568576},undefined},{{riak_search_vnode,616571003248974668617179538802181898917346541568},undefined},{{riak_search_vnode,627988984790622347665645826557777860008408514560},undefined},{{riak_search_vnode,639406966332270026714112114313373821099470487552},undefined},{{riak_search_vnode,650824947873917705762578402068969782190532460544},undefined},{{riak_search_vnode,662242929415565384811044689824565743281594433536},undefined},{{riak_search_vnode,673660910957213063859510977580161704372656406528},undefined},{{riak_search_vnode,685078892498860742907977265335757665463718379520},undefined},{{riak_search_vnode,696496874040508421956443553091353626554780352512},undefined},{{riak_search_vnode,707914855582156101004909840846949587645842325504},undefined},{{riak_search_vnode,719332837123803780053376128602545548736904298496},undefined},{{riak_search_vnode,730750818665451459101842416358141509827966271488},undefined},{{riak_search_vnode,742168800207099138150308704113737470919028244480},undefined},{{riak_search_vnode,753586781748746817198774991869333432010090217472},undefined},{{riak_search_vnode,765004763290394496247241279624929393101152190464},undefined},{{riak_search_vnode,776422744832042175295707567380525354192214163456},undefined},{{riak_search_vnode,787840726373689854344173855136121315283276136448},undefined},{{riak_search_vnode,799258707915337533392640142891717276374338109440},undefined},{{riak_search_vnode,810676689456985212441106430647313237465400082432},undefined},{{riak_search_vnode,822094670998632891489572718402909198556462055424},undefined},{{riak_search_vnode,833512652540280570538039006158505159647524028416},undefined},{{riak_search_vnode,844930634081928249586505293914101120738586001408},undefined},{{riak_search_vnode,856348615623575928634971581669697081829647974400},undefined},{{riak_search_vnode,867766597165223607683437869425293042920709947392},undefined},{{riak_search_vnode,879184578706871286731904157180889004011771920384},undefined},{{riak_search_vnode,890602560248518965780370444936484965102833893376},undefined},{{riak_search_vnode,902020541790166644828836732692080926193895866368},undefined},{{riak_search_vnode,913438523331814323877303020447676887284957839360},undefined},{{riak_search_vnode,924856504873462002925769308203272848376019812352},undefined},{{riak_search_vnode,936274486415109681974235595958868809467081785344},undefined},{{riak_search_vnode,947692467956757361022701883714464770558143758336},undefined},{{riak_search_vnode,959110449498405040071168171470060731649205731328},undefined},{{riak_search_vnode,970528431040052719119634459225656692740267704320},undefined},{{riak_search_vnode,981946412581700398168100746981252653831329677312},undefined},{{riak_search_vnode,993364394123348077216567034736848614922391650304},undefined},{{riak_search_vnode,1004782375664995756265033322492444576013453623296},undefined},{{riak_search_vnode,1016200357206643435313499610248040537104515596288},undefined},{{riak_search_vnode,1027618338748291114361965898003636498195577569280},undefined},{{riak_search_vnode,1039036320289938793410432185759232459286639542272},undefined},{{riak_search_vnode,1050454301831586472458898473514828420377701515264},undefined},{{riak_search_vnode,1061872283373234151507364761270424381468763488256},undefined},{{riak_search_vnode,1073290264914881830555831049026020342559825461248},undefined},{{riak_search_vnode,1084708246456529509604297336781616303650887434240},undefined},{{riak_search_vnode,1096126227998177188652763624537212264741949407232},undefined},{{riak_search_vnode,1107544209539824867701229912292808225833011380224},undefined},{{riak_search_vnode,1118962191081472546749696200048404186924073353216},undefined},{{riak_search_vnode,1130380172623120225798162487804000148015135326208},undefined},{{riak_search_vnode,1141798154164767904846628775559596109106197299200},undefined},{{riak_search_vnode,1153216135706415583895095063315192070197259272192},undefined},{{riak_search_vnode,1164634117248063262943561351070788031288321245184},undefined},{{riak_search_vnode,1176052098789710941992027638826383992379383218176},undefined},{{riak_search_vnode,1187470080331358621040493926581979953470445191168},undefined},{{riak_search_vnode,1198888061873006300088960214337575914561507164160},undefined},{{riak_search_vnode,1210306043414653979137426502093171875652569137152},undefined},{{riak_search_vnode,1221724024956301658185892789848767836743631110144},undefined},{{riak_search_vnode,1233142006497949337234359077604363797834693083136},undefined},{{riak_search_vnode,1244559988039597016282825365359959758925755056128},undefined},{{riak_search_vnode,1255977969581244695331291653115555720016817029120},undefined},{{riak_search_vnode,1267395951122892374379757940871151681107879002112},undefined},{{riak_search_vnode,1278813932664540053428224228626747642198940975104},undefined},{{riak_search_vnode,1290231914206187732476690516382343603290002948096},undefined},{{riak_search_vnode,1301649895747835411525156804137939564381064921088},undefined},{{riak_search_vnode,1313067877289483090573623091893535525472126894080},undefined},{{riak_search_vnode,1324485858831130769622089379649131486563188867072},undefined},{{riak_search_vnode,1335903840372778448670555667404727447654250840064},undefined},{{riak_search_vnode,1347321821914426127719021955160323408745312813056},undefined},{{riak_search_vnode,1358739803456073806767488242915919369836374786048},undefined},{{riak_search_vnode,1370157784997721485815954530671515330927436759040},undefined},{{riak_search_vnode,1381575766539369164864420818427111292018498732032},undefined},{{riak_search_vnode,1392993748081016843912887106182707253109560705024},undefined},{{riak_search_vnode,1404411729622664522961353393938303214200622678016},undefined},{{riak_search_vnode,1415829711164312202009819681693899175291684651008},undefined},{{riak_search_vnode,1427247692705959881058285969449495136382746624000},undefined},{{riak_search_vnode,1438665674247607560106752257205091097473808596992},undefined},{{riak_search_vnode,1450083655789255239155218544960687058564870569984},undefined}],[]}
>> > ** Reason for termination ==
>> > **
>> {{badmatch,{error,{{badmatch,{error,emfile}},[{riak_kv_vnode,init,1},{riak_core_vnode,init,1},{gen_fsm,init_it,6},{proc_lib,init_p_do_apply,3}]}}},[{riak_core_vnode_manager,get_vnode,3},{riak_core_vnode_manager,handle_call,3},{gen_server,handle_msg,5},{proc_lib,init_p_do_apply,3}]}
>> > 2012-05-18 04:58:20 =CRASH REPORT====
>> >  crasher:
>> >    initial call: riak_core_vnode_manager:init/1
>> >    pid: <0.15696.398>
>> >    registered_name: riak_core_vnode_manager
>> >    exception exit:
>> {{badmatch,{error,{{badmatch,{error,emfile}},[{riak_kv_vnode,init,1},{riak_core_vnode,init,1},{gen_fsm,init_it,6},{proc_lib,init_p_do_apply,3}]}}},[{riak_core_vnode_manager,get_vnode,3},{riak_core_vnode_manager,handle_call,3},{gen_server,handle_msg,5},{proc_lib,init_p_do_apply,3}]}
>> >      in function  gen_server:terminate/6
>> >      in call from proc_lib:init_p_do_apply/3
>> >    ancestors: [riak_core_sup,<0.91.0>]
>> >    messages: []
>> >    links: [<0.92.0>]
>> >    dictionary: []
>> >    trap_exit: false
>> >    status: running
>> >    heap_size: 17711
>> >    stack_size: 24
>> >    reductions: 170595
>> >  neighbours:
>> > 2012-05-18 04:58:20 =CRASH REPORT====
>> >  crasher:
>> >    initial call: riak_core_vnode_proxy:init/1
>> >    pid: <0.15703.398>
>> >    registered_name:
>> proxy_riak_kv_vnode_947692467956757361022701883714464770558143758336
>> >    exception exit:
>> {{{badmatch,{error,{{badmatch,{error,emfile}},[{riak_kv_vnode,init,1},{riak_core_vnode,init,1},{gen_fsm,init_it,6},{proc_lib,init_p_do_apply,3}]}}},[{riak_core_vnode_manager,get_vnode,3},{riak_core_vnode_manager,handle_call,3},{gen_server,handle_msg,5},{proc_lib,init_p_do_apply,3}]},{gen_server,call,[riak_core_vnode_manager,{947692467956757361022701883714464770558143758336,riak_kv_vnode,get_vnode},infinity]}}
>> >      in function  gen_server:call/3
>> >      in call from riak_core_vnode_proxy:get_vnode_pid/1
>> >      in call from riak_core_vnode_proxy:handle_proxy/2
>> >      in call from riak_core_vnode_proxy:loop/2
>> >      in call from proc_lib:init_p_do_apply/3
>> >    ancestors: [riak_core_vnode_proxy_sup,riak_core_sup,<0.91.0>]
>> >    messages: []
>> >    links: [<0.107.0>]
>> >    dictionary: []
>> >    trap_exit: false
>> >    status: running
>> >    heap_size: 377
>> >    stack_size: 24
>> >    reductions: 89
>> >  neighbours:
>> > 2012-05-18 04:58:20 =SUPERVISOR REPORT====
>> >     Supervisor: {local,riak_core_sup}
>> >     Context:    child_terminated
>> >     Reason:
>> {{badmatch,{error,{{badmatch,{error,emfile}},[{riak_kv_vnode,init,1},{riak_core_vnode,init,1},{gen_fsm,init_it,6},{proc_lib,init_p_do_apply,3}]}}},[{riak_core_vnode_manager,get_vnode,3},{riak_core_vnode_manager,handle_call,3},{gen_server,handle_msg,5},{proc_lib,init_p_do_apply,3}]}
>> >     Offender:
>> [{pid,<0.15696.398>},{name,riak_core_vnode_manager},{mfargs,{riak_core_vnode_manager,start_link,[]}},{restart_type,permanent},{shutdown,5000},{child_type,worker}]
>> >
>> > 2012-05-18 04:58:20 =SUPERVISOR REPORT====
>> >     Supervisor: {local,riak_core_vnode_proxy_sup}
>> >     Context:    child_terminated
>> >     Reason:
>> {{{badmatch,{error,{{badmatch,{error,emfile}},[{riak_kv_vnode,init,1},{riak_core_vnode,init,1},{gen_fsm,init_it,6},{proc_lib,init_p_do_apply,3}]}}},[{riak_core_vnode_manager,get_vnode,3},{riak_core_vnode_manager,handle_call,3},{gen_server,handle_msg,5},{proc_lib,init_p_do_apply,3}]},{gen_server,call,[riak_core_vnode_manager,{947692467956757361022701883714464770558143758336,riak_kv_vnode,get_vnode},infinity]}}
>> >     Offender:
>> [{pid,<0.15703.398>},{name,{riak_kv_vnode,947692467956757361022701883714464770558143758336}},{mfargs,{riak_core_vnode_proxy,start_link,[riak_kv_vnode,947692467956757361022701883714464770558143758336]}},{restart_type,permanent},{shutdown,5000},{child_type,worker}]
>> >
>> > 2012-05-18 04:58:20 =SUPERVISOR REPORT====
>> >     Supervisor: {local,riak_core_vnode_proxy_sup}
>> >     Context:    shutdown
>> >     Reason:     reached_max_restart_intensity
>> >     Offender:
>> [{pid,<0.15703.398>},{name,{riak_kv_vnode,947692467956757361022701883714464770558143758336}},{mfargs,{riak_core_vnode_proxy,start_link,[riak_kv_vnode,947692467956757361022701883714464770558143758336]}},{restart_type,permanent},{shutdown,5000},{child_type,worker}]
>> >
>> > 2012-05-18 04:58:20 =ERROR REPORT====
>> > ** Generic server riak_kv_vnode_master terminating
>> > ** Last message in was
>> {'$gen_cast',{riak_vnode_req_v1,947692467956757361022701883714464770558143758336,{fsm,undefined,<11602.10975.494>},{riak_kv_put_req_v1,{<<"_rsid_pages">>,<<"K4fUuIQqNsUHdH2rNqMzk6AKSot">>},{r_object,<<"_rsid_pages">>,<<"K4fUuIQqNsUHdH2rNqMzk6AKSot">>,[{r_content,{dict,3,16,16,8,80,48,{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},{{[],[],[],[],[],[],[],[],[],[],[[<<"X-Riak-VTag">>,53,49,103,55,99,86,83,98,79,82,83,73,110,68,56,84,109,118,55,89,71,72]],[[<<"index">>]],[],[[<<"X-Riak-Last-Modified">>|{1337,331480,246971}]],[],[]}}},{riak_idx_doc,<<"pages">>,<<"K4fUuIQqNsUHdH2rNqMzk6AKSot">>,[{<<"_type">>,<<"Page">>,[{<<"Page">>,[0]}]},{<<"classification">>,<<"0">>,[{<<"0000000000">>,[0]}]},{<<"contact_email">>,<<"
>> david at serieaticketsonline.com<mailto:david at serieaticketsonline.com
>> >">>,[{<<"david">>,[0]},{<<"serieaticketsonline.com<
>> http://serieaticketsonline.com
>> >">>,[1]}]},{<<"created_at">>,<<"2012-05-18">>,[{<<"2012-05-18">>,[0]}]},{<<"domain">>,<<"
>> serieaticketsonline.com<http://serieaticketsonline.com>">>,[{<<"
>> serieaticketsonline.com<http://serieaticketsonline.com>">>,[0]}]},{<<"outbound_links">>,<<"16">>,[{<<"0000000016">>,[0]}]},{<<"revaluate">>,<<"false">>,[]},{<<"search_terms">>,<<"44
>> 999 This Day and Age Safetysuit Civil Twilight Reign of Kindo As Tall as
>> Lions Copeland Alpha Rev Secondhand Serenade 38th Parallel Sent By Ravens
>> Boys Like Girls Manic Drive P.O.D Ivoryline We Shot the Moon Deas Vail Neon
>> Horse Grey Holiday Jon Mclaughlin Jonezetta Pillar Barcelona Mutemath Lydia
>> Jonsi Matt Nathanson Rocket Summer Rapture Ruckus My Morning Jacket
>> Doubledge Hey Monday Newworldson Above the Golden State Matt Wertz James
>> Morrison VersaEmerge Thousand Foot Krutch The Drawing Room Manafest Jimmy
>> Needham Waking Ashland Flyleaf Falling Up A Dream to the Moon Since October
>> Fireflight Stereophonics The Fray Arcade Fire Mat Kearney Turn off the
>> Stars Skillet Snow Patrol Shiny Toy Guns Lifehouse Children 18:3
>> Celldweller Article One HB Abandon Lost Ocean Staple Spoken Ever Stays Red
>> This Beautiful Republic The Send Oceans Above Needtobreathe The Beatles
>> Breaking Benjamin I See Stars Radiohead Muse AFI Alkaline Trio Pink Floyd
>> U2 REM Blues Traveler Creedence Clearwater Revival Poets of the Fall
>> Disciple Emarosa Evanescence Fall Out Boy Underoath The Killers Linkin Park
>> My Chemical Romance Red Hot Chili Peppers Simon and Garfunkel Sixpence
>> (None The Richer) System of a Down Weezer The Who Tokio Hotel Paramore
>> Green Day The Cure Depeche Mode Black Veil Brides Korn Oomph! 3 Days Grace
>> Escape the Fate The All-American Rejects All Time Low 30 Seconds to Mars
>> Plain White T's Kings of Leon Silversun Pickups Nirvana Kerli David Bowie
>> ACDC LED ZEPLIN BEATLES FOO FIGHTERS GUNS N ROSES PINK FLYOD THE JIMI
>> HENDRIX EXPERIANCE BLACK SABBATH IRON MAIDEN THE ROLLING STONES K-ROCK
>> AEROSMITH THE BLACK KEYS THE CLASH PEARL JAM THE ANIMALS THE DOORS BON JOVI
>> IGGY AND THE STOOGES VAMPIRE WEEKEND DRAGON FORCE WARRANT QUEEN DEEP PURPLE
>> THE RAMONES KISS THE POLICE COLDPLAY MOTORHEARDS black veiled brides
>> misfits ten years trapt kid cudi neon trees silverstein velvet revolver
>> hoobastank adema seether shinedown 3 doors down default whitesnake motley
>> crue panic at the disco disterbed anberlin lovehatehero slipknot rise
>> against ateyu Cream Tokyo Police Club MGMT Passion Pit Animal Collective
>> Cocoon Death Cab for Cutie Def Leopard Dirty Projectors Elton John Keith
>> Urban Black Eyed Peas Florence and the Machine The Postal Service Chromeo
>> Freelance Whales Blind Pilot The Man Beach House Ghostland Observatory The
>> Temper Trap Hall & Oates Led Zeppelin Santana Van Morrison Journey The Pet
>> Shop Boys Lynayrd Skynard ZZ Top Sufjan Stevens The Welcome Wagon Lionel
>> Richie Pablo Cruise Discovery Tears for Fears El Ten Eleven Prince Stevie
>> Wonder Fleetwood Mac Bjork Tom Petty and the Heartbreakers Wolf Parade
>> Shout Out Louds Yeah Yeah Yeahs Keane Matt & Kim She and Him Frans
>> Ferdinand">>,[{<<"moon">>,[30,81]},{<<"journey">>,[354]},{<<"who">>,[173]},{<<"motley">>,[289]},{<<"club">>,[304]},{<<"needtobreathe">>,"z"},{<<"rolling">>,[231]},{<<"him">>,[401]},{<<"beatles">>,[124,215]},{<<"barcelona">>,")"},{<<"jimi">>,[223]},{<<"iron">>,[228]},{<<"hoobastank">>,[281]},{<<"fray">>,"W"},{<<"eyed">>,[324]},{<<"dirty">>,[317]},{<<"petty">>,[384]},{<<"staple">>,"n"},{<<"kerli">>,[209]},{<<"abandon">>,"k"},{<<"republic">>,"u"},{<<"mode">>,[182]},{<<"doubledge">>,"5"},{<<"john">>,[320]},{<<"spoken">>,"o"},{<<"bon">>,[246]},{<<"pilot">>,[337]},{<<"999">>,[0]},{<<"day">>,[2,178]},{<<"parallel">>,[18]},{<<"pillar">>,"("},{<<"santana">>,[351]},{<<"versaemerge">>,"A"},{<<"passion">>,[306]},{<<"skillet">>,"`"},{<<"whales">>,[335]},{<<"animal">>,[308]},{<<"house">>,[341]},{<<"observatory">>,[343]},{<<"ruckus">>,"2"},{<<"manafest">>,"H"},{<<"falling">>,"N"},{<<"beautiful">>,"t"},{<<"weezer">>,[171]},{<<"collective">>,[309]},{<<"cruise">>,[370]},{<<"sufjan">>,[362]},{<<"force">>,[255]},{<<"roses">>,[219]},{<<"ten">>,[271,375]},{<<"ever">>,"p"},{<<"tom">>,[383]},{<<"krutch">>,"D"},{<<"kings">>,[204]},{<<"blind">>,[336]},{<<"rem">>,[136]},{<<"clearwater">>,[140]},{<<"one">>,"j"},{<<"man">>,[339]},{<<"silverstein">>,[278]},{<<"dream">>,"O"},{<<"kearney">>,"["},{<<"jonezetta">>,"'"},{<<"cure">>,[180]},{<<"clash">>,[239]},{<<"shop">>,[357]},{<<"kid">>,[274]},{<<"stars">>,[95,128]},{<<"warrant">>,[256]},{<<"copeland">>,[12]},{<<"experiance">>,[225]},{<<"boys">>,[21,358]},{<<"stereophonics">>,"U"},{<<"trio">>,[133]},{<<"louds">>,[392]},{<<"weekend">>,[253]},{<<"animals">>,[243]},{<<"peas">>,[325]},{<<"tears">>,[372]},{<<"manic">>,[24]},{<<"down">>,[170,286]},{<<"maiden">>,[229]},{<<"misfits">>,[270]},{<<"wonder">>,[379]},{<<"summer">>,"0"},{<<"patrol">>,"b"},{<<"jonsi">>,","},{<<"stooges">>,[251]},{<<"shot">>,[28]},{<<"vail">>,"
>> "},{<<"out">>,[149,391]},{<<"simon">>,[162]},{<<"beach">>,[340]},{<<"shiny">>,"c"},{<<"default">>,[287]},{<<"oceans">>,"x"},{<<"low">>,[199]},{<<"police">>,[264,303]},{<<"sent">>,[19]},{<<"sabbath">>,[227]},{<<"cab">>,[312]},{<<"rejects">>,[196]},{<<"discovery">>,[371]},{<<"breaking">>,"}"},{<<"golden">>,";"},{<<"since">>,"R"},{<<"parade">>,[389]},{<<"pink">>,[134,220]},{<<"welcome">>,[365]},{<<"kindo">>,"\t"},{<<"death">>,[311]},{<<"postal">>,[331]},{<<"time">>,[198]},{<<"matt">>,[45,61,397]},{<<"snow">>,"a"},{<<"linkin">>,[154]},{<<"pickups">>,[207]},{<<"hendrix">>,[224]},{<<"disterbed">>,[294]},{<<"prince">>,[377]},{<<"mac">>,[381]},{<<"sixpence">>,[165]},{<<"garfunkel">>,[164]},{<<"green">>,[177]},{<<"queen">>,[257]},{<<"pablo">>,[369]},{<<"trapt">>,[273]},{<<"mutemath">>,"*"},{<<"rocket">>,"/"},{<<"underoath">>,[151]},{<<"ghostland">>,[342]},{<<"system">>,[169]},{<<"cudi">>,[275]},{<<"rise">>,[298]},{<<"led">>,[213,349]},{<<"state">>,"<"},{<<"room">>,"G"},{<<"romance">>,[157]},{<<"richer">>,[168]},{<<"rev">>,[14]},{<<"flyleaf">>,"M"},{<<"years">>,[272]},{<<"slipknot">>,[297]},{<<"cream">>,[301]},{<<"needham">>,"J"},{<<"twilight">>,[7]},{<<"article">>,"i"},{<<"evanescence">>,[147]},{<<"flyod">>,[221]},{<<"trees">>,[277]},{<<"anberlin">>,[295]},{<<"benjamin">>,"~"},{<<"service">>,[332]},{<<"thousand">>,"B"},{<<"shout">>,[390]},{<<"nathanson">>,"."},{<<"ashland">>,"L"},{<<"age">>,[4]},{<<"ocean">>,"m"},{<<"mat">>,"Z"},{<<"velvet">>,[279]},{<<"iggy">>,[248]},{<<"celldweller">>,"h"},{<<"leon">>,[205]},{<<"peppers">>,[161]},{<<"heartbreakers">>,[387]},{<<"paramore">>,[176]},{<<"frans">>,[402]},{<<"motorheards">>,[266]},{<<"ferdinand">>,[403]},{<<"monday">>,"7"},{<<"wertz">>,">"},{<<"escape">>,[190]},{<<"secondhand">>,[15]},{<<"jimmy">>,"I"},{<<"veil">>,[184]},{<<"toy">>,"d"},{<<"chromeo">>,[333]},{<<"wolf">>,[388]},{<<"james">>,"?"},{<<"top">>,[361]},{<<"turn">>,"\"},{<<"panic">>,[291]},{<<"cocoon">>,[310]},{<<"lifehouse">>,"f"},{<<"tokio">>,[174]},{<<"yeah">>,[393,394]},{<<"lovehatehero">>,[296]},{<<"kim">>,[398]},{<<"all">>,[194,197]},{<<"lydia">>,"+"},{<<"foo">>,[216]},{<<"disciple">>,[145]},{<<"vampire">>,[252]},{<<"lost">>,"l"},{<<"silversun">>,[206]},{<<"stevie">>,[378]},{<<"38th">>,[17]},{<<"horse">>,"""},{<<"deep">>,[258]},{<<"florence">>,[326]},{<<"nirvana">>,[208]},{<<"eleven">>,[376]},{<<"drive">>,[25]},{<<"zeppelin">>,[350]},{<<"emarosa">>,[146]},{<<"hall">>,[347]},{<<"bowie">>,[211]},{<<"above">>,"9y"},{<<"seconds">>,[200]},{<<"deas">>,[31]},{<<"arcade">>,"X"},{<<"plain">>,[202]},{<<"projectors">>,[318]},{<<"acdc">>,[212]},{<<"jovi">>,[247]},{<<"shinedown">>,[284]},{<<"traveler">>,[138]},{<<"alpha">>,"\r"},{<<"korn">>,[186]},{<<"alkaline">>,[132]},{<<"creedence">>,[139]},{<<"mars">>,[201]},{<<"like">>,[22]},{<<"temper">>,[345]},{<<"neon">>,[33,276]},{<<"fire">>,"Y"},{<<"cutie">>,[314]},{<<"park">>,[155]},{<<"against">>,[299]},{<<"she">>,[399]},{<<"boy">>,[150]},{<<"elton">>,[319]},{<<"wagon">>,[366]},{<<"purple">>,[259]},{<<"serenade">>,[16]},{<<"children">>,"g"},{<<"hotel">>,[175]},{<<"civil">>,[6]},{<<"jacket">>,"4"},{<<"ramones">>,[261]},{<<"oates">>,[348]},{<<"red">>,[114,158]},{<<"fears">>,[374]},{<<"revolver">>,[280]},{<<"david">>,[210]},{<<"jam">>,[241]},{<<"pit">>,[307]},{<<"keith">>,[321]},{<<"ivoryline">>,[27]},{<<"waking">>,"K"},{<<"richie">>,[368]},{<<"fall">>,[144,148]},{<<"killers">>,[153]},{<<"grace">>,[189]},{<<"skynard">>,[360]},{<<"p.o.d">>,[26]},{<<"chili">>,[160]},{<<"pearl">>,[240]},{<<"machine">>,[329]},{<<"adema">>,[282]},{<<"safetysuit">>,[5]},{<<"none">>,[166]},{<<"oomph">>,[187]},{<<"reign">>,[8]},{<<"october">>,"S"},{<<"def">>,[315]},{<<"girls">>,[23]},{<<"crue">>,[290]},{<<"grey">>,"#"},{<<"floyd">>,[135]},{<<"ravens">>,[20]},{<<"stones">>,[232]},{<<"tall">>,"\n"},{<<"van">>,[352]},{<<"drawing">>,"F"},{<<"urban">>,[322]},{<<"foot">>,"C"},{<<"veiled">>,[268]},{<<"days">>,[188]},{<<"kiss">>,[262]},{<<"stays">>,"q"},{<<"seether">>,[283]},{<<"hey">>,"6"},{<<"dragon">>,[254]},{<<"radiohead">>,[129]},{<<"lynayrd">>,[359]},{<<"brides">>,[185,269]},{<<"tokyo">>,[302]},{<<"morning">>,"3"},{<<"muse">>,[130]},{<<"pet">>,[356]},{<<"bjork">>,[382]},{<<"holiday">>,"$"},{<<"lionel">>,[367]},{<<"morrison">>,[64,353]},{<<"whitesnake">>,[288]},{<<"rock">>,[233]},{<<"disco">>,[293]},{<<"depeche">>,[181]},{<<"keane">>,[396]},{<<"fate">>,[192]},{<<"off">>,"]"},{<<"mgmt">>,[305]},{<<"blues">>,[137]},{<<"yeahs">>,[395]},{<<"newworldson">>,"8"},{<<"see">>,[127]},{<<"fleetwood">>,[380]},{<<"lions">>,[11]},{<<"white">>,[203]},{<<"coldplay">>,[265]},{<<"trap">>,[346]},{<<"revival">>,[141]},{<<"aerosmith">>,[234]},{<<"guns">>,[101,218]},{<<"american">>,[195]},{<<"ateyu">>,[300]},{<<"leopard">>,[316]},{<<"stevens">>,[363]},{<<"fighters">>,[217]},{<<"chemical">>,[156]},{<<"doors">>,[245,285]},{<<"jon">>,"%"},{<<"rapture">>,"1"},{<<"keys">>,[237]},{<<"send">>,"w"},{<<"afi">>,[131]},{<<"mclaughlin">>,"&"},{<<"freelance">>,[334]},{<<"poets">>,[142]},{<<"fireflight">>,"T"},{<<"hot">>,[159]},{<<"black">>,[183,226,236,267,323]},{<<"zeplin">>,[214]}]},{<<"updated_at">>,<<"2012-05-18">>,[{<<"2012-05-18">>,[0]}]},{<<"url">>,<<"
>> http://www.serieaticketsonline.com/content-milan-players-looking-forward-for-second-leg-against-barcelona/
>> ">>,[{<<"barcelona">>,[11]},{<<"leg">>,"\t"},{<<"http">>,[0]},{<<"players">>,[4]},{<<"
>> www.serieaticketsonline.com<http://www.serieaticketsonline.com
>> >">>,[1]},{<<"second">>,[8]},{<<"content">>,[2]},{<<"looking">>,[5]},{<<"against">>,"\n"},{<<"milan">>,[3]},{<<"forward">>,[6]}]}],[],[],true}}],[{<<15,27,99,155,79,178,239,200>>,{3,63504550680}}],{dict,1,16,16,8,80,48,{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[[clean|true]],[]}}},undefined},13537270,63504550680,[]}}}
>> > ** When Server state ==
>> {state,undefined,undefined,riak_kv_vnode,riak_kv_legacy_vnode}
>> > ** Reason for termination ==
>> > **
>> {badarg,[{gen_fsm,send_event,2},{riak_core_vnode_master,handle_cast,2},{gen_server,handle_msg,5},{proc_lib,init_p_do_apply,3}]}
>> > 2012-05-18 04:58:20 =CRASH REPORT====
>> >  crasher:
>> >    initial call: riak_core_vnode_master:init/1
>> >    pid: <0.421.0>
>> >    registered_name: riak_kv_vnode_master
>> >    exception exit:
>> {badarg,[{gen_fsm,send_event,2},{riak_core_vnode_master,handle_cast,2},{gen_server,handle_msg,5},{proc_lib,init_p_do_apply,3}]}
>> >      in function  gen_server:terminate/6
>> >      in call from proc_lib:init_p_do_apply/3
>> >    ancestors: [riak_kv_sup,<0.323.0>]
>> >    messages: []
>> >    links: [<0.420.0>]
>> >    dictionary: []
>> >    trap_exit: false
>> >    status: running
>> >    heap_size: 6765
>> >    stack_size: 24
>> >    reductions: 547963388
>> >  neighbours:
>> > 2012-05-18 04:58:20 =SUPERVISOR REPORT====
>> >     Supervisor: {local,riak_kv_sup}
>> >     Context:    child_terminated
>> >     Reason:
>> {badarg,[{gen_fsm,send_event,2},{riak_core_vnode_master,handle_cast,2},{gen_server,handle_msg,5},{proc_lib,init_p_do_apply,3}]}
>> >     Offender:
>> [{pid,<0.421.0>},{name,riak_kv_vnode_master},{mfargs,{riak_core_vnode_master,start_link,[riak_kv_vnode,riak_kv_legacy_vnode]}},{restart_type,permanent},{shutdown,5000},{child_type,worker}]
>> >
>> > 2012-05-18 04:58:20 =SUPERVISOR REPORT====
>> >     Supervisor: {local,riak_core_sup}
>> >     Context:    child_terminated
>> >     Reason:     shutdown
>> >     Offender:
>> [{pid,<0.107.0>},{name,riak_core_vnode_proxy_sup},{mfargs,{riak_core_vnode_proxy_sup,start_link,[]}},{restart_type,permanent},{shutdown,5000},{child_type,supervisor}]
>> >
>> > 2012-05-18 04:58:20 =ERROR REPORT====
>> > ** Generic server riak_kv_stat terminating
>> > ** Last message in was
>> {'$gen_cast',{update,{get_fsm,undefined,279,1,1350},63504536300}}
>> > ** When Server state ==
>> {state,{spiral,63504536300,[11,0,0,0,0,0,0,0,0,0,0,18,0,0,0,3,17,0,0,0,0,0,0,0,0,0,23,0,0,0,9,12,0,0,0,14,0,0,0,0,0,3,5,0,0,0,2,6,0,0,0,0,0,0,0,0,0,30,0,0,16,27,33,0,0,0,0,0,0,0,0,0,0,0,0,0,0,8,6,0,0,14,0,0,0,0,0,7,22]},{spiral,63504536300,[6,0,0,0,0,0,0,0,0,0,0,12,0,0,0,3,10,0,0,0,0,0,0,0,0,0,16,0,0,0,6,7,0,0,0,7,0,0,0,0,0,3,2,0,0,0,2,2,0,0,0,0,0,0,0,0,0,18,0,0,11,18,22,0,0,0,0,0,0,0,0,0,0,0,0,0,0,4,6,0,0,7,0,0,0,0,0,7,12]},5887366,3601568,{spiral,63504536300,[42,0,0,0,0,0,0,0,0,0,0,42,0,0,0,42,42,0,0,0,0,0,0,0,0,0,42,0,0,0,42,84,0,0,0,84,0,0,0,0,0,42,42,0,0,0,42,42,0,0,0,0,0,0,0,0,0,42,0,0,0,84,42,0,0,0,0,0,0,0,0,42,0,0,0,0,0,84,0,0,0,84,0,0,0,0,0,42,168]},10079908,{spiral,63504536300,[6,0,0,0,0,0,0,0,0,0,0,12,0,0,0,3,10,0,0,0,0,0,0,0,0,0,16,0,0,0,6,7,0,0,0,7,0,0,0,0,0,3,2,0,0,0,2,2,0,0,0,0,0,0,0,0,0,18,0,0,11,18,22,0,0,0,0,0,0,0,0,0,0,0,0,0,0,4,6,0,0,7,0,0,0,0,0,7,12]},3601568,{spiral,63504536300,[3,0,0,0,0,0,0,0,0,0,0,3,0,0,0,0,1,0,0,0,0,0,0,0,0,0,3,0,0,0,0,3,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,4,0,0,0,3,3,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,7]},788755,{spiral,63504477129,[1,0,0,0,0,0,0,0,0,0,0,2,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,0,0,0,0,0,0,0,0,0,0,3,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,3,0]},32,{spiral,63504536300,[0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]},17236,2394509,1571512,{slide,63504471451,60,60,"/var/lib/riak/slide-data/3053/1337.266551.794838",{file_descriptor,prim_file,{#Port<0.2186351>,40}},63504536289},{slide,63504471451,60,60,"/var/lib/riak/slide-data/3053/1337.266551.794979",{file_descriptor,prim_file,{#Port<0.2186352>,128}},63504536289},{slide,63504471451,60,60,"/var/lib/riak/slide-data/3053/1337.266551.794984",{file_descriptor,prim_file,{#Port<0.2186349>,69}},63504536289},{slide,63504475275,60,60,"/var/lib/riak/slide-data/3053/1337.266551.794989",{file_descriptor,prim_file,{#Port<0.2186353>,177}},63504536289},{spiral,63504471351,[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]},0,0,{spiral,63504536285,[2,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1]},132,undefined,259259,0,undefined,undefined,0,0,true}
>> > ** Reason for termination ==
>> > ** {emfile,[{erlang,open_port,[{spawn,"/bin/sh -s unix:cmd
>> 2>&1"},[stream]]},{os,start_port_srv_handle,1},{os,start_port_srv_loop,0}]}
>> > 2012-05-18 04:58:20 =CRASH REPORT====
>> >  crasher:
>> >    initial call: gen:init_it/7
>> >    pid: <0.426.0>
>> >    registered_name: riak_kv_stat
>> >    exception exit: {emfile,[{erlang,open_port,[{spawn,"/bin/sh -s
>> unix:cmd
>> 2>&1"},[stream]]},{os,start_port_srv_handle,1},{os,start_port_srv_loop,0}]}
>> >      in function  gen_server2:terminate/6
>> >      in call from proc_lib:init_p_do_apply/3
>> >    ancestors: [riak_kv_sup,<0.323.0>]
>> >    messages:
>> [{'$gen_cast',{update,vnode_get,63504536300}},{'$gen_cast',{update,vnode_get,63504536300}},{'$gen_cast',{update,vnode_get,63504536300}},{'$gen_cast',{update,vnode_get,63504536300}},{'$gen_cast',{update,{get_fsm,undefined,337,1,1350},63504536300}},{'$gen_cast',{update,{vnode_index_write,0,0},63504536300}},{'$gen_cast',{update,vnode_put,63504536300}},{'$gen_cast',{update,{vnode_index_write,0,0},63504536300}},{'$gen_cast',{update,vnode_put,63504536300}},{'$gen_cast',{update,{put_fsm_time,603},63504536300}},{'$gen_cast',{update,{vnode_index_write,0,0},63504536300}},{'$gen_cast',{update,vnode_put,63504536300}},{'$gen_cast',{update,{vnode_index_write,0,0},63504536300}},{'$gen_cast',{update,vnode_put,63504536300}},{'$gen_cast',{update,{vnode_index_write,0,0},63504536300}},{'$gen_cast',{update,vnode_put,63504536300}},{'$gen_cast',{update,{put_fsm_time,13883},63504536300}},{'$gen_cast',{update,{vnode_index_write,0,0},63504536300}},{'$gen_cast',{update,vnode_put,63504536300}}]
>> >    links: [#Port<0.2186353>,<0.420.0>,#Port<0.2186351>,#Port<0.2186352>]
>> >    dictionary: []
>> >    trap_exit: true
>> >    status: running
>> >    heap_size: 6765
>> >    stack_size: 24
>> >    reductions: 1873973117
>> >  neighbours:
>> > 2012-05-18 04:58:20 =SUPERVISOR REPORT====
>> >     Supervisor: {local,riak_kv_sup}
>> >     Context:    child_terminated
>> >     Reason:     {emfile,[{erlang,open_port,[{spawn,"/bin/sh -s unix:cmd
>> 2>&1"},[stream]]},{os,start_port_srv_handle,1},{os,start_port_srv_loop,0}]}
>> >     Offender:
>> [{pid,<0.426.0>},{name,riak_kv_stat},{mfargs,{riak_kv_stat,start_link,[]}},{restart_type,permanent},{shutdown,5000},{child_type,worker}]
>> >
>> >
>> > App Config:
>> >
>> > %% -*- mode: erlang;erlang-indent-level: 4;indent-tabs-mode: nil -*-
>> > %% ex: ft=erlang ts=4 sw=4 et
>> > [
>> > %% Riak Core config
>> > {riak_core, [
>> >              %% Default location of ringstate
>> >              {ring_state_dir, "/var/lib/riak/ring"},
>> >
>> >     %% The number of partitions to divide the hash space into
>> >     %% The default is 64. Set to a size to allow for growth as
>> >     %% this cannot be changed easily after a cluster is set up.
>> >     %% Each node should have at least 10 partitions assigned to it.
>> >     {ring_creation_size, 128},
>> >
>> >              %% http is a list of IP addresses and TCP ports that the
>> Riak
>> >              %% HTTP interface will bind.
>> >              {http, [ {"10.38.24.196", 8098 } ]},
>> >
>> >              %% https is a list of IP addresses and TCP ports that the
>> Riak
>> >              %% HTTPS interface will bind.
>> >              {https, [{"50.28.61.141", 8069 }]},
>> >
>> >              %% Default cert and key locations for https can be
>> overridden
>> >              %% with the ssl config variable, for example:
>> >              {ssl, [
>> >                     {certfile, "/etc/riak/poseidon_cert.pem"},
>> >                     {keyfile, "/etc/riak/poseidon_key.pem"}
>> >                    ]},
>> >
>> >              %% riak_handoff_port is the TCP port that Riak uses for
>> >              %% intra-cluster data handoff.
>> >              {handoff_port, 8099 },
>> >
>> >              %% To encrypt riak_core intra-cluster data handoff traffic,
>> >              %% uncomment the following line and edit its path to an
>> >              %% appropriate certfile and keyfile.  (This example uses a
>> >              %% single file with both items concatenated together.)
>> >              %{handoff_ssl_options, [{certfile, "/tmp/erlserver.pem"}]},
>> >
>> >              %% Platform-specific installation paths (substituted by
>> rebar)
>> >              {platform_bin_dir, "/usr/sbin"},
>> >              {platform_data_dir, "/var/lib/riak"},
>> >              {platform_etc_dir, "/etc/riak"},
>> >              {platform_lib_dir, "/usr/lib/riak"},
>> >              {platform_log_dir, "/var/log/riak"}
>> >             ]},
>> >
>> > %% Riak KV config
>> > {riak_kv, [
>> >            %% Storage_backend specifies the Erlang module defining the
>> storage
>> >            %% mechanism that will be used on this node.
>> >            {storage_backend, riak_kv_eleveldb_backend},
>> >
>> >            %% pb_ip is the IP address that the Riak Protocol Buffers
>> interface
>> >            %% will bind to.  If this is undefined, the interface will
>> not run.
>> >            {pb_ip,   "127.0.0.1" },
>> >
>> >            %% pb_port is the TCP port that the Riak Protocol Buffers
>> interface
>> >            %% will bind to
>> >            {pb_port, 8087 },
>> >
>> >            %% pb_backlog is the maximum length to which the queue of
>> pending
>> >            %% connections may grow. If set, it must be an integer >= 0.
>> >            %% By default the value is 5. If you anticipate a huge
>> number of
>> >            %% connections being initialised *simultaneously*, set this
>> number
>> >            %% higher.
>> >            %% {pb_backlog, 64},
>> >
>> >            %% raw_name is the first part of all URLS used by the Riak
>> raw HTTP
>> >            %% interface.  See riak_web.erl and raw_http_resource.erl for
>> >            %% details.
>> >            %{raw_name, "riak"},
>> >
>> >            %% mapred_name is URL used to submit map/reduce requests to
>> Riak.
>> >            {mapred_name, "mapred"},
>> >
>> >            %% mapred_system indicates which version of the MapReduce
>> >            %% system should be used: 'pipe' means riak_pipe will
>> >            %% power MapReduce queries, while 'legacy' means that luke
>> >            %% will be used
>> >            {mapred_system, pipe},
>> >
>> >            %% mapred_2i_pipe indicates whether secondary-index
>> >            %% MapReduce inputs are queued in parallel via their own
>> >            %% pipe ('true'), or serially via a helper process
>> >            %% ('false' or undefined).  Set to 'false' or leave
>> >            %% undefined during a rolling upgrade from 1.0.
>> >            {mapred_2i_pipe, true},
>> >
>> >            %% directory used to store a transient queue for pending
>> >            %% map tasks
>> >            %% Only valid when mapred_system == legacy
>> >            %% {mapred_queue_dir, "/var/lib/riak/mr_queue" },
>> >
>> >            %% Each of the following entries control how many Javascript
>> >            %% virtual machines are available for executing map, reduce,
>> >            %% pre- and post-commit hook functions.
>> >            {map_js_vm_count, 8 },
>> >            {reduce_js_vm_count, 6 },
>> >            {hook_js_vm_count, 2 },
>> >
>> >            %% Number of items the mapper will fetch in one request.
>> >            %% Larger values can impact read/write performance for
>> >            %% non-MapReduce requests.
>> >            %% Only valid when mapred_system == legacy
>> >            %% {mapper_batch_size, 5},
>> >
>> >            %% js_max_vm_mem is the maximum amount of memory, in
>> megabytes,
>> >            %% allocated to the Javascript VMs. If unset, the default is
>> >            %% 8MB.
>> >            {js_max_vm_mem, 8},
>> >
>> >            %% js_thread_stack is the maximum amount of thread stack, in
>> megabyes,
>> >            %% allocate to the Javascript VMs. If unset, the default is
>> 16MB.
>> >            %% NOTE: This is not the same as the C thread stack.
>> >            {js_thread_stack, 16},
>> >
>> >            %% Number of objects held in the MapReduce cache. These will
>> be
>> >            %% ejected when the cache runs out of room or the bucket/key
>> >            %% pair for that entry changes
>> >            %% Only valid when mapred_system == legacy
>> >            %% {map_cache_size, 10000},
>> >
>> >            %% js_source_dir should point to a directory containing
>> Javascript
>> >            %% source files which will be loaded by Riak when it
>> initializes
>> >            %% Javascript VMs.
>> >            {js_source_dir, "/tmp/js_source"},
>> >
>> >            %% http_url_encoding determines how Riak treats URL encoded
>> >            %% buckets, keys, and links over the REST API. When set to
>> 'on'
>> >            %% Riak always decodes encoded values sent as URLs and
>> Headers.
>> >            %% Otherwise, Riak defaults to compatibility mode where links
>> >            %% are decoded, but buckets and keys are not. The
>> compatibility
>> >            %% mode will be removed in a future release.
>> >            {http_url_encoding, on},
>> >
>> >            %% riak_stat enables the use of the "riak-admin status"
>> command to
>> >            %% retrieve information the Riak node for performance and
>> debugging needs
>> >            {riak_kv_stat, true},
>> >
>> >            %% When using riak_kv_stat, use the legacy routines for
>> tracking
>> >            {legacy_stats, true},
>> >
>> >            %% Switch to vnode-based vclocks rather than client ids.
>>  This
>> >            %% significantly reduces the number of vclock entries.
>> >            %% Only set true if *all* nodes in the cluster are upgraded
>> to 1.0
>> >            {vnode_vclocks, true},
>> >
>> >            %% This option enables compatability of bucket and key
>> listing
>> >            %% with 0.14 and earlier versions. Once a rolling upgrade to
>> >            %% a version > 0.14 is completed for a cluster, this should
>> be
>> >            %% set to false for improved performance for bucket and key
>> >            %% listing operations.
>> >            {legacy_keylisting, false},
>> >
>> >            %% This option toggles compatibility of keylisting with 1.0
>> >            %% and earlier versions.  Once a rolling upgrade to a version
>> >            %% > 1.0 is completed for a cluster, this should be set to
>> >            %% true for better control of memory usage during key listing
>> >            %% operations
>> >            {listkeys_backpressure, true}
>> >           ]},
>> >
>> > %% Riak Search Config
>> > {riak_search, [
>> >                %% To enable Search functionality set this 'true'.
>> >                {enabled, true}
>> >               ]},
>> >
>> > %% Merge Index Config
>> > {merge_index, [
>> >                %% The root dir to store search merge_index data
>> >                {data_root, "/var/lib/riak/merge_index"},
>> >
>> >                %% The root dir to store secondary index merge_index data
>> >                {data_root_2i, "/var/lib/riak/merge_index_2i"},
>> >
>> >                %% Size, in bytes, of the in-memory buffer.  When this
>> >                %% threshold has been reached the data is transformed
>> >                %% into a segment file which resides on disk.
>> >                {buffer_rollover_size, 1048576},
>> >
>> >                %% Overtime the segment files need to be compacted.
>> >                %% This is the maximum number of segments that will be
>> >                %% compacted at once.  A lower value will lead to
>> >                %% quicker but more frequent compactions.
>> >                {max_compact_segments, 20}
>> >               ]},
>> >
>> > %% Bitcask Config
>> > {bitcask, [
>> >             {data_root, "/var/lib/riak/bitcask"}
>> >           ]},
>> >
>> > %% eLevelDB Config
>> > {eleveldb, [
>> >             {data_root, "/var/lib/riak/leveldb"}
>> >            ]},
>> >
>> > %% Lager Config
>> > {lager, [
>> >            %% What handlers to install with what arguments
>> >            %% The defaults for the logfiles are to rotate the files when
>> >            %% they reach 10Mb or at midnight, whichever comes first,
>> and keep
>> >            %% the last 5 rotations. See the lager README for a
>> description of
>> >            %% the time rotation format:
>> >            %% https://github.com/basho/lager/blob/master/README.org
>> >            %%
>> >            %% If you wish to disable rotation, you can either set the
>> size to 0
>> >            %% and the rotation time to "", or instead specify a 2-tuple
>> that only
>> >            %% consists of {Logfile, Level}.
>> >            {handlers, [
>> >                {lager_console_backend, info},
>> >                {lager_file_backend, [
>> >                    {"/var/log/riak/error.log", error, 10485760, "$D0",
>> 5},
>> >                    {"/var/log/riak/console.log", info, 10485760, "$D0",
>> 5}
>> >                ]}
>> >            ]},
>> >
>> >            %% Whether to write a crash log, and where.
>> >            %% Commented/omitted/undefined means no crash logger.
>> >            {crash_log, "/var/log/riak/crash.log"},
>> >
>> >            %% Maximum size in bytes of events in the crash log -
>> defaults to 65536
>> >            {crash_log_msg_size, 65536},
>> >
>> >            %% Maximum size of the crash log in bytes, before its
>> rotated, set
>> >            %% to 0 to disable rotation - default is 0
>> >            {crash_log_size, 10485760},
>> >
>> >            %% What time to rotate the crash log - default is no time
>> >            %% rotation. See the lager README for a description of this
>> format:
>> >            %% https://github.com/basho/lager/blob/master/README.org
>> >            {crash_log_date, "$D0"},
>> >
>> >            %% Number of rotated crash logs to keep, 0 means keep only
>> the
>> >            %% current one - default is 0
>> >            {crash_log_count, 5},
>> >
>> >            %% Whether to redirect error_logger messages into lager -
>> defaults to true
>> >            {error_logger_redirect, true}
>> >        ]},
>> >
>> > %% riak_sysmon config
>> > {riak_sysmon, [
>> >         %% To disable forwarding events of a particular type, use a
>> >         %% limit of 0.
>> >         {process_limit, 30},
>> >         {port_limit, 2},
>> >
>> >         %% Finding reasonable limits for a given workload is a matter
>> >         %% of experimentation.
>> >         {gc_ms_limit, 100},
>> >         {heap_word_limit, 40111000},
>> >
>> >         %% Configure the following items to 'false' to disable logging
>> >         %% of that event type.
>> >         {busy_port, true},
>> >         {busy_dist_port, true}
>> >        ]},
>> >
>> > %% SASL config
>> > {sasl, [
>> >         {sasl_error_logger, false}
>> >        ]},
>> >
>> > %% riak_control config
>> > {riak_control, [
>> >                %% Set to false to disable the admin panel.
>> >                {enabled, true},
>> >
>> >                %% Authentication style used for access to the admin
>> >                %% panel. Valid styles are 'userlist' <TODO>.
>> >                {auth, userlist},
>> >
>> >                %% If auth is set to 'userlist' then this is the
>> >                %% list of usernames and passwords for access to the
>> >                %% admin panel.
>> >                {userlist, [{"Aquaman", "At1ant1s"}
>> >                           ]},
>> >
>> >                %% The admin panel is broken up into multiple
>> >                %% components, each of which is enabled or disabled
>> >                %% by one of these settings.
>> >                {admin, true}
>> >                ]}
>> > ].
>> >
>> > Hopefully this is helpful. Thanks Dave.
>> >
>> > -Gustav
>> >
>> > On Tue, May 22, 2012 at 6:35 AM, Dave Smith <dizzyd at basho.com<mailto:
>> dizzyd at basho.com>> wrote:
>> > Hi Gustav,
>> >
>> > Can you share the logs or any other information from the server that
>> crashes _first_? It might be good to also include app.config, just so we
>> can double-check that no other settings have changed.
>> >
>> > Thanks,
>> >
>> > D.
>> >
>> > --
>> > Dave Smith
>> > VP, Engineering
>> > Basho Technologies, Inc.
>> > dizzyd at basho.com<mailto:dizzyd at basho.com>
>> >
>> >
>> >
>> > On Friday, May 18, 2012 at 10:03 AM, Gustav Gennrich wrote:
>> >
>> >> Hi,
>> >>
>> >> Below is our setup to better diagnose:
>> >>
>> >> Ring creation size: 128
>> >> Backend: LevelDB
>> >>
>> >> All other settings in app.config are using the default values
>> >>
>> >> We have 3 servers (nodes) each of which have 8GB of ram.
>> >>
>> >> Question 1: We are attempting to load data into Riak using CSVs that
>> we generated from data previously stored is our MySQL database. We are
>> using Ripple to create the documents and associations (links). The problem
>> we are having is Riak continue to grow in memory consumption until it
>> finally crashed while loading this data. We are only able to add around a
>> couple hundred thousand rows from the CSV's before this happens. Riak then
>> responds with 500 errors and eventually all of the nodes crash and the
>> nodes need to be restarted.
>> >>
>> >> Question 2 (unrelated): I am attempting to use some of the already
>> defined js functions in ripple.js in a mapreduce, however, I keep getting
>> the following error:
>> >>
>> >> Riak::MapReduceError:
>> {"phase":0,"error":"[{<<\"lineno\">>,1},{<<\"message\">>,<<\"ReferenceError:
>> Ripple is not defined\">>},{<<\"source\">>,<<\"unknown\">>}]"
>> >>
>> >> In ripple.yml I have the following (***** represent hidden values):
>> >>
>> >> # Configure Riak connections for the Ripple library.
>> >> development:
>> >> # http_port: 8091
>> >> # pb_port: 8087
>> >> # host: 127.0.0.1
>> >> basic_auth: *****
>> >> http_port: 80
>> >> pb_port: 8087
>> >> host: *****
>> >> http_backend: Excon
>> >> solr: "/solr"
>> >> js_source_dir: <%= Rails.root + "app/mapreduce" %>
>> >>
>> >> # The test environment has additional keys for configuring the
>> >> # Riak::TestServer for your test/spec suite:
>> >> #
>> >> # * bin_dir specifies the path to the "riak" script that you use to
>> >> # start Riak (just the directory)
>> >> # * js_source_dir specifies where your custom Javascript functions for
>> >> # MapReduce should be loaded from. Usually app/mapreduce.
>> >> test:
>> >> http_port: 9000
>> >> pb_port: 9002
>> >> host: localhost
>> >>
>> >> source: /usr/local/bin # Default for Homebrew.
>> >> js_source_dir: <%= Rails.root + "/app/mapreduce" %>
>> >>
>> >> production:
>> >> nodes:
>> >> - host: *****
>> >> http_port: 8098
>> >> pb_port: 8087
>> >> - host: *****
>> >> http_port: 8098
>> >> pb_port: 8087
>> >> - host: *****
>> >> http_port: 8098
>> >> pb_port: 8087
>> >> http_backend: Excon
>> >> solr: "/solr"
>> >> js_source_dir: <%= Rails.root + "app/mapreduce" %>
>> >>
>> >>
>> >> I don't see why I am getting this error when I have defined where the
>> js_source_dir is.
>> >>
>> >> Any help would be greatly appreciated!
>> >>
>> >> Thanks,
>> >>
>> >> Gustav
>> >> _______________________________________________
>> >> riak-users mailing list
>> >> riak-users at lists.basho.com<mailto:riak-users at lists.basho.com> (mailto:
>> riak-users at lists.basho.com<mailto:riak-users at lists.basho.com>)
>> >> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>> >
>> >
>> >
>> >
>> > _______________________________________________
>> > riak-users mailing list
>> > riak-users at lists.basho.com
>> > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.basho.com/pipermail/riak-users_lists.basho.com/attachments/20120525/8c03da2e/attachment.html>


More information about the riak-users mailing list