Warning "Can not start proc_lib:init_p"

Ingo Rockel ingo.rockel at bluelionmobile.com
Thu Apr 4 11:34:59 EDT 2013


thanks, but it was a very obvious c&p error :) and we already have the 
ERL_MAX_ETS_TABLES set to 8192 as it is in the default vm.args.

The only other messages were about a lot of handoff going on.

Maybe the node was getting some data concerning the 2GB object?

Ingo

Am 04.04.2013 17:25, schrieb Evan Vigil-McClanahan:
> Major error on my part here!
>
>> your vm.args:
>> -env ERL_MAX_ETS_TABLES 819
>
> This should be
>
> -env ERL_MAX_ETS_TABLES 8192
>
> Sorry for the sloppy cut and paste.  Please do not do the former
> thing, or it will be very bad.
>
>> This is a good idea for all systems but is especially important for
>> people with large rings.
>>
>> Were there any other messages?  Riak constantly spawns new processes,
>> but they don't tend to build up unless the backend is misbehaving (or
>> a few other less likely conditions), and a backup of spawned processes
>> is the only thing I can think of that would make +P help with OOM
>> issues.
>>
>> On Thu, Apr 4, 2013 at 9:21 AM, Ingo Rockel
>> <ingo.rockel at bluelionmobile.com> wrote:
>>> A grep for "too many processes" didn't reveal anything. The process got
>>> killed by the oom-killer.
>>>
>>> Am 04.04.2013 16:12, schrieb Evan Vigil-McClanahan:
>>>
>>>> That's odd.  It was getting killed by the OOM killer, or crashing
>>>> because it couldn't allocate more memory?  That's suggestive of
>>>> something else that's wrong, since the +P doesn't do any memory
>>>> limiting.  Are you getting 'too many processes' emulator errors on
>>>> that node?
>>>>
>>>> On Thu, Apr 4, 2013 at 8:47 AM, Ingo Rockel
>>>> <ingo.rockel at bluelionmobile.com> wrote:
>>>>>
>>>>> the crashing node seems to be caused by the raised +P param, after last
>>>>> crash I commented the param and now the node runs just fine.
>>>>>
>>>>> Am 04.04.2013 15:43, schrieb Ingo Rockel:
>>>>>
>>>>>> Hi Evan,
>>>>>>
>>>>>> we added monitoring of the object sizes and there was one object on one
>>>>>> of the three nodes mentioned which was > 2GB!!
>>>>>>
>>>>>> We just changed the application code to get the id of this object to be
>>>>>> able to delete it. But is does happen only about once a day.
>>>>>>
>>>>>> We right now have another node constantly crashing with oom about 12
>>>>>> minutes after start (always the same time frame), could this be related
>>>>>> to the big object issue? It is not one of the three nodes. The node logs
>>>>>> a lot of handoff receiving is going on.
>>>>>>
>>>>>> Again, thanks for the help!
>>>>>>
>>>>>> Regards,
>>>>>>
>>>>>>        Ingo
>>>>>>
>>>>>> Am 04.04.2013 15:30, schrieb Evan Vigil-McClanahan:
>>>>>>>
>>>>>>>
>>>>>>> If it's always the same three nodes it could well be same very large
>>>>>>> object being updated each day.  Is there anything else that looks
>>>>>>> suspicious in your logs?  Another sign of large objects is large_heap
>>>>>>> (or long_gc) messages from riak_sysmon.
>>>>>>>
>>>>>>> On Thu, Apr 4, 2013 at 3:58 AM, Ingo Rockel
>>>>>>> <ingo.rockel at bluelionmobile.com> wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>> Hi Evan,
>>>>>>>>
>>>>>>>> thanks for all the infos! I adjusted the leveldb-config as suggested,
>>>>>>>> except
>>>>>>>> the cache, which I reduced to 16MB, keeping this above the default
>>>>>>>> helped a
>>>>>>>> lot at least during load testing. And I added +P 130072 to the
>>>>>>>> vm.args. Will
>>>>>>>> be applied to the riak nodes the next hours.
>>>>>>>>
>>>>>>>> We have a monitoring using zabbbix, but haven't included the object
>>>>>>>> sizes so
>>>>>>>> far, will be added today.
>>>>>>>>
>>>>>>>> We double-checked the Linux-Performance-Doc to be sure everything is
>>>>>>>> applied
>>>>>>>> to the nodes, especially as the problems always are caused from the
>>>>>>>> same
>>>>>>>> three nodes. But everything looks fine.
>>>>>>>>
>>>>>>>> Ingo
>>>>>>>>
>>>>>>>> Am 03.04.2013 18:42, schrieb Evan Vigil-McClanahan:
>>>>>>>>
>>>>>>>>> Another engineer mentions that you posted your eleveldb section and I
>>>>>>>>> totally missed it:
>>>>>>>>>
>>>>>>>>> The eleveldb section:
>>>>>>>>>
>>>>>>>>>      %% eLevelDB Config
>>>>>>>>>      {eleveldb, [
>>>>>>>>>                  {data_root, "/var/lib/riak/leveldb"},
>>>>>>>>>                  {cache_size, 33554432},
>>>>>>>>>                  {write_buffer_size_min, 67108864}, %% 64 MB in bytes
>>>>>>>>>                  {write_buffer_size_max, 134217728}, %% 128 MB in
>>>>>>>>> bytes
>>>>>>>>>                  {max_open_files, 4000}
>>>>>>>>>                 ]},
>>>>>>>>>
>>>>>>>>> This is likely going to make you unhappy as time goes on; Since all
>>>>>>>>> of
>>>>>>>>> those settings are per-vnode, your max memory utilization is well
>>>>>>>>> beyond your physical memory.  I'd remove the tunings for the caches
>>>>>>>>> and buffers and drop max open files to 500, perhaps.  Make sure that
>>>>>>>>> you've followed everything in:
>>>>>>>>>
>>>>>>>>> http://docs.basho.com/riak/latest/cookbooks/Linux-Performance-Tuning/,
>>>>>>>>> etc.
>>>>>>>>>
>>>>>>>>> On Wed, Apr 3, 2013 at 9:33 AM, Evan Vigil-McClanahan
>>>>>>>>> <emcclanahan at basho.com> wrote:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Again, all of these things are signs of large objects, so if you
>>>>>>>>>> could
>>>>>>>>>> track the object_size stats on the cluster, I think that we might
>>>>>>>>>> see
>>>>>>>>>> something.  Even if you have no monitoring, a simple shell script
>>>>>>>>>> curling /stats/ on each node once a minute should do the job for a
>>>>>>>>>> day
>>>>>>>>>> or two.
>>>>>>>>>>
>>>>>>>>>> On Wed, Apr 3, 2013 at 9:29 AM, Ingo Rockel
>>>>>>>>>> <ingo.rockel at bluelionmobile.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> We just had it again (around this time of the day we have our
>>>>>>>>>>> highest
>>>>>>>>>>> user
>>>>>>>>>>> activity).
>>>>>>>>>>>
>>>>>>>>>>> I will set +P to 131072 tomorrow, anything else I should check or
>>>>>>>>>>> change?
>>>>>>>>>>>
>>>>>>>>>>> What about this memory-high-watermark which I get sporadically?
>>>>>>>>>>>
>>>>>>>>>>> Ingo
>>>>>>>>>>>
>>>>>>>>>>> Am 03.04.2013 17:57, schrieb Evan Vigil-McClanahan:
>>>>>>>>>>>
>>>>>>>>>>>> As for +P it's been raised in R16 (which is on the current man
>>>>>>>>>>>> page)
>>>>>>>>>>>> on R15 it's only 32k.
>>>>>>>>>>>>
>>>>>>>>>>>> The behavior that you're describing does sound like a very large
>>>>>>>>>>>> object getting put into the cluster (which may cause backups and
>>>>>>>>>>>> push
>>>>>>>>>>>> you up against the process limit, could have caused scheduler
>>>>>>>>>>>> collapse
>>>>>>>>>>>> on 1.2, etc.).
>>>>>>>>>>>>
>>>>>>>>>>>> On Wed, Apr 3, 2013 at 8:39 AM, Ingo Rockel
>>>>>>>>>>>> <ingo.rockel at bluelionmobile.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Evan,
>>>>>>>>>>>>>
>>>>>>>>>>>>> sys_process_count is somewhere between 5k and 11k on the nodes
>>>>>>>>>>>>> right
>>>>>>>>>>>>> now.
>>>>>>>>>>>>> Concerning your suggested +P config, according to the
>>>>>>>>>>>>> erlang-docs, the
>>>>>>>>>>>>> default for this param already is 262144, so setting it to 655536
>>>>>>>>>>>>> would
>>>>>>>>>>>>> in
>>>>>>>>>>>>> fact lower it?
>>>>>>>>>>>>>
>>>>>>>>>>>>> We chose the ring size to be able to handle growth which was the
>>>>>>>>>>>>> main
>>>>>>>>>>>>> reason
>>>>>>>>>>>>> to switch from mysql to nosql/riak. We have 12 Nodes, so about 86
>>>>>>>>>>>>> vnodes
>>>>>>>>>>>>> per
>>>>>>>>>>>>> node.
>>>>>>>>>>>>>
>>>>>>>>>>>>> No, we don't monitor object sizes, the majority of objects is
>>>>>>>>>>>>> very
>>>>>>>>>>>>> small
>>>>>>>>>>>>> (below 200 bytes), but we have objects storing references to this
>>>>>>>>>>>>> small
>>>>>>>>>>>>> objects which might grow to a few megabytes in size, most of
>>>>>>>>>>>>> these are
>>>>>>>>>>>>> paged
>>>>>>>>>>>>> though and should not exceed one megabyte. Only one type is not
>>>>>>>>>>>>> paged
>>>>>>>>>>>>> (implementation reasons).
>>>>>>>>>>>>>
>>>>>>>>>>>>> The outgoing/incoming traffic constantly is around 100 Mbit, if
>>>>>>>>>>>>> the
>>>>>>>>>>>>> peformance drops happen, we suddenly see spikes up to 1GBit. And
>>>>>>>>>>>>> these
>>>>>>>>>>>>> spikes constantly happen on three nodes as long as the
>>>>>>>>>>>>> performance
>>>>>>>>>>>>> drop
>>>>>>>>>>>>> exists.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Ingo
>>>>>>>>>>>>>
>>>>>>>>>>>>> Am 03.04.2013 17:12, schrieb Evan Vigil-McClanahan:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Ingo,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> riak-admin status | grep sys_process_count
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> will tell you how many processes are running.  The default
>>>>>>>>>>>>>> process
>>>>>>>>>>>>>> limit on erlang is a little low, and we'd suggest raising in
>>>>>>>>>>>>>> (especially with your extra-large ring_size).   Erlang
>>>>>>>>>>>>>> processes are
>>>>>>>>>>>>>> cheap, so 65535 or even double that will be fine.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Busy dist ports are still worrying.  Are you monitoring object
>>>>>>>>>>>>>> sizes?
>>>>>>>>>>>>>> Are there any spikes there associated with performance drops?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Wed, Apr 3, 2013 at 8:03 AM, Ingo Rockel
>>>>>>>>>>>>>> <ingo.rockel at bluelionmobile.com> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi Evan,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I set swt very_low and zdbbl to 64MB, setting this params
>>>>>>>>>>>>>>> helped
>>>>>>>>>>>>>>> reducing
>>>>>>>>>>>>>>> the busy_dist_port and Monitor got {suppressed,... Messages a
>>>>>>>>>>>>>>> lot.
>>>>>>>>>>>>>>> But
>>>>>>>>>>>>>>> when
>>>>>>>>>>>>>>> the performance of the cluster suddenly drops we still see
>>>>>>>>>>>>>>> these
>>>>>>>>>>>>>>> messages.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> The cluster was updated to 1.3 in the meantime.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> The eleveldb section:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>        %% eLevelDB Config
>>>>>>>>>>>>>>>        {eleveldb, [
>>>>>>>>>>>>>>>                    {data_root, "/var/lib/riak/leveldb"},
>>>>>>>>>>>>>>>                    {cache_size, 33554432},
>>>>>>>>>>>>>>>                    {write_buffer_size_min, 67108864}, %% 64 MB
>>>>>>>>>>>>>>> in bytes
>>>>>>>>>>>>>>>                    {write_buffer_size_max, 134217728}, %% 128 MB
>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>> bytes
>>>>>>>>>>>>>>>                    {max_open_files, 4000}
>>>>>>>>>>>>>>>                   ]},
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> the ring size is 1024 and the machines have 48GB of memory.
>>>>>>>>>>>>>>> Concerning
>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>> params from vm.args:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> -env ERL_MAX_PORTS 4096
>>>>>>>>>>>>>>> -env ERL_MAX_ETS_TABLES 8192
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> +P isn't set
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Ingo
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Am 03.04.2013 16:53, schrieb Evan Vigil-McClanahan:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> For your prior mail, I thought that someone had answered.  Our
>>>>>>>>>>>>>>>> initial
>>>>>>>>>>>>>>>> suggestion was to add +swt very_low to your vm.args, as well
>>>>>>>>>>>>>>>> as
>>>>>>>>>>>>>>>> setting the +zdbbl setting that Jon recommended in the list
>>>>>>>>>>>>>>>> post
>>>>>>>>>>>>>>>> you
>>>>>>>>>>>>>>>> pointed to.  If those help, moving to 1.3 should help more.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Other limits in vm.args that can cause problems are +P,
>>>>>>>>>>>>>>>> ERL_MAX_PORTS,
>>>>>>>>>>>>>>>> and  ERL_MAX_ETS_TABLES.  Are any of these set?  If so, to
>>>>>>>>>>>>>>>> what?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Can you also pate the eleveldb section of your app.config?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Wed, Apr 3, 2013 at 7:41 AM, Ingo Rockel
>>>>>>>>>>>>>>>> <ingo.rockel at bluelionmobile.com> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Hi Evan,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I'm not sure, I find a lot of these:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> 2013-03-30 23:27:52.992 [error]
>>>>>>>>>>>>>>>>> <0.8036.323>@riak_api_pb_server:handle_info:141 Unrecognized
>>>>>>>>>>>>>>>>> message
>>>>>>>>>>>>>>>>> {22243034,{error,timeout}}
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> and some of these at the same time one of the kind below gets
>>>>>>>>>>>>>>>>> logged
>>>>>>>>>>>>>>>>> (although the one has a different time stamp):
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> 2013-03-30 23:27:53.056 [error]
>>>>>>>>>>>>>>>>> <0.9457.323>@riak_kv_console:status:178
>>>>>>>>>>>>>>>>> Status failed error:terminated
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Ingo
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Am 03.04.2013 16:24, schrieb Evan Vigil-McClanahan:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Resending to the list:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Ingo,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> That is an indication that the protocol buffers server can't
>>>>>>>>>>>>>>>>>> spawn a
>>>>>>>>>>>>>>>>>> put fsm, which means that a put cannot be done for some
>>>>>>>>>>>>>>>>>> reason or
>>>>>>>>>>>>>>>>>> another.  Are there any other messages that appear around
>>>>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>>> time
>>>>>>>>>>>>>>>>>> that might indicate why?
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Wed, Apr 3, 2013 at 12:09 AM, Ingo Rockel
>>>>>>>>>>>>>>>>>> <ingo.rockel at bluelionmobile.com> wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> we have some performance issues with our riak cluster,
>>>>>>>>>>>>>>>>>>> from time
>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>> time
>>>>>>>>>>>>>>>>>>> we
>>>>>>>>>>>>>>>>>>> have a sudden drop in performance (already asked the list
>>>>>>>>>>>>>>>>>>> about
>>>>>>>>>>>>>>>>>>> this,
>>>>>>>>>>>>>>>>>>> no-one
>>>>>>>>>>>>>>>>>>> had an idea though). Although not the same time but on the
>>>>>>>>>>>>>>>>>>> problematic
>>>>>>>>>>>>>>>>>>> nodes
>>>>>>>>>>>>>>>>>>> we have a lot of these messages from time to time:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> 2013-04-02 21:41:11.173 [warning] <0.25646.475> ** Can not
>>>>>>>>>>>>>>>>>>> start
>>>>>>>>>>>>>>>>>>> proc_lib:init_p
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> ,[<0.14556.474>,[<0.9519.474>,riak_api_pb_sup,riak_api_sup,<0.1291.0>],riak_kv_p
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> ut_fsm,start_link,[{raw,65032165,<0.9519.474>},{r_object,<<109>>,<<77,115,124,49
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> ,53,55,57,56,57,56,50,124,49,51,54,52,57,51,49,54,49,49,53,49,50,52,53,54>>,[{r_
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> content,{dict,0,16,16,8,80,48,{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},<<>>}],[],{dict,2,16,16,8,8
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> 0,48,{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},{{[],[],[],[],[],[],[],[]
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> ,[],[],[[<<99,111,110,116,101,110,116,45,116,121,112,101>>,97,112,112,108,105,99
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> ,97,116,105,111,110,47,106,115,111,110]],[],[],[],[],[[<<99,104,97,114,115,101,1
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> 16>>,85,84,70,45,56]]}}},<<123,34,115,116,34,58,50,44,34,116,34,58,49,44,34,99,3
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> 4,58,34,66,117,116,32,115,104,101,32,105,115,32,103,111,110,101,44,32,110,32,101
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> ,118,101,110,32,116,104,111,117,103,104,32,105,109,32,110,111,116,32,105,110,32,
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> 117,114,32,99,105,116,121,32,105,32,108,111,118,101,32,117,32,110,100,32,105,32,
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> 109,101,97,110,32,105,116,32,58,39,40,34,44,34,114,34,58,49,52,51,52,54,52,51,57
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> ,44,34,115,34,58,49,53,55,57,56,57,56,50,44,34,99,116,34,58,49,51,54,52,57,51,49
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> ,54,49,49,53,49,50,44,34,97,110,34,58,102,97,108,115,101,44,34,115,107,34,58,49,
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> 51,54,52,57,51,49,54,49,49,53,49,50,52,53,54,44,34,115,117,34,58,48,125>>},[{tim
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> eout,60000}]]] on 'riak at 172.22.3.12' **
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Can anyone explain to me what these messages mean and if I
>>>>>>>>>>>>>>>>>>> need
>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>> do
>>>>>>>>>>>>>>>>>>> something about it? Could these messages be in any way
>>>>>>>>>>>>>>>>>>> related
>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>> performance issues?
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Ingo
>>>>>>>>>>>>>>>>>>>
>>>


-- 
Software Architect

Blue Lion mobile GmbH
Tel. +49 (0) 221 788 797 14
Fax. +49 (0) 221 788 797 19
Mob. +49 (0) 176 24 87 30 89

ingo.rockel at bluelionmobile.com
 >>> qeep: Hefferwolf

www.bluelionmobile.com
www.qeep.net




More information about the riak-users mailing list