riak-cs fails to start after reimporting Docker container

Jean-Marc Le Roux jeanmarc.leroux at aerys.in
Wed Feb 15 16:16:22 EST 2017


Forgot to mention ACLs are alright AFAIK :

root at b4394bf1de78:/var/lib/riak# ls -la
total 52
drwxr-xr-x. 10 riak riak  179 Feb  9 23:43 .
drwxr-xr-x.  1 root root   95 Feb 15 20:48 ..
-r--------.  1 riak riak   20 Feb  9 01:00 .erlang.cookie
drwxrwxr-x. 67 riak riak 8192 Feb 15 21:31 anti_entropy
drwxrwxr-x. 66 riak riak 8192 Feb  9 23:42 bitcask
drwxrwxr-x.  3 riak riak   40 Feb  9 23:42 cluster_meta
drwxrwxr-x.  2 riak riak  225 Feb 15 22:09 generated.configs
drwxrwxr-x.  2 riak riak 8192 Feb 15 22:09 kv_vnode
drwxrwxr-x. 66 riak riak 8192 Feb  9 23:42 leveldb
drwxrwxr-x.  2 riak riak    6 Feb 15 22:14 riak_kv_exchange_fsm
drwxr-xr-x.  2 riak riak  186 Feb 15 22:09 ring

2017-02-15 22:13 GMT+01:00 Jean-Marc Le Roux <jeanmarc.leroux at aerys.in>:

> Hi,
>
> I'll try to send the log archive ASAP.
> Here is what I get in /var/log/riak/error.log after running riak-admin
> repair-2i :
>
> 2017-02-15 22:09:06.535 [error] <0.3287.0>@riak_kv_2i_aae:repair_partition:297
> Failed to acquire hashtree lock on partition 125597796958124469533129165311
> 5555720016817029120
> 2017-02-15 22:09:06.535 [error] <0.3288.0>@riak_kv_2i_aae:repair_partition:297
> Failed to acquire hashtree lock on partition 127881393266454005342822422862
> 6747642198940975104
> 2017-02-15 22:09:06.535 [error] <0.3289.0>@riak_kv_2i_aae:repair_partition:297
> Failed to acquire hashtree lock on partition 479555224749202520035584085735
> 030365824602865664
> 2017-02-15 22:09:06.535 [error] <0.3290.0>@riak_kv_2i_aae:repair_partition:297
> Failed to acquire hashtree lock on partition 502391187832497878132516661246
> 222288006726811648
> 2017-02-15 22:09:06.535 [error] <0.3291.0>@riak_kv_2i_aae:repair_partition:297
> Failed to acquire hashtree lock on partition 111896219108147254674969620004
> 8404186924073353216
>
> I tried to remove all "LOCK" files in /var/lib/riak but to no avail...
> I'm guessing there is something here...
>
> Any idea ?
>
> 2017-02-09 17:37 GMT+01:00 Luke Bakken <lbakken at basho.com>:
>
>> Hi Jean-Marc -
>>
>> Can you provide a complete archive of the log directory? I wonder if
>> another file might have more information.
>>
>> --
>> Luke Bakken
>> Engineer
>> lbakken at basho.com
>>
>> On Thu, Feb 9, 2017 at 1:58 AM, Jean-Marc Le Roux
>> <jeanmarc.leroux at aerys.in> wrote:
>> >
>> > Hello,
>> >
>> > here is the original github issue :
>> >
>> > https://github.com/basho/riak_cs/issues/1329
>> >
>> > I'm using riak-cs 2.1.1-1.el6 with stanchion 1.5.0-1.el6 on CentOS 6.8
>> in a Docker container.
>> > To make the data persistent, the following directories are mounted from
>> outside the container :
>> >
>> > /var/log
>> > /var/lib/riak/
>> >
>> > Everything works fine except when I remove/reimport the container.
>> > Even when it's the same container.
>> > The riak data is here in /var/lib/riak (bitcask and leveldb stuff).
>> ACLs look fine on those files.
>> >
>> > Riak starts. Stanchion starts. But riak-cs won't start.
>> > With a riak-cs concole, it looks like the problem is here :
>> >>
>> >> (riak-cs at 127.0.0.1)1> [os_mon] memory supervisor port (memsup):
>> Erlang has closed
>> >>
>> >> =INFO REPORT==== 18-Jan-2017::09:38:31 ===
>> >>     alarm_handler: {clear,system_memory_high_watermark}
>> >> [os_mon] cpu supervisor port (cpu_sup): Erlang has closed
>> >> {"Kernel pid terminated",application_controller,"{application_start_
>> failure,riak_cs,{notfound,{riak_cs_app,start,[normal,[]]}}}"}
>> >
>> > var/log/riak-cs/access.log.2017_01_18_09 is empty.
>> > Here is what /var/log/riak-cs/crash.log says:
>> >>
>> >> 2017-01-18 09:38:31 =CRASH REPORT====
>> >>   crasher:
>> >>     initial call: application_master:init/4
>> >>     pid: <0.148.0>
>> >>     registered_name: []
>> >>     exception exit: {{notfound,{riak_cs_app,start,
>> [normal,[]]}},[{application_master,init,4,[{file,"applicatio
>> n_master.erl"},{line,133}]},{proc_lib,init_p_do_apply,3,[{
>> file,"proc_lib.erl"},{line,239}]}]}
>> >>     ancestors: [<0.147.0>]
>> >>     messages: [{'EXIT',<0.149.0>,normal}]
>> >>     links: [<0.147.0>,<0.7.0>]
>> >>     dictionary: []
>> >>     trap_exit: true
>> >>     status: running
>> >>     heap_size: 376
>> >>     stack_size: 27
>> >>     reductions: 119
>> >>   neighbours:
>>
>
>
>
> --
> *Jean-Marc Le Roux*
>
>
> Founder and CEO of Aerys (http://aerys.in)
>
> Blog: http://blogs.aerys.in/jeanmarc-leroux
> Cell: (+33)6 20 56 45 78 <+33%206%2020%2056%2045%2078>
> Phone: (+33)9 72 40 17 58 <+33%209%2072%2040%2017%2058>
>



-- 
*Jean-Marc Le Roux*


Founder and CEO of Aerys (http://aerys.in)

Blog: http://blogs.aerys.in/jeanmarc-leroux
Cell: (+33)6 20 56 45 78
Phone: (+33)9 72 40 17 58
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.basho.com/pipermail/riak-users_lists.basho.com/attachments/20170215/e1a86932/attachment-0002.html>


More information about the riak-users mailing list