riak-cs reclaim storage

Kota Uenishi kota at basho.com
Mon Jun 9 21:07:23 EDT 2014


Well, maybe you had deleted your objects when you set leeway_second
with 1 day, and later you changed leeway_second to 5min? Changing
leeway_second won't change deletion timing of already deleted objects.

FYI, maybe it's not what you want, but there's a detailed description
[1] on internal design and behaviour of garbage collection.

Of course Riak CS and Riak is not a black box, how Riak CS uses Riak
is open-sourced and not hidden. For developers and hard-core
operators, there's just very primitive tool to diagnose Riak CS by
hitting Riak directly. It is riak_cs_inspector, installed in
/usr/lib/riak-cs/lib/riak_cs-1.4.5/priv/tools/internal/riak_cs_inspector.erl
. See [2] for detailed usage. It prints Riak CS internal data in
structured manner but it is still hard to find the answer "how many
objects to be deleted today?" .

I think your assumption is very natural, because there should be some
operational visibility tools like you said.

[1] https://github.com/basho/riak_cs/wiki/Object-Chunking-and-Garbage-Collection#garbage-collection
[2] https://gist.github.com/kuenishi/bf286cf28f16ebbf1256

On Tue, Jun 10, 2014 at 1:46 AM, Tyler Flint <tylerflint at gmail.com> wrote:
> Unfortunately, it doesn’t seem to have made a difference.
>
> I did restart the riak-cs process after updating the app.config. Triggering a riak-cs-gc batch did trigger a job, but did not seem to yield any results.
>
> I’m a bit concerned that I’m unable to root cause the issue here. Is there some sort of visibility tools that I’m overlooking? I would assume that I’d be able to query into riak-cs and find “x entries are ready to be garbage collected” or something to provide some visibility. I’m not even sure if the problem is within riak-cs or riak/bitcask.
>
> Are riak-cs and riak just black boxes? I can’t imagine this to be the case if people are using these solutions in production. Please tell me I’m just an idiot and not aware of some utility/configuration to make this work.
>
> Thanks
>
> On Jun 8, 2014, at 7:22 PM, Tyler Flint <tylerflint at gmail.com> wrote:
>
>> Yes it is riak 1.4.5
>>
>> Let me see if that helps.
>>
>> Thanks
>>
>>
>>
>>> On Jun 8, 2014, at 7:20 PM, Kota Uenishi <kota at basho.com> wrote:
>>>
>>> Spelling mistake, sorry:
>>>
>>> {gc_paginated_indexes, true},
>>>
>>>> On Mon, Jun 9, 2014 at 10:10 AM, Kota Uenishi <kota at basho.com> wrote:
>>>> Hi,
>>>>
>>>> Are you using 1.4.5? I'm not sure but you may have a log message like this:
>>>>
>>>> "Error occurred trying to query from time 0 to ~p in gc key index. Reason: ~p"
>>>>
>>>> in console.log. If so, it's even failing not only GC but in listing
>>>> objects to be collected. Then setting a line
>>>>
>>>> {gc_pagenated_indexes, true},
>>>>
>>>> into riak_cs section of your app.config . This is implicit option and
>>>> its default is false.
>>>>
>>>>
>>>>> On Fri, Jun 6, 2014 at 11:52 PM, Tyler Flint <tylerflint at gmail.com> wrote:
>>>>> Thank you in advance for any assistance.
>>>>>
>>>>> To explore riak-cs, I setup a 2 node cluster with riak and riak-cs on each node, and put the riak nodes on the same cluster. I then proceeded to use risk-cs by uploading a 3G file using the multi-part upload protocol. I was unaware, however, that I needed to finalize the multipart upload and thus had many in progress multi-part uploads. Through the duration of my troubleshooting, the storage on the filesystem grew to 224G. Once I discovered my error, I proceeded to cancel all of the previous uploads. At this point there are no uploads in progress, and only a single 3G file available in the bucket. However, it has been 48 hours since the exploration and the storage use is still 224G on both machines.
>>>>>
>>>>> I have attempted multiple strategies in attempt to reclaim space, all without any effect.
>>>>>
>>>>> Within riak-cs:
>>>>>
>>>>> - I have shortened the gc leeway duration to 5 minutes and set all of the gc intervals as low as 5 minutes.
>>>>> - I have manually triggered the riak-cs-gc batch multiple times. (After running batch, status shows a gc run and eventually complete)
>>>>>
>>>>> Within riak:
>>>>>
>>>>> - I have adjusted the bitcask settings in attempt to trigger a merge.
>>>>> - I have force merged all of the bitcask partitions manually in the erlang console.
>>>>>
>>>>> None of these strategies seem to have any effect whatsoever.
>>>>>
>>>>> At this point I’m not even sure where to start, nor do I know if the issue is within riak-cs or riak (via bitcask). Any suggestions would be appreciated.
>>>>>
>>>>> Thanks
>>>>>
>>>>> Tyler
>>>>> _______________________________________________
>>>>> riak-users mailing list
>>>>> riak-users at lists.basho.com
>>>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>>>
>>>>
>>>>
>>>> --
>>>> Kota UENISHI / @kuenishi
>>>> Basho Japan KK
>>>
>>>
>>>
>>> --
>>> Kota UENISHI / @kuenishi
>>> Basho Japan KK
>



-- 
Kota UENISHI / @kuenishi
Basho Japan KK




More information about the riak-users mailing list