riak-cs reclaim storage

Tyler Flint tylerflint at gmail.com
Mon Jun 9 12:46:55 EDT 2014


Unfortunately, it doesn’t seem to have made a difference. 

I did restart the riak-cs process after updating the app.config. Triggering a riak-cs-gc batch did trigger a job, but did not seem to yield any results.

I’m a bit concerned that I’m unable to root cause the issue here. Is there some sort of visibility tools that I’m overlooking? I would assume that I’d be able to query into riak-cs and find “x entries are ready to be garbage collected” or something to provide some visibility. I’m not even sure if the problem is within riak-cs or riak/bitcask.

Are riak-cs and riak just black boxes? I can’t imagine this to be the case if people are using these solutions in production. Please tell me I’m just an idiot and not aware of some utility/configuration to make this work.

Thanks

On Jun 8, 2014, at 7:22 PM, Tyler Flint <tylerflint at gmail.com> wrote:

> Yes it is riak 1.4.5
> 
> Let me see if that helps. 
> 
> Thanks
> 
> 
> 
>> On Jun 8, 2014, at 7:20 PM, Kota Uenishi <kota at basho.com> wrote:
>> 
>> Spelling mistake, sorry:
>> 
>> {gc_paginated_indexes, true},
>> 
>>> On Mon, Jun 9, 2014 at 10:10 AM, Kota Uenishi <kota at basho.com> wrote:
>>> Hi,
>>> 
>>> Are you using 1.4.5? I'm not sure but you may have a log message like this:
>>> 
>>> "Error occurred trying to query from time 0 to ~p in gc key index. Reason: ~p"
>>> 
>>> in console.log. If so, it's even failing not only GC but in listing
>>> objects to be collected. Then setting a line
>>> 
>>> {gc_pagenated_indexes, true},
>>> 
>>> into riak_cs section of your app.config . This is implicit option and
>>> its default is false.
>>> 
>>> 
>>>> On Fri, Jun 6, 2014 at 11:52 PM, Tyler Flint <tylerflint at gmail.com> wrote:
>>>> Thank you in advance for any assistance.
>>>> 
>>>> To explore riak-cs, I setup a 2 node cluster with riak and riak-cs on each node, and put the riak nodes on the same cluster. I then proceeded to use risk-cs by uploading a 3G file using the multi-part upload protocol. I was unaware, however, that I needed to finalize the multipart upload and thus had many in progress multi-part uploads. Through the duration of my troubleshooting, the storage on the filesystem grew to 224G. Once I discovered my error, I proceeded to cancel all of the previous uploads. At this point there are no uploads in progress, and only a single 3G file available in the bucket. However, it has been 48 hours since the exploration and the storage use is still 224G on both machines.
>>>> 
>>>> I have attempted multiple strategies in attempt to reclaim space, all without any effect.
>>>> 
>>>> Within riak-cs:
>>>> 
>>>> - I have shortened the gc leeway duration to 5 minutes and set all of the gc intervals as low as 5 minutes.
>>>> - I have manually triggered the riak-cs-gc batch multiple times. (After running batch, status shows a gc run and eventually complete)
>>>> 
>>>> Within riak:
>>>> 
>>>> - I have adjusted the bitcask settings in attempt to trigger a merge.
>>>> - I have force merged all of the bitcask partitions manually in the erlang console.
>>>> 
>>>> None of these strategies seem to have any effect whatsoever.
>>>> 
>>>> At this point I’m not even sure where to start, nor do I know if the issue is within riak-cs or riak (via bitcask). Any suggestions would be appreciated.
>>>> 
>>>> Thanks
>>>> 
>>>> Tyler
>>>> _______________________________________________
>>>> riak-users mailing list
>>>> riak-users at lists.basho.com
>>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>> 
>>> 
>>> 
>>> --
>>> Kota UENISHI / @kuenishi
>>> Basho Japan KK
>> 
>> 
>> 
>> -- 
>> Kota UENISHI / @kuenishi
>> Basho Japan KK





More information about the riak-users mailing list