riak-cs reclaim storage

Hector Castro hector at basho.com
Mon Jun 9 21:13:03 EDT 2014


This mailing list entry was discussed a bit in IRC today. Below are a few notes from the exchange between Tyler and our very own Joe Caswell. Hopefully they help provide some insight into Riak CS GC for others:

- Riak CS data blocks are stored in Bitcask.

- `riak-admin vnode-status` outputs several important Bitcask details: fragmentation percentage, dead bytes, and total size for each Bitcask file.

- If the dead bytes portion of `riak-admin node-status` doesn’t account for the amount of storage you expect to reclaim, Riak CS GC may not have happened yet.

- As an example:

{"/var/lib/riak/bitcask/0/3054.bitcask.data",4,91282721,2147205296},

This snippet shows a 2GB file (3054.bitcask.data) that is 4% fragmented and has 90MB of dead bytes that could be removed via a merge.

- Bitcask’s default `frag_merge_trigger` is 60% and `dead_bytes_merge_trigger` is 512MB.

- The merge triggering process in Riak is called very 3 minutes. If any triggers are met, the merge is added to a queue. A single merge worker pulls from that queue per node (merge triggers occur per vnode).

- Adding `{log_needs_merge, true}` to the Bitcask section produces a log entry whenever a trigger is met.

- If you want to reclaim space used by Riak CS faster, lower Bitcask’s default merge triggers and thresholds.

- For additional details around tuning Bitcask, see:

http://docs.basho.com/riak/latest/ops/advanced/backends/bitcask/#Configuring-Bitcask
http://docs.basho.com/riak/latest/ops/advanced/backends/bitcask/#Tuning-Bitcask 

--  
Hector

On June 9, 2014 at 9:08:03 PM, Kota Uenishi (kota at basho.com) wrote:
> Well, maybe you had deleted your objects when you set leeway_second
> with 1 day, and later you changed leeway_second to 5min? Changing
> leeway_second won't change deletion timing of already deleted objects.
>  
> FYI, maybe it's not what you want, but there's a detailed description
> [1] on internal design and behaviour of garbage collection.
>  
> Of course Riak CS and Riak is not a black box, how Riak CS uses Riak
> is open-sourced and not hidden. For developers and hard-core
> operators, there's just very primitive tool to diagnose Riak CS by
> hitting Riak directly. It is riak_cs_inspector, installed in
> /usr/lib/riak-cs/lib/riak_cs-1.4.5/priv/tools/internal/riak_cs_inspector.erl  
> . See [2] for detailed usage. It prints Riak CS internal data in
> structured manner but it is still hard to find the answer "how many
> objects to be deleted today?" .
>  
> I think your assumption is very natural, because there should be some
> operational visibility tools like you said.
>  
> [1] https://github.com/basho/riak_cs/wiki/Object-Chunking-and-Garbage-Collection#garbage-collection  
> [2] https://gist.github.com/kuenishi/bf286cf28f16ebbf1256
>  
> On Tue, Jun 10, 2014 at 1:46 AM, Tyler Flint wrote:
> > Unfortunately, it doesn’t seem to have made a difference.
> >
> > I did restart the riak-cs process after updating the app.config. Triggering a riak-cs-gc  
> batch did trigger a job, but did not seem to yield any results.
> >
> > I’m a bit concerned that I’m unable to root cause the issue here. Is there some sort of  
> visibility tools that I’m overlooking? I would assume that I’d be able to query into riak-cs  
> and find “x entries are ready to be garbage collected” or something to provide some visibility.  
> I’m not even sure if the problem is within riak-cs or riak/bitcask.
> >
> > Are riak-cs and riak just black boxes? I can’t imagine this to be the case if people are  
> using these solutions in production. Please tell me I’m just an idiot and not aware of  
> some utility/configuration to make this work.
> >
> > Thanks
> >
> > On Jun 8, 2014, at 7:22 PM, Tyler Flint wrote:
> >
> >> Yes it is riak 1.4.5
> >>
> >> Let me see if that helps.
> >>
> >> Thanks
> >>
> >>
> >>
> >>> On Jun 8, 2014, at 7:20 PM, Kota Uenishi wrote:
> >>>
> >>> Spelling mistake, sorry:
> >>>
> >>> {gc_paginated_indexes, true},
> >>>
> >>>> On Mon, Jun 9, 2014 at 10:10 AM, Kota Uenishi wrote:
> >>>> Hi,
> >>>>
> >>>> Are you using 1.4.5? I'm not sure but you may have a log message like this:
> >>>>
> >>>> "Error occurred trying to query from time 0 to ~p in gc key index. Reason: ~p"
> >>>>
> >>>> in console.log. If so, it's even failing not only GC but in listing
> >>>> objects to be collected. Then setting a line
> >>>>
> >>>> {gc_pagenated_indexes, true},
> >>>>
> >>>> into riak_cs section of your app.config . This is implicit option and
> >>>> its default is false.
> >>>>
> >>>>
> >>>>> On Fri, Jun 6, 2014 at 11:52 PM, Tyler Flint wrote:
> >>>>> Thank you in advance for any assistance.
> >>>>>
> >>>>> To explore riak-cs, I setup a 2 node cluster with riak and riak-cs on each node, and  
> put the riak nodes on the same cluster. I then proceeded to use risk-cs by uploading a 3G  
> file using the multi-part upload protocol. I was unaware, however, that I needed to finalize  
> the multipart upload and thus had many in progress multi-part uploads. Through the duration  
> of my troubleshooting, the storage on the filesystem grew to 224G. Once I discovered  
> my error, I proceeded to cancel all of the previous uploads. At this point there are no  
> uploads in progress, and only a single 3G file available in the bucket. However, it has  
> been 48 hours since the exploration and the storage use is still 224G on both machines.  
> >>>>>
> >>>>> I have attempted multiple strategies in attempt to reclaim space, all without  
> any effect.
> >>>>>
> >>>>> Within riak-cs:
> >>>>>
> >>>>> - I have shortened the gc leeway duration to 5 minutes and set all of the gc intervals  
> as low as 5 minutes.
> >>>>> - I have manually triggered the riak-cs-gc batch multiple times. (After running  
> batch, status shows a gc run and eventually complete)
> >>>>>
> >>>>> Within riak:
> >>>>>
> >>>>> - I have adjusted the bitcask settings in attempt to trigger a merge.
> >>>>> - I have force merged all of the bitcask partitions manually in the erlang console.  
> >>>>>
> >>>>> None of these strategies seem to have any effect whatsoever.
> >>>>>
> >>>>> At this point I’m not even sure where to start, nor do I know if the issue is within  
> riak-cs or riak (via bitcask). Any suggestions would be appreciated.
> >>>>>
> >>>>> Thanks
> >>>>>
> >>>>> Tyler
> >>>>> _______________________________________________
> >>>>> riak-users mailing list
> >>>>> riak-users at lists.basho.com
> >>>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com  
> >>>>
> >>>>
> >>>>
> >>>> --
> >>>> Kota UENISHI / @kuenishi
> >>>> Basho Japan KK
> >>>
> >>>
> >>>
> >>> --
> >>> Kota UENISHI / @kuenishi
> >>> Basho Japan KK
> >
>  
>  
>  
> --
> Kota UENISHI / @kuenishi
> Basho Japan KK
>  
> _______________________________________________
> riak-users mailing list
> riak-users at lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>  





More information about the riak-users mailing list