Nightly Prune

Chad Engler Chad.Engler at patlive.com
Thu Feb 7 10:58:36 EST 2013


Let me clarify that a bit, after 10 days there will be about 2 - 3
millions records in that bucket, about 200,000 - 300,000 of those
records will need to be pruned off nightly.

 

-Chad

 

From: riak-users [mailto:riak-users-bounces at lists.basho.com] On Behalf
Of Chad Engler
Sent: Thursday, February 07, 2013 10:53 AM
To: riak-users at lists.basho.com
Subject: Nightly Prune

 

We need records in our database to only live for a certain period of
time (and we are using Bitcask). At first glance, expiry_secs seems to
be exactly what we are looking for.

 

The problem is, these records need to be pruned when they are older than
10 days *according to date information in the record*, not the time that
it was inserted. Many times we will repost events and things from the
day before or a week before into riak. So if a week old entry goes in,
it should only live for 3 more days before being pruned.

 

I've noticed that people are saying map reduces over large buckets are
not recommended for things like this, so what is a good way to prune off
old entries when they are over 10 days old (according to date data
within the record)? Please also note we will have 2 - 3 million keys in
a single bucket, which needs to be pruned along with 200,000 - 250,000
keys in another bucket that also need to be pruned.

 

Thanks!

 

Chad Engler

Telephony Programmer

PATLive

1.800.775.7790 x746

chad.engler at patlive.com <mailto:chad.engler at patlive.com> 

 

Hosted Communications | Friendly Service    www.patlive.com
<http://www.patlive.com/> 

 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.basho.com/pipermail/riak-users_lists.basho.com/attachments/20130207/4cb06fe4/attachment.html>


More information about the riak-users mailing list