Riak CS expiration

Toby Corkindale toby at dryft.net
Mon Feb 9 00:39:20 EST 2015


On 7 February 2015 at 06:21, Kota Uenishi <kota at basho.com> wrote:
> Bitcask doesn't have such interface like hooking anything. As of
> internal data structure of Riak CS built on top of Riak, bitcask keys
> don't correspond simply to leveldb key. Say, things like this cannot
> happen: "all tombstones of blocks of this object were collected and
> space were reclaimed by merged, so I'm gonna officially hit S3
> interface to delete that object". This is because blocks are
> distributed all across the cluster.
>
> I'd rather recommend to make up a simple cron job that walks over
> objects and see timestamp and hit CS API to delete them when they're
> old, or wait for lifecycle API gets implemented.

Definitely. And it's seriously only a few lines of code -- mine is
implemented like this, in Scala:

    val allFiles = s3.listObjects(sys.env("S3_BUCKET"))

    allFiles filter(_.getLastModifiedDate.before(expiryDate)) foreach { f =>
      logger.info(s"Expiring file=${f.getKey}")
      s3.deleteObject(bucket, f.getKey)
    }


> On Fri, Feb 6, 2015 at 12:24 AM, Damien Krotkine <dkrotkine at gmail.com> wrote:
>>
>> Hi,
>>
>> As far as I understand, Riak CS uses bitcask for data storage and
>> leveldb for metadata.
>>
>> Trying to implement expiration of data in riak cs, bitcask expiration
>> works fine, but of course metadata are still in the leveldb backend. Is
>> there any way to expire metadata from riak cs automatically ?
>>
>> Ideally, I'd like to be able to install a hook or callback on the
>> bitcask compaction. Is that possible ?
>>
>> Thanks,
>> dams
>>
>> _______________________________________________
>> riak-users mailing list
>> riak-users at lists.basho.com
>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
>
>
> --
> Kota UENISHI / @kuenishi
> Basho Japan KK
>
> _______________________________________________
> riak-users mailing list
> riak-users at lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com



-- 
Turning and turning in the widening gyre
The falcon cannot hear the falconer
Things fall apart; the center cannot hold
Mere anarchy is loosed upon the world




More information about the riak-users mailing list