Leveled - Another Erlang Key-Value store

Martin Sumner martin.sumner at adaptip.co.uk
Tue Feb 28 09:04:36 EST 2017


Over the past few months I've been working on an alternative pure-Erlang
Key/Value store to act as a backend to Riak KV.  This is now open source
and available at

https://github.com/martinsumner/leveled

The store is a work-in-progress prototype, originally started to better
understand the impact of different trade-offs in LSM-Tree design.  The aim
is to:

- provide a fully-featured Riak backend (e.g. secondary index, object
expiry support etc)
- provide stable throughput with larger object sizes (> 4KB)
- provide a simpler and more flexible path to making Riak KV changes
end-to-end

The prime change in the store when compared to HanoiDB or eleveldb is that
storage is split with only Keys & Metadata being placed in the merge tree,
and the full object living to the side in a series of CDB-based journals.
The intention of this is to:

- reduce write amplification and ease page-cache pollution issues on
scanning events.
- support faster HEAD requests than GET requests, and in parallel an
alternative Riak KV branch has been produced to move from an n-GET model to
a n-HEAD 1-GET model of fetching data for both KV GET and KV PUT operations

The impact of this has been to improve throughput for larger object sizes
where disk I/O and not CPU is the current limit on throughput.  The
advantage increases the greater the object size, and the tighter the
constraint on disk.

Please visit the github page, I've tried to write up as much about the
project as I can.  There's the results of various volume tests, information
on the research which prompted the design, an overview of the design itself
and some hints as to what I expect to try next with leveled.

Any feedback, please mail me, raise an issue on github, or ping me @masleeds

Cheers

Martin
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.basho.com/pipermail/riak-users_lists.basho.com/attachments/20170228/349e2e4c/attachment-0002.html>


More information about the riak-users mailing list