Counting .SST tables in Level DB

Matthew Von-Maszewski matthewv at basho.com
Mon Oct 21 13:35:17 EDT 2013


responses inline in '[ ]'
On Oct 21, 2013, at 1:20 PM, Dave King <djking at gmail.com> wrote:

> From the Riak LevelDB Page
> http://docs.basho.com/riak/latest/ops/advanced/backends/leveldb/
> 
> "Where server resources allow, the value of max_open_files should exceed the count of .sst table files within the vnode's database directory."
> 
> Is the vnode's database directory the numbered directory under the leveldb directory? So the process is to count the number of .sst files distributed acrooss the sst_# directories.    Is that correct?
> 
[Yes to both questions above.]


> If It's not correct what are the rules for counting .sst files?
> 
[ Your previous question was correct.  But here is a simple command line count:

find /var/db/riak/leveldb/0 -name '*.sst' | wc -l

Substitute your favorite numbered directory for the "/0" in my example.]

> I'm guessing this is something you'd have to tune after deployment as the number of .sst files would grow with data growth.  Any guide lines for calculating the max_open_files before deployment?
> 
[The underlying goal is cover your working set of files via max_open_files, then give the remainder of memory to the cache_size (again size by vnode).  I have attached a spreadsheet for Riak 1.4 that helps with the calculations.  

This sizing issue is a royal pain the tush.  Riak 2.0 will simplify this greatly.  You tell it the total amount of memory that leveldb can use, no matter how many vnodes, no matter whether active anti-entropy is used.  Everything is then calculated and dynamically updated as file counts change and as vnode counts on a server change.  Look forward to that in 2.0.]


> - Peace
> Dave
> 
> _______________________________________________
> riak-users mailing list
> riak-users at lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.basho.com/pipermail/riak-users_lists.basho.com/attachments/20131021/82491425/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: leveldb_sizing_1.4.xls
Type: application/octet-stream
Size: 27136 bytes
Desc: not available
URL: <http://lists.basho.com/pipermail/riak-users_lists.basho.com/attachments/20131021/82491425/attachment.xls>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.basho.com/pipermail/riak-users_lists.basho.com/attachments/20131021/82491425/attachment-0001.html>


More information about the riak-users mailing list