object sizes

Matthew Brender mbrender at basho.com
Thu Apr 16 16:03:55 EDT 2015


Hi Alex,

That is an interesting question! I haven't seen a request like that in
our backlog, so feel free to open a new issue [1]. I'm curious: why
not use something like sys.getsizeof [2]?

[1] https://github.com/basho/riak-python-client/issues
[2] http://stackoverflow.com/questions/449560/how-do-i-determine-the-size-of-an-object-in-python

Matt Brender | Developer Advocacy Lead
Basho Technologies
t: @mjbrender


On Mon, Apr 13, 2015 at 7:26 AM, Alex De la rosa
<alex.rosa.box at gmail.com> wrote:
> Hi Bryan,
>
> Thanks for your answer; i don't know how to code in erlang, so all my system
> relies on Python.
>
> Following Ciprian's curl suggestion, I tried to compare it with this python
> code during the weekend:
>
> Map object:
> curl -I
>> 1058 bytes
> print sys.getsizeof(obj.value)
>> 3352 bytes
>
> Standard object:
> curl -I
>> 9718 bytes
> print sys.getsizeof(obj.encoded_data)
>> 9755 bytes
>
> The standard object seems pretty accurate in both approaches even the image
> binary data was only 5kbs (I assume some overhead here)
>
> The map object is about 3x the difference between curl and getting the
> object via Python.
>
> Not so sure if this is a realistic way to measure their growth (moreover
> because the objects i would need this monitorization are Maps, not unaltered
> binary data that I can know the size before storing it).
>
> Would it be possible in some way that the Python get() function would return
> something like "obj.content-lenght" returning the size is currently taking?
> that would be a pretty nice feature.
>
> Thanks!
> Alex
>
> On Mon, Apr 13, 2015 at 12:47 PM, bryan hunt <bhunt at basho.com> wrote:
>>
>> Alex,
>>
>>
>> Maps and Sets are stored just like a regular Riak object, but using a
>> particular data structure and object serialization format. As you have
>> observed, there is an overhead, and you want to monitor the growth of these
>> data structures.
>>
>> It is possible to write a MapReduce map function (in Erlang) which
>> retrieves a provided object by type/bucket/id and returns the size of it's
>> data. Would such a thing be of use?
>>
>> It would not be hard to write such a module, and I might even have some
>> code for doing so if you are interested. There are also reasonably good
>> examples in our documentation -
>> http://docs.basho.com/riak/latest/dev/advanced/mapreduce
>>
>> I haven't looked at the Python PB API in a while, but I'm reasonably
>> certain it supports the invocation of MapReduce jobs.
>>
>> Bryan
>>
>>
>> On 10 Apr 2015, at 13:51, Alex De la rosa <alex.rosa.box at gmail.com> wrote:
>>
>> Also, I forgot, i'm most interested on bucket_types instead of simple riak
>> buckets. Being able how my mutable data inside a MAP/SET has grown.
>>
>> For a traditional standard bucket I can calculate the size of what I'm
>> sending before, so Riak won't get data bigger than 1MB. Problem arise in
>> MAPS/SETS that can grown.
>>
>> Thanks,
>> Alex
>>
>> On Fri, Apr 10, 2015 at 2:47 PM, Alex De la rosa <alex.rosa.box at gmail.com>
>> wrote:
>>>
>>> Well... using the HTTP Rest API would make no sense when using the PB
>>> API... would be extremely costly to maintain, also it may include some extra
>>> bytes on the transport.
>>>
>>> I would be interested on being able to know the size via Python itself
>>> using the PB API as I'm doing.
>>>
>>> Thanks anyway,
>>> Alex
>>>
>>> On Fri, Apr 10, 2015 at 1:58 PM, Ciprian Manea <ciprian at basho.com> wrote:
>>>>
>>>> Hi Alex,
>>>>
>>>> You can always query the size of a riak object using `curl` and the REST
>>>> API:
>>>>
>>>> i.e. curl -I <riak-node-ip>:8098/buckets/test/keys/demo
>>>>
>>>>
>>>> Regards,
>>>> Ciprian
>>>>
>>>> On Thu, Apr 9, 2015 at 12:11 PM, Alex De la rosa
>>>> <alex.rosa.box at gmail.com> wrote:
>>>>>
>>>>> Hi there,
>>>>>
>>>>> I'm using the python client (by the way).
>>>>>
>>>>> obj = RIAK.bucket('my_bucket').get('my_key')
>>>>>
>>>>> Is there any way to know the actual size of an object stored in Riak?
>>>>> to make sure something mutable (like a set) didn't added up to more than 1MB
>>>>> in storage size.
>>>>>
>>>>> Thanks!
>>>>> Alex
>>>>>
>>>>> _______________________________________________
>>>>> riak-users mailing list
>>>>> riak-users at lists.basho.com
>>>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>>>>
>>>>
>>>
>>
>> _______________________________________________
>> riak-users mailing list
>> riak-users at lists.basho.com
>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>
>>
>
>
> _______________________________________________
> riak-users mailing list
> riak-users at lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>


More information about the riak-users mailing list