riakkit, a python riak object mapper, has hit beta!

Andrey V. Martyanov realduke at gmail.com
Thu Mar 1 23:41:00 EST 2012


Riakkit in its current implementation has a  lot of overhead. But what I
get for this price? Almost nothing. Only field validation make sense.
Simple key-value mapping is about 50-70 lines of code, not a big deal, even
for small project. Riak used mostly in web-based projects. Almost all
modern projects use JSON. The simple workflow is get JSON from the client,
validate, store in the database, query the database, send JSON to the
client. If our mapper doesn't understand the JSON and cannot operate on it
then it's not useful. We can use jsonpickle with a lot more profit. Just
jsonpickle.encode the object and store it in Riak. I'm sorry, this is my
position!

Best regards,
Andrey Martyanov

On Fri, Mar 2, 2012 at 9:01 AM, Andrey V. Martyanov <realduke at gmail.com>wrote:

> I'm just saying that to efficiently store and get the data user should
> have ability to customize the relationship. Riak has several options to get
> related data: secondary indexes, map-reduce etc. I think that the most
> important task of mapper is to validate the data and to track these
> dependencies. When you have enough meta information about objects storing
> these objects is not a big problem.
>
>
> On Fri, Mar 2, 2012 at 8:39 AM, Shuhao Wu <admin at thekks.net> wrote:
>
>> The thing about riak-python-client is that it's already pretty high
>> level, but not high level enough to be that simple.
>>
>> From what I understand, tracking relationship between models will still
>> involve a certain level of magic. Right now the magic is that when you save
>> any document, all the related document will be saved into the database as
>> well. You're saying that just tracking those relationship change is good
>> enough and saving will be up to the user?
>>
>> Shuhao
>>
>>
>> On Thu, Mar 1, 2012 at 10:20 PM, Andrey V. Martyanov <realduke at gmail.com>wrote:
>>
>>> I'm not sure about forking the riak-python-client. The the whole idea of
>>> separation the low-level client library from the high-level mapper and the
>>> like is quite good. Some sort of SQLAlchemy does.
>>>
>>> Best regards,
>>> Andrey Martyanov
>>>
>>>
>>> On Fri, Mar 2, 2012 at 8:15 AM, Shuhao Wu <admin at thekks.net> wrote:
>>>
>>>> Another thing I was considering is forking riak-python-client. Some of
>>>> the stuff in there could use some fixing as well, and maybe just take out
>>>> the core transports and such and plug that right into riakkit and skip the
>>>> notion of RiakObject all together as those API is also somewhat weird.
>>>> (such as add_link and remove_link)
>>>>
>>>> Shuhao
>>>>
>>>>
>>>>
>>>> On Thu, Mar 1, 2012 at 10:02 PM, Shuhao Wu <admin at thekks.net> wrote:
>>>>
>>>>> Hi!
>>>>>
>>>>> First off, the code is *LGPL* licensed, not GPL. So you're free to
>>>>> put this into any code you want, it's just that when you want to modify
>>>>> riakkit, you have to release your modified version.
>>>>>
>>>>> I just went over your idea and it would be a valid solution to some of
>>>>> the problems I'm having right now with my own projects with riakkit. When I
>>>>> initially started writing riakkit, I needed something to do RAD with, which
>>>>> would make sense to have an all-in-one, very simple to use framework,
>>>>> rather than working with serialization and what not. However, now I'm
>>>>> beginning to see problems in this approach when it comes to scalability as
>>>>> I'm slowing moving out of the RAD phase. I haven't done anything as I have
>>>>> not encountered any major issues yet (though I probably will).
>>>>>
>>>>> So here's what I'm thinking for the next branch ( I still have to
>>>>> maintain the master branch due to some code i've written and need ): I'm
>>>>> going to break riakkit into 2 pieces. The first piece is the underlying
>>>>> piece, which will do what you said, serialize/deserialize objects,
>>>>> validation, track relationships and what not, however, it doesn't load it
>>>>> those references up nor does it try to accomplish everything all in one
>>>>> shot, and a RAD layer that's compatible with the current API that will
>>>>> allow RAD to work with it, but still reserve the flexibility of having that
>>>>> underlying API that's relatively easy (as oppose to dead simple) to work
>>>>> with and not as messy as what's going on right now.
>>>>>
>>>>> I do agree that the code is messy.. Most of the time it's me fixing
>>>>> bugs I didn't anticipate as I go along and hacking everything into place..
>>>>>
>>>>> One thing I'm not too clear about is what you mean by "Track
>>>>> dependencies between models". I'm not sure how you think I should go about
>>>>> approaching this (which right now is a pretty big disaster.. though
>>>>> relatively stable)
>>>>>
>>>>> Cheers,
>>>>>
>>>>> Shuhao
>>>>>
>>>>> P.S. I don't want to go into the whole PEP8 discussion and 4 spaces vs
>>>>> whatever.. I just like the 2 spaces and the style guide that Google uses.
>>>>> Is that really a big deal if you're using it as a library?
>>>>>
>>>>> On Thu, Mar 1, 2012 at 9:35 PM, Andrey V. Martyanov <
>>>>> realduke at gmail.com> wrote:
>>>>>
>>>>>> Hi!
>>>>>>
>>>>>> Two things force me to avoid using Riakkit: the code is GPL licensed
>>>>>> and non pep8 compliant. So, I cannot fork it and make changes. The code
>>>>>> needs clean up, for example, json module is imported in document.py, but
>>>>>> not used. Riakkit has no conflict resolution logic, it's basically delegate
>>>>>> the work to underlying python client. If I'd like to implement my save
>>>>>> method I need to copy and rewrite more than 150 lines of code mess. What I
>>>>>> think the mapper should do:
>>>>>> 1. Serialize the objects to valid json;
>>>>>> 2. Deserialize the data back to class instances;
>>>>>> 3. Validate the fields data;
>>>>>> 4. Track dependencies between models;
>>>>>>
>>>>>> Simple example using Riakkit-like syntax;
>>>>>>
>>>>>> class Post(Document):
>>>>>>
>>>>>>     bucket_name  = 'posts'
>>>>>>
>>>>>>     title = types.StringProperty(required=True)
>>>>>>     content = types.StringProperty(required=True)
>>>>>>
>>>>>>
>>>>>> class Comment(Document):
>>>>>>
>>>>>>     bucket_name = 'comments'
>>>>>>
>>>>>>     text = types.StringProperty(required=True)
>>>>>>     post = types.IndexReferenceProperty(reference_class=Post,
>>>>>> collection_name='comments')
>>>>>>
>>>>>> >>> post = Post('title', 'content')
>>>>>> >>> comment1 = ('comment1', post=post)
>>>>>> >>> comment2 = ('comment2', post=post)
>>>>>> >>> post.is_valid()
>>>>>> True
>>>>>> >> post.serialize()
>>>>>> {
>>>>>>     "key": "post_key",
>>>>>>     "title": "title",
>>>>>>     "content": "content",
>>>>>>     "comments": [
>>>>>>         {
>>>>>>             "key": "comment1_key",
>>>>>>             "text": "comment1"
>>>>>>         },
>>>>>>         {
>>>>>>             "key": "comment2_key",
>>>>>>             "text": "comment2"
>>>>>>         },
>>>>>>     ]
>>>>>> }
>>>>>> >>> post.store()
>>>>>> True
>>>>>> >>> post = Post.get_with_key('post_key')
>>>>>> >>> post.comments
>>>>>> ['Post comment1', 'Post comment2']
>>>>>> >>> post_json_data = get_json_data_from_request()
>>>>>> >>> post = Post.deserialize(post_json_data)
>>>>>> >>> post.title
>>>>>> 'title'
>>>>>> >>> post.comments
>>>>>> ['Post comment1', 'Post comment2']
>>>>>>
>>>>>> Something like that.
>>>>>>
>>>>>>
>>>>>> Best regards,
>>>>>> Andrey Martyanov
>>>>>>
>>>>>> On Fri, Mar 2, 2012 at 5:39 AM, Sean Carey <carey at basho.com> wrote:
>>>>>>
>>>>>>> Keep up the great work Shuhao!
>>>>>>>
>>>>>>>
>>>>>>> On Mar 1, 2012, at 7:36 PM, Shuhao Wu <admin at thekks.net> wrote:
>>>>>>>
>>>>>>> Hey guys,
>>>>>>>
>>>>>>> I started writing riakkit approximately 3 months ago and announced
>>>>>>> it here. In case you missed that, Riakkit is basically an object mapper for
>>>>>>> python and riak, which seemed to be lacking in at the time. Now that 3
>>>>>>> months has passed and I have developed this project into much more than
>>>>>>> what I had envisioned to begin with. It's currently also being used and
>>>>>>> tested in a project of mine (web application) as well and functioning quite
>>>>>>> nicely. I would say that currently this is the only object mapper that can
>>>>>>> compete with their counterparts in other languages for Riak in Python,
>>>>>>> although a lot of work still needs to be done before it could fully compete.
>>>>>>>
>>>>>>> So today I felt that riakkit is "stable" and "clean" enough to
>>>>>>> deserve the first beta release. So I whipped up a website and it's released
>>>>>>> to pypi. Here's (again), a demo (the full tutorial can be found on the
>>>>>>> links below)
>>>>>>>
>>>>>>> class User(Document):
>>>>>>>     client = riak.RiakClient()
>>>>>>>     bucket_name = "users"
>>>>>>>
>>>>>>>     name = StringProperty(required=True)
>>>>>>>
>>>>>>> >>> user = User(name="foo")
>>>>>>> >>> user.save()
>>>>>>> >>> print user.name
>>>>>>> foo
>>>>>>>
>>>>>>> If anyone is interested, you can find the website here:
>>>>>>> http://ultimatebuster.github.com/riakkit and the code at
>>>>>>> http://github.com/ultimatebuster/riakkit
>>>>>>>
>>>>>>> Cheers!
>>>>>>>
>>>>>>> Shuhao
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> riak-users mailing list
>>>>>>> riak-users at lists.basho.com
>>>>>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> riak-users mailing list
>>>>>>> riak-users at lists.basho.com
>>>>>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> riak-users mailing list
>>>>>> riak-users at lists.basho.com
>>>>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.basho.com/pipermail/riak-users_lists.basho.com/attachments/20120302/2d033f9f/attachment.html>


More information about the riak-users mailing list