riakkit, a python riak object mapper, has hit beta!

Andrey V. Martyanov realduke at gmail.com
Fri Mar 2 08:17:34 EST 2012


Hi Shuhao,

I don't have enough time to write my own implementation and to see how it
works in practice. But I share my thoughts about that.
It's possible to save the current API, it's common now and used in
SQLAlchemy and Django ORM. But underlying implementation I see a bit
different. The simple workflow:
1. Define the models with specific fields and properties;
2. Generate the JSON Schema [1] based on these fields;
3. Create an instances of our models;
4. Validate against the schema;
5. Generate the optimal sequence of actions we need to do to store the
data, based on the meta information of our models;
6. Store the data, resolve the conflicts if occur.

Also it should be possible to deserialize the JSON objects into python
objects.

[1] - http://json-schema.org/

Best regards,
Andrey Martyanov

>
>
> On Fri, Mar 2, 2012 at 5:47 PM, Shuhao Wu <admin at thekks.net> wrote:
>
>> Yeah. You're looking at something like dictshield. As riakkit is more a
>> RAD tool in its current state. You bring in an excellent point,
>> nonetheless, which gave me some intuition and ideas about the next step.
>>
>> Cheers,
>> Shuhao
>> Sent from my phone.
>> On Mar 1, 2012 11:41 PM, "Andrey V. Martyanov" <realduke at gmail.com>
>> wrote:
>>
>>> Riakkit in its current implementation has a  lot of overhead. But what I
>>> get for this price? Almost nothing. Only field validation make sense.
>>> Simple key-value mapping is about 50-70 lines of code, not a big deal, even
>>> for small project. Riak used mostly in web-based projects. Almost all
>>> modern projects use JSON. The simple workflow is get JSON from the client,
>>> validate, store in the database, query the database, send JSON to the
>>> client. If our mapper doesn't understand the JSON and cannot operate on it
>>> then it's not useful. We can use jsonpickle with a lot more profit. Just
>>> jsonpickle.encode the object and store it in Riak. I'm sorry, this is my
>>> position!
>>>
>>> Best regards,
>>> Andrey Martyanov
>>>
>>> On Fri, Mar 2, 2012 at 9:01 AM, Andrey V. Martyanov <realduke at gmail.com>wrote:
>>>
>>>> I'm just saying that to efficiently store and get the data user should
>>>> have ability to customize the relationship. Riak has several options to get
>>>> related data: secondary indexes, map-reduce etc. I think that the most
>>>> important task of mapper is to validate the data and to track these
>>>> dependencies. When you have enough meta information about objects storing
>>>> these objects is not a big problem.
>>>>
>>>>
>>>> On Fri, Mar 2, 2012 at 8:39 AM, Shuhao Wu <admin at thekks.net> wrote:
>>>>
>>>>> The thing about riak-python-client is that it's already pretty high
>>>>> level, but not high level enough to be that simple.
>>>>>
>>>>> From what I understand, tracking relationship between models will
>>>>> still involve a certain level of magic. Right now the magic is that when
>>>>> you save any document, all the related document will be saved into the
>>>>> database as well. You're saying that just tracking those relationship
>>>>> change is good enough and saving will be up to the user?
>>>>>
>>>>> Shuhao
>>>>>
>>>>>
>>>>> On Thu, Mar 1, 2012 at 10:20 PM, Andrey V. Martyanov <
>>>>> realduke at gmail.com> wrote:
>>>>>
>>>>>> I'm not sure about forking the riak-python-client. The the whole idea
>>>>>> of separation the low-level client library from the high-level mapper and
>>>>>> the like is quite good. Some sort of SQLAlchemy does.
>>>>>>
>>>>>> Best regards,
>>>>>> Andrey Martyanov
>>>>>>
>>>>>>
>>>>>> On Fri, Mar 2, 2012 at 8:15 AM, Shuhao Wu <admin at thekks.net> wrote:
>>>>>>
>>>>>>> Another thing I was considering is forking riak-python-client. Some
>>>>>>> of the stuff in there could use some fixing as well, and maybe just take
>>>>>>> out the core transports and such and plug that right into riakkit and skip
>>>>>>> the notion of RiakObject all together as those API is also somewhat weird.
>>>>>>> (such as add_link and remove_link)
>>>>>>>
>>>>>>> Shuhao
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Thu, Mar 1, 2012 at 10:02 PM, Shuhao Wu <admin at thekks.net> wrote:
>>>>>>>
>>>>>>>> Hi!
>>>>>>>>
>>>>>>>> First off, the code is *LGPL* licensed, not GPL. So you're free to
>>>>>>>> put this into any code you want, it's just that when you want to modify
>>>>>>>> riakkit, you have to release your modified version.
>>>>>>>>
>>>>>>>> I just went over your idea and it would be a valid solution to some
>>>>>>>> of the problems I'm having right now with my own projects with riakkit.
>>>>>>>> When I initially started writing riakkit, I needed something to do RAD
>>>>>>>> with, which would make sense to have an all-in-one, very simple to use
>>>>>>>> framework, rather than working with serialization and what not. However,
>>>>>>>> now I'm beginning to see problems in this approach when it comes to
>>>>>>>> scalability as I'm slowing moving out of the RAD phase. I haven't done
>>>>>>>> anything as I have not encountered any major issues yet (though I probably
>>>>>>>> will).
>>>>>>>>
>>>>>>>> So here's what I'm thinking for the next branch ( I still have to
>>>>>>>> maintain the master branch due to some code i've written and need ): I'm
>>>>>>>> going to break riakkit into 2 pieces. The first piece is the underlying
>>>>>>>> piece, which will do what you said, serialize/deserialize objects,
>>>>>>>> validation, track relationships and what not, however, it doesn't load it
>>>>>>>> those references up nor does it try to accomplish everything all in one
>>>>>>>> shot, and a RAD layer that's compatible with the current API that will
>>>>>>>> allow RAD to work with it, but still reserve the flexibility of having that
>>>>>>>> underlying API that's relatively easy (as oppose to dead simple) to work
>>>>>>>> with and not as messy as what's going on right now.
>>>>>>>>
>>>>>>>> I do agree that the code is messy.. Most of the time it's me fixing
>>>>>>>> bugs I didn't anticipate as I go along and hacking everything into place..
>>>>>>>>
>>>>>>>> One thing I'm not too clear about is what you mean by "Track
>>>>>>>> dependencies between models". I'm not sure how you think I should go about
>>>>>>>> approaching this (which right now is a pretty big disaster.. though
>>>>>>>> relatively stable)
>>>>>>>>
>>>>>>>> Cheers,
>>>>>>>>
>>>>>>>> Shuhao
>>>>>>>>
>>>>>>>> P.S. I don't want to go into the whole PEP8 discussion and 4 spaces
>>>>>>>> vs whatever.. I just like the 2 spaces and the style guide that Google
>>>>>>>> uses. Is that really a big deal if you're using it as a library?
>>>>>>>>
>>>>>>>> On Thu, Mar 1, 2012 at 9:35 PM, Andrey V. Martyanov <
>>>>>>>> realduke at gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Hi!
>>>>>>>>>
>>>>>>>>> Two things force me to avoid using Riakkit: the code is GPL
>>>>>>>>> licensed and non pep8 compliant. So, I cannot fork it and make changes. The
>>>>>>>>> code needs clean up, for example, json module is imported in document.py,
>>>>>>>>> but not used. Riakkit has no conflict resolution logic, it's basically
>>>>>>>>> delegate the work to underlying python client. If I'd like to implement my
>>>>>>>>> save method I need to copy and rewrite more than 150 lines of code mess.
>>>>>>>>> What I think the mapper should do:
>>>>>>>>> 1. Serialize the objects to valid json;
>>>>>>>>> 2. Deserialize the data back to class instances;
>>>>>>>>> 3. Validate the fields data;
>>>>>>>>> 4. Track dependencies between models;
>>>>>>>>>
>>>>>>>>> Simple example using Riakkit-like syntax;
>>>>>>>>>
>>>>>>>>> class Post(Document):
>>>>>>>>>
>>>>>>>>>     bucket_name  = 'posts'
>>>>>>>>>
>>>>>>>>>     title = types.StringProperty(required=True)
>>>>>>>>>     content = types.StringProperty(required=True)
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> class Comment(Document):
>>>>>>>>>
>>>>>>>>>     bucket_name = 'comments'
>>>>>>>>>
>>>>>>>>>     text = types.StringProperty(required=True)
>>>>>>>>>     post = types.IndexReferenceProperty(reference_class=Post,
>>>>>>>>> collection_name='comments')
>>>>>>>>>
>>>>>>>>> >>> post = Post('title', 'content')
>>>>>>>>> >>> comment1 = ('comment1', post=post)
>>>>>>>>> >>> comment2 = ('comment2', post=post)
>>>>>>>>> >>> post.is_valid()
>>>>>>>>> True
>>>>>>>>> >> post.serialize()
>>>>>>>>> {
>>>>>>>>>     "key": "post_key",
>>>>>>>>>     "title": "title",
>>>>>>>>>     "content": "content",
>>>>>>>>>     "comments": [
>>>>>>>>>         {
>>>>>>>>>             "key": "comment1_key",
>>>>>>>>>             "text": "comment1"
>>>>>>>>>         },
>>>>>>>>>         {
>>>>>>>>>             "key": "comment2_key",
>>>>>>>>>             "text": "comment2"
>>>>>>>>>         },
>>>>>>>>>     ]
>>>>>>>>> }
>>>>>>>>> >>> post.store()
>>>>>>>>> True
>>>>>>>>> >>> post = Post.get_with_key('post_key')
>>>>>>>>> >>> post.comments
>>>>>>>>> ['Post comment1', 'Post comment2']
>>>>>>>>> >>> post_json_data = get_json_data_from_request()
>>>>>>>>> >>> post = Post.deserialize(post_json_data)
>>>>>>>>> >>> post.title
>>>>>>>>> 'title'
>>>>>>>>> >>> post.comments
>>>>>>>>> ['Post comment1', 'Post comment2']
>>>>>>>>>
>>>>>>>>> Something like that.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Best regards,
>>>>>>>>> Andrey Martyanov
>>>>>>>>>
>>>>>>>>> On Fri, Mar 2, 2012 at 5:39 AM, Sean Carey <carey at basho.com>wrote:
>>>>>>>>>
>>>>>>>>>> Keep up the great work Shuhao!
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Mar 1, 2012, at 7:36 PM, Shuhao Wu <admin at thekks.net> wrote:
>>>>>>>>>>
>>>>>>>>>> Hey guys,
>>>>>>>>>>
>>>>>>>>>> I started writing riakkit approximately 3 months ago and
>>>>>>>>>> announced it here. In case you missed that, Riakkit is basically an object
>>>>>>>>>> mapper for python and riak, which seemed to be lacking in at the time. Now
>>>>>>>>>> that 3 months has passed and I have developed this project into much more
>>>>>>>>>> than what I had envisioned to begin with. It's currently also being used
>>>>>>>>>> and tested in a project of mine (web application) as well and functioning
>>>>>>>>>> quite nicely. I would say that currently this is the only object mapper
>>>>>>>>>> that can compete with their counterparts in other languages for Riak in
>>>>>>>>>> Python, although a lot of work still needs to be done before it could fully
>>>>>>>>>> compete.
>>>>>>>>>>
>>>>>>>>>> So today I felt that riakkit is "stable" and "clean" enough to
>>>>>>>>>> deserve the first beta release. So I whipped up a website and it's released
>>>>>>>>>> to pypi. Here's (again), a demo (the full tutorial can be found on the
>>>>>>>>>> links below)
>>>>>>>>>>
>>>>>>>>>> class User(Document):
>>>>>>>>>>     client = riak.RiakClient()
>>>>>>>>>>     bucket_name = "users"
>>>>>>>>>>
>>>>>>>>>>     name = StringProperty(required=True)
>>>>>>>>>>
>>>>>>>>>> >>> user = User(name="foo")
>>>>>>>>>> >>> user.save()
>>>>>>>>>> >>> print user.name
>>>>>>>>>> foo
>>>>>>>>>>
>>>>>>>>>> If anyone is interested, you can find the website here:
>>>>>>>>>> http://ultimatebuster.github.com/riakkit and the code at
>>>>>>>>>> http://github.com/ultimatebuster/riakkit
>>>>>>>>>>
>>>>>>>>>> Cheers!
>>>>>>>>>>
>>>>>>>>>> Shuhao
>>>>>>>>>>
>>>>>>>>>> _______________________________________________
>>>>>>>>>> riak-users mailing list
>>>>>>>>>> riak-users at lists.basho.com
>>>>>>>>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> _______________________________________________
>>>>>>>>>> riak-users mailing list
>>>>>>>>>> riak-users at lists.basho.com
>>>>>>>>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> riak-users mailing list
>>>>>>>>> riak-users at lists.basho.com
>>>>>>>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>> _______________________________________________
>>> riak-users mailing list
>>> riak-users at lists.basho.com
>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>>
>>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.basho.com/pipermail/riak-users_lists.basho.com/attachments/20120302/3fde9c72/attachment.html>


More information about the riak-users mailing list