riakkit, a python riak object mapper, has hit beta!

Andrey V. Martyanov realduke at gmail.com
Fri Mar 2 08:22:39 EST 2012


I'm sorry for the repost, but my gmail client by default sends an email to
a specific subscriber, not to riak-users.

Hi Shuhao,

I don't have enough time to write my own implementation and to see how it
works in practice. But I share my thoughts about that.
It's possible to save the current API, it's common now and used in
SQLAlchemy and Django ORM. But underlying implementation I see a bit
different. The simple workflow:
1. Define the models with specific fields and properties;
2. Generate the JSON Schema [1] based on these fields;
3. Create an instances of our models;
4. Validate against the schema;
5. Generate the optimal sequence of actions we need to do to store the
data, based on the meta information of our models;
6. Store the data, resolve the conflicts if occur.

Also it should be possible to deserialize the JSON objects into python
objects.

[1] - http://json-schema.org/

Best regards,
Andrey Martyanov

On Fri, Mar 2, 2012 at 6:17 PM, Andrey V. Martyanov <realduke at gmail.com>wrote:

> Hi Shuhao,
>
> I don't have enough time to write my own implementation and to see how it
> works in practice. But I share my thoughts about that.
> It's possible to save the current API, it's common now and used in
> SQLAlchemy and Django ORM. But underlying implementation I see a bit
> different. The simple workflow:
> 1. Define the models with specific fields and properties;
> 2. Generate the JSON Schema [1] based on these fields;
> 3. Create an instances of our models;
> 4. Validate against the schema;
> 5. Generate the optimal sequence of actions we need to do to store the
> data, based on the meta information of our models;
> 6. Store the data, resolve the conflicts if occur.
>
> Also it should be possible to deserialize the JSON objects into python
> objects.
>
> [1] - http://json-schema.org/
>
> Best regards,
> Andrey Martyanov
>
>>
>>
>> On Fri, Mar 2, 2012 at 5:47 PM, Shuhao Wu <admin at thekks.net> wrote:
>>
>>> Yeah. You're looking at something like dictshield. As riakkit is more a
>>> RAD tool in its current state. You bring in an excellent point,
>>> nonetheless, which gave me some intuition and ideas about the next step.
>>>
>>> Cheers,
>>> Shuhao
>>> Sent from my phone.
>>> On Mar 1, 2012 11:41 PM, "Andrey V. Martyanov" <realduke at gmail.com>
>>> wrote:
>>>
>>>> Riakkit in its current implementation has a  lot of overhead. But what
>>>> I get for this price? Almost nothing. Only field validation make sense.
>>>> Simple key-value mapping is about 50-70 lines of code, not a big deal, even
>>>> for small project. Riak used mostly in web-based projects. Almost all
>>>> modern projects use JSON. The simple workflow is get JSON from the client,
>>>> validate, store in the database, query the database, send JSON to the
>>>> client. If our mapper doesn't understand the JSON and cannot operate on it
>>>> then it's not useful. We can use jsonpickle with a lot more profit. Just
>>>> jsonpickle.encode the object and store it in Riak. I'm sorry, this is my
>>>> position!
>>>>
>>>> Best regards,
>>>> Andrey Martyanov
>>>>
>>>> On Fri, Mar 2, 2012 at 9:01 AM, Andrey V. Martyanov <realduke at gmail.com
>>>> > wrote:
>>>>
>>>>> I'm just saying that to efficiently store and get the data user should
>>>>> have ability to customize the relationship. Riak has several options to get
>>>>> related data: secondary indexes, map-reduce etc. I think that the most
>>>>> important task of mapper is to validate the data and to track these
>>>>> dependencies. When you have enough meta information about objects storing
>>>>> these objects is not a big problem.
>>>>>
>>>>>
>>>>> On Fri, Mar 2, 2012 at 8:39 AM, Shuhao Wu <admin at thekks.net> wrote:
>>>>>
>>>>>> The thing about riak-python-client is that it's already pretty high
>>>>>> level, but not high level enough to be that simple.
>>>>>>
>>>>>> From what I understand, tracking relationship between models will
>>>>>> still involve a certain level of magic. Right now the magic is that when
>>>>>> you save any document, all the related document will be saved into the
>>>>>> database as well. You're saying that just tracking those relationship
>>>>>> change is good enough and saving will be up to the user?
>>>>>>
>>>>>> Shuhao
>>>>>>
>>>>>>
>>>>>> On Thu, Mar 1, 2012 at 10:20 PM, Andrey V. Martyanov <
>>>>>> realduke at gmail.com> wrote:
>>>>>>
>>>>>>> I'm not sure about forking the riak-python-client. The the whole
>>>>>>> idea of separation the low-level client library from the high-level mapper
>>>>>>> and the like is quite good. Some sort of SQLAlchemy does.
>>>>>>>
>>>>>>> Best regards,
>>>>>>> Andrey Martyanov
>>>>>>>
>>>>>>>
>>>>>>> On Fri, Mar 2, 2012 at 8:15 AM, Shuhao Wu <admin at thekks.net> wrote:
>>>>>>>
>>>>>>>> Another thing I was considering is forking riak-python-client. Some
>>>>>>>> of the stuff in there could use some fixing as well, and maybe just take
>>>>>>>> out the core transports and such and plug that right into riakkit and skip
>>>>>>>> the notion of RiakObject all together as those API is also somewhat weird.
>>>>>>>> (such as add_link and remove_link)
>>>>>>>>
>>>>>>>> Shuhao
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Thu, Mar 1, 2012 at 10:02 PM, Shuhao Wu <admin at thekks.net>wrote:
>>>>>>>>
>>>>>>>>> Hi!
>>>>>>>>>
>>>>>>>>> First off, the code is *LGPL* licensed, not GPL. So you're free
>>>>>>>>> to put this into any code you want, it's just that when you want to modify
>>>>>>>>> riakkit, you have to release your modified version.
>>>>>>>>>
>>>>>>>>> I just went over your idea and it would be a valid solution to
>>>>>>>>> some of the problems I'm having right now with my own projects with
>>>>>>>>> riakkit. When I initially started writing riakkit, I needed something to do
>>>>>>>>> RAD with, which would make sense to have an all-in-one, very simple to use
>>>>>>>>> framework, rather than working with serialization and what not. However,
>>>>>>>>> now I'm beginning to see problems in this approach when it comes to
>>>>>>>>> scalability as I'm slowing moving out of the RAD phase. I haven't done
>>>>>>>>> anything as I have not encountered any major issues yet (though I probably
>>>>>>>>> will).
>>>>>>>>>
>>>>>>>>> So here's what I'm thinking for the next branch ( I still have to
>>>>>>>>> maintain the master branch due to some code i've written and need ): I'm
>>>>>>>>> going to break riakkit into 2 pieces. The first piece is the underlying
>>>>>>>>> piece, which will do what you said, serialize/deserialize objects,
>>>>>>>>> validation, track relationships and what not, however, it doesn't load it
>>>>>>>>> those references up nor does it try to accomplish everything all in one
>>>>>>>>> shot, and a RAD layer that's compatible with the current API that will
>>>>>>>>> allow RAD to work with it, but still reserve the flexibility of having that
>>>>>>>>> underlying API that's relatively easy (as oppose to dead simple) to work
>>>>>>>>> with and not as messy as what's going on right now.
>>>>>>>>>
>>>>>>>>> I do agree that the code is messy.. Most of the time it's me
>>>>>>>>> fixing bugs I didn't anticipate as I go along and hacking everything into
>>>>>>>>> place..
>>>>>>>>>
>>>>>>>>> One thing I'm not too clear about is what you mean by "Track
>>>>>>>>> dependencies between models". I'm not sure how you think I should go about
>>>>>>>>> approaching this (which right now is a pretty big disaster.. though
>>>>>>>>> relatively stable)
>>>>>>>>>
>>>>>>>>> Cheers,
>>>>>>>>>
>>>>>>>>> Shuhao
>>>>>>>>>
>>>>>>>>> P.S. I don't want to go into the whole PEP8 discussion and 4
>>>>>>>>> spaces vs whatever.. I just like the 2 spaces and the style guide that
>>>>>>>>> Google uses. Is that really a big deal if you're using it as a library?
>>>>>>>>>
>>>>>>>>> On Thu, Mar 1, 2012 at 9:35 PM, Andrey V. Martyanov <
>>>>>>>>> realduke at gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> Hi!
>>>>>>>>>>
>>>>>>>>>> Two things force me to avoid using Riakkit: the code is GPL
>>>>>>>>>> licensed and non pep8 compliant. So, I cannot fork it and make changes. The
>>>>>>>>>> code needs clean up, for example, json module is imported in document.py,
>>>>>>>>>> but not used. Riakkit has no conflict resolution logic, it's basically
>>>>>>>>>> delegate the work to underlying python client. If I'd like to implement my
>>>>>>>>>> save method I need to copy and rewrite more than 150 lines of code mess.
>>>>>>>>>> What I think the mapper should do:
>>>>>>>>>> 1. Serialize the objects to valid json;
>>>>>>>>>> 2. Deserialize the data back to class instances;
>>>>>>>>>> 3. Validate the fields data;
>>>>>>>>>> 4. Track dependencies between models;
>>>>>>>>>>
>>>>>>>>>> Simple example using Riakkit-like syntax;
>>>>>>>>>>
>>>>>>>>>> class Post(Document):
>>>>>>>>>>
>>>>>>>>>>     bucket_name  = 'posts'
>>>>>>>>>>
>>>>>>>>>>     title = types.StringProperty(required=True)
>>>>>>>>>>     content = types.StringProperty(required=True)
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> class Comment(Document):
>>>>>>>>>>
>>>>>>>>>>     bucket_name = 'comments'
>>>>>>>>>>
>>>>>>>>>>     text = types.StringProperty(required=True)
>>>>>>>>>>     post = types.IndexReferenceProperty(reference_class=Post,
>>>>>>>>>> collection_name='comments')
>>>>>>>>>>
>>>>>>>>>> >>> post = Post('title', 'content')
>>>>>>>>>> >>> comment1 = ('comment1', post=post)
>>>>>>>>>> >>> comment2 = ('comment2', post=post)
>>>>>>>>>> >>> post.is_valid()
>>>>>>>>>> True
>>>>>>>>>> >> post.serialize()
>>>>>>>>>> {
>>>>>>>>>>     "key": "post_key",
>>>>>>>>>>     "title": "title",
>>>>>>>>>>     "content": "content",
>>>>>>>>>>     "comments": [
>>>>>>>>>>         {
>>>>>>>>>>             "key": "comment1_key",
>>>>>>>>>>             "text": "comment1"
>>>>>>>>>>         },
>>>>>>>>>>         {
>>>>>>>>>>             "key": "comment2_key",
>>>>>>>>>>             "text": "comment2"
>>>>>>>>>>         },
>>>>>>>>>>     ]
>>>>>>>>>> }
>>>>>>>>>> >>> post.store()
>>>>>>>>>> True
>>>>>>>>>> >>> post = Post.get_with_key('post_key')
>>>>>>>>>> >>> post.comments
>>>>>>>>>> ['Post comment1', 'Post comment2']
>>>>>>>>>> >>> post_json_data = get_json_data_from_request()
>>>>>>>>>> >>> post = Post.deserialize(post_json_data)
>>>>>>>>>> >>> post.title
>>>>>>>>>> 'title'
>>>>>>>>>> >>> post.comments
>>>>>>>>>> ['Post comment1', 'Post comment2']
>>>>>>>>>>
>>>>>>>>>> Something like that.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Best regards,
>>>>>>>>>> Andrey Martyanov
>>>>>>>>>>
>>>>>>>>>> On Fri, Mar 2, 2012 at 5:39 AM, Sean Carey <carey at basho.com>wrote:
>>>>>>>>>>
>>>>>>>>>>> Keep up the great work Shuhao!
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Mar 1, 2012, at 7:36 PM, Shuhao Wu <admin at thekks.net> wrote:
>>>>>>>>>>>
>>>>>>>>>>> Hey guys,
>>>>>>>>>>>
>>>>>>>>>>> I started writing riakkit approximately 3 months ago and
>>>>>>>>>>> announced it here. In case you missed that, Riakkit is basically an object
>>>>>>>>>>> mapper for python and riak, which seemed to be lacking in at the time. Now
>>>>>>>>>>> that 3 months has passed and I have developed this project into much more
>>>>>>>>>>> than what I had envisioned to begin with. It's currently also being used
>>>>>>>>>>> and tested in a project of mine (web application) as well and functioning
>>>>>>>>>>> quite nicely. I would say that currently this is the only object mapper
>>>>>>>>>>> that can compete with their counterparts in other languages for Riak in
>>>>>>>>>>> Python, although a lot of work still needs to be done before it could fully
>>>>>>>>>>> compete.
>>>>>>>>>>>
>>>>>>>>>>> So today I felt that riakkit is "stable" and "clean" enough to
>>>>>>>>>>> deserve the first beta release. So I whipped up a website and it's released
>>>>>>>>>>> to pypi. Here's (again), a demo (the full tutorial can be found on the
>>>>>>>>>>> links below)
>>>>>>>>>>>
>>>>>>>>>>> class User(Document):
>>>>>>>>>>>     client = riak.RiakClient()
>>>>>>>>>>>     bucket_name = "users"
>>>>>>>>>>>
>>>>>>>>>>>     name = StringProperty(required=True)
>>>>>>>>>>>
>>>>>>>>>>> >>> user = User(name="foo")
>>>>>>>>>>> >>> user.save()
>>>>>>>>>>> >>> print user.name
>>>>>>>>>>> foo
>>>>>>>>>>>
>>>>>>>>>>> If anyone is interested, you can find the website here:
>>>>>>>>>>> http://ultimatebuster.github.com/riakkit and the code at
>>>>>>>>>>> http://github.com/ultimatebuster/riakkit
>>>>>>>>>>>
>>>>>>>>>>> Cheers!
>>>>>>>>>>>
>>>>>>>>>>> Shuhao
>>>>>>>>>>>
>>>>>>>>>>> _______________________________________________
>>>>>>>>>>> riak-users mailing list
>>>>>>>>>>> riak-users at lists.basho.com
>>>>>>>>>>>
>>>>>>>>>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> _______________________________________________
>>>>>>>>>>> riak-users mailing list
>>>>>>>>>>> riak-users at lists.basho.com
>>>>>>>>>>>
>>>>>>>>>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> _______________________________________________
>>>>>>>>>> riak-users mailing list
>>>>>>>>>> riak-users at lists.basho.com
>>>>>>>>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>> _______________________________________________
>>>> riak-users mailing list
>>>> riak-users at lists.basho.com
>>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>>>
>>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.basho.com/pipermail/riak-users_lists.basho.com/attachments/20120302/a6928086/attachment.html>


More information about the riak-users mailing list