riakkit, a python riak object mapper, has hit beta!

Andrey V. Martyanov realduke at gmail.com
Fri Mar 2 08:35:03 EST 2012


Also, I think we can add RoR-like migrations on top of that with automatic
schema versioning and all the stuff.

Best regards,
Andrey Martyanov

On Fri, Mar 2, 2012 at 6:22 PM, Andrey V. Martyanov <realduke at gmail.com>wrote:

> I'm sorry for the repost, but my gmail client by default sends an email to
> a specific subscriber, not to riak-users.
>
>
> Hi Shuhao,
>
> I don't have enough time to write my own implementation and to see how it
> works in practice. But I share my thoughts about that.
> It's possible to save the current API, it's common now and used in
> SQLAlchemy and Django ORM. But underlying implementation I see a bit
> different. The simple workflow:
> 1. Define the models with specific fields and properties;
> 2. Generate the JSON Schema [1] based on these fields;
> 3. Create an instances of our models;
> 4. Validate against the schema;
> 5. Generate the optimal sequence of actions we need to do to store the
> data, based on the meta information of our models;
> 6. Store the data, resolve the conflicts if occur.
>
> Also it should be possible to deserialize the JSON objects into python
> objects.
>
> [1] - http://json-schema.org/
>
> Best regards,
> Andrey Martyanov
>
> On Fri, Mar 2, 2012 at 6:17 PM, Andrey V. Martyanov <realduke at gmail.com>wrote:
>
>> Hi Shuhao,
>>
>> I don't have enough time to write my own implementation and to see how it
>> works in practice. But I share my thoughts about that.
>> It's possible to save the current API, it's common now and used in
>> SQLAlchemy and Django ORM. But underlying implementation I see a bit
>> different. The simple workflow:
>> 1. Define the models with specific fields and properties;
>> 2. Generate the JSON Schema [1] based on these fields;
>> 3. Create an instances of our models;
>> 4. Validate against the schema;
>> 5. Generate the optimal sequence of actions we need to do to store the
>> data, based on the meta information of our models;
>> 6. Store the data, resolve the conflicts if occur.
>>
>> Also it should be possible to deserialize the JSON objects into python
>> objects.
>>
>> [1] - http://json-schema.org/
>>
>> Best regards,
>> Andrey Martyanov
>>
>>>
>>>
>>> On Fri, Mar 2, 2012 at 5:47 PM, Shuhao Wu <admin at thekks.net> wrote:
>>>
>>>> Yeah. You're looking at something like dictshield. As riakkit is more a
>>>> RAD tool in its current state. You bring in an excellent point,
>>>> nonetheless, which gave me some intuition and ideas about the next step.
>>>>
>>>> Cheers,
>>>> Shuhao
>>>> Sent from my phone.
>>>> On Mar 1, 2012 11:41 PM, "Andrey V. Martyanov" <realduke at gmail.com>
>>>> wrote:
>>>>
>>>>> Riakkit in its current implementation has a  lot of overhead. But what
>>>>> I get for this price? Almost nothing. Only field validation make sense.
>>>>> Simple key-value mapping is about 50-70 lines of code, not a big deal, even
>>>>> for small project. Riak used mostly in web-based projects. Almost all
>>>>> modern projects use JSON. The simple workflow is get JSON from the client,
>>>>> validate, store in the database, query the database, send JSON to the
>>>>> client. If our mapper doesn't understand the JSON and cannot operate on it
>>>>> then it's not useful. We can use jsonpickle with a lot more profit. Just
>>>>> jsonpickle.encode the object and store it in Riak. I'm sorry, this is my
>>>>> position!
>>>>>
>>>>> Best regards,
>>>>> Andrey Martyanov
>>>>>
>>>>> On Fri, Mar 2, 2012 at 9:01 AM, Andrey V. Martyanov <
>>>>> realduke at gmail.com> wrote:
>>>>>
>>>>>> I'm just saying that to efficiently store and get the data user
>>>>>> should have ability to customize the relationship. Riak has several options
>>>>>> to get related data: secondary indexes, map-reduce etc. I think that the
>>>>>> most important task of mapper is to validate the data and to track these
>>>>>> dependencies. When you have enough meta information about objects storing
>>>>>> these objects is not a big problem.
>>>>>>
>>>>>>
>>>>>> On Fri, Mar 2, 2012 at 8:39 AM, Shuhao Wu <admin at thekks.net> wrote:
>>>>>>
>>>>>>> The thing about riak-python-client is that it's already pretty high
>>>>>>> level, but not high level enough to be that simple.
>>>>>>>
>>>>>>> From what I understand, tracking relationship between models will
>>>>>>> still involve a certain level of magic. Right now the magic is that when
>>>>>>> you save any document, all the related document will be saved into the
>>>>>>> database as well. You're saying that just tracking those relationship
>>>>>>> change is good enough and saving will be up to the user?
>>>>>>>
>>>>>>> Shuhao
>>>>>>>
>>>>>>>
>>>>>>> On Thu, Mar 1, 2012 at 10:20 PM, Andrey V. Martyanov <
>>>>>>> realduke at gmail.com> wrote:
>>>>>>>
>>>>>>>> I'm not sure about forking the riak-python-client. The the whole
>>>>>>>> idea of separation the low-level client library from the high-level mapper
>>>>>>>> and the like is quite good. Some sort of SQLAlchemy does.
>>>>>>>>
>>>>>>>> Best regards,
>>>>>>>> Andrey Martyanov
>>>>>>>>
>>>>>>>>
>>>>>>>> On Fri, Mar 2, 2012 at 8:15 AM, Shuhao Wu <admin at thekks.net> wrote:
>>>>>>>>
>>>>>>>>> Another thing I was considering is forking riak-python-client.
>>>>>>>>> Some of the stuff in there could use some fixing as well, and maybe just
>>>>>>>>> take out the core transports and such and plug that right into riakkit and
>>>>>>>>> skip the notion of RiakObject all together as those API is also somewhat
>>>>>>>>> weird. (such as add_link and remove_link)
>>>>>>>>>
>>>>>>>>> Shuhao
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Thu, Mar 1, 2012 at 10:02 PM, Shuhao Wu <admin at thekks.net>wrote:
>>>>>>>>>
>>>>>>>>>> Hi!
>>>>>>>>>>
>>>>>>>>>> First off, the code is *LGPL* licensed, not GPL. So you're free
>>>>>>>>>> to put this into any code you want, it's just that when you want to modify
>>>>>>>>>> riakkit, you have to release your modified version.
>>>>>>>>>>
>>>>>>>>>> I just went over your idea and it would be a valid solution to
>>>>>>>>>> some of the problems I'm having right now with my own projects with
>>>>>>>>>> riakkit. When I initially started writing riakkit, I needed something to do
>>>>>>>>>> RAD with, which would make sense to have an all-in-one, very simple to use
>>>>>>>>>> framework, rather than working with serialization and what not. However,
>>>>>>>>>> now I'm beginning to see problems in this approach when it comes to
>>>>>>>>>> scalability as I'm slowing moving out of the RAD phase. I haven't done
>>>>>>>>>> anything as I have not encountered any major issues yet (though I probably
>>>>>>>>>> will).
>>>>>>>>>>
>>>>>>>>>> So here's what I'm thinking for the next branch ( I still have to
>>>>>>>>>> maintain the master branch due to some code i've written and need ): I'm
>>>>>>>>>> going to break riakkit into 2 pieces. The first piece is the underlying
>>>>>>>>>> piece, which will do what you said, serialize/deserialize objects,
>>>>>>>>>> validation, track relationships and what not, however, it doesn't load it
>>>>>>>>>> those references up nor does it try to accomplish everything all in one
>>>>>>>>>> shot, and a RAD layer that's compatible with the current API that will
>>>>>>>>>> allow RAD to work with it, but still reserve the flexibility of having that
>>>>>>>>>> underlying API that's relatively easy (as oppose to dead simple) to work
>>>>>>>>>> with and not as messy as what's going on right now.
>>>>>>>>>>
>>>>>>>>>> I do agree that the code is messy.. Most of the time it's me
>>>>>>>>>> fixing bugs I didn't anticipate as I go along and hacking everything into
>>>>>>>>>> place..
>>>>>>>>>>
>>>>>>>>>> One thing I'm not too clear about is what you mean by "Track
>>>>>>>>>> dependencies between models". I'm not sure how you think I should go about
>>>>>>>>>> approaching this (which right now is a pretty big disaster.. though
>>>>>>>>>> relatively stable)
>>>>>>>>>>
>>>>>>>>>> Cheers,
>>>>>>>>>>
>>>>>>>>>> Shuhao
>>>>>>>>>>
>>>>>>>>>> P.S. I don't want to go into the whole PEP8 discussion and 4
>>>>>>>>>> spaces vs whatever.. I just like the 2 spaces and the style guide that
>>>>>>>>>> Google uses. Is that really a big deal if you're using it as a library?
>>>>>>>>>>
>>>>>>>>>> On Thu, Mar 1, 2012 at 9:35 PM, Andrey V. Martyanov <
>>>>>>>>>> realduke at gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi!
>>>>>>>>>>>
>>>>>>>>>>> Two things force me to avoid using Riakkit: the code is GPL
>>>>>>>>>>> licensed and non pep8 compliant. So, I cannot fork it and make changes. The
>>>>>>>>>>> code needs clean up, for example, json module is imported in document.py,
>>>>>>>>>>> but not used. Riakkit has no conflict resolution logic, it's basically
>>>>>>>>>>> delegate the work to underlying python client. If I'd like to implement my
>>>>>>>>>>> save method I need to copy and rewrite more than 150 lines of code mess.
>>>>>>>>>>> What I think the mapper should do:
>>>>>>>>>>> 1. Serialize the objects to valid json;
>>>>>>>>>>> 2. Deserialize the data back to class instances;
>>>>>>>>>>> 3. Validate the fields data;
>>>>>>>>>>> 4. Track dependencies between models;
>>>>>>>>>>>
>>>>>>>>>>> Simple example using Riakkit-like syntax;
>>>>>>>>>>>
>>>>>>>>>>> class Post(Document):
>>>>>>>>>>>
>>>>>>>>>>>     bucket_name  = 'posts'
>>>>>>>>>>>
>>>>>>>>>>>     title = types.StringProperty(required=True)
>>>>>>>>>>>     content = types.StringProperty(required=True)
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> class Comment(Document):
>>>>>>>>>>>
>>>>>>>>>>>     bucket_name = 'comments'
>>>>>>>>>>>
>>>>>>>>>>>     text = types.StringProperty(required=True)
>>>>>>>>>>>     post = types.IndexReferenceProperty(reference_class=Post,
>>>>>>>>>>> collection_name='comments')
>>>>>>>>>>>
>>>>>>>>>>> >>> post = Post('title', 'content')
>>>>>>>>>>> >>> comment1 = ('comment1', post=post)
>>>>>>>>>>> >>> comment2 = ('comment2', post=post)
>>>>>>>>>>> >>> post.is_valid()
>>>>>>>>>>> True
>>>>>>>>>>> >> post.serialize()
>>>>>>>>>>> {
>>>>>>>>>>>     "key": "post_key",
>>>>>>>>>>>     "title": "title",
>>>>>>>>>>>     "content": "content",
>>>>>>>>>>>     "comments": [
>>>>>>>>>>>         {
>>>>>>>>>>>             "key": "comment1_key",
>>>>>>>>>>>             "text": "comment1"
>>>>>>>>>>>         },
>>>>>>>>>>>         {
>>>>>>>>>>>             "key": "comment2_key",
>>>>>>>>>>>             "text": "comment2"
>>>>>>>>>>>         },
>>>>>>>>>>>     ]
>>>>>>>>>>> }
>>>>>>>>>>> >>> post.store()
>>>>>>>>>>> True
>>>>>>>>>>> >>> post = Post.get_with_key('post_key')
>>>>>>>>>>> >>> post.comments
>>>>>>>>>>> ['Post comment1', 'Post comment2']
>>>>>>>>>>> >>> post_json_data = get_json_data_from_request()
>>>>>>>>>>> >>> post = Post.deserialize(post_json_data)
>>>>>>>>>>> >>> post.title
>>>>>>>>>>> 'title'
>>>>>>>>>>> >>> post.comments
>>>>>>>>>>> ['Post comment1', 'Post comment2']
>>>>>>>>>>>
>>>>>>>>>>> Something like that.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Best regards,
>>>>>>>>>>> Andrey Martyanov
>>>>>>>>>>>
>>>>>>>>>>> On Fri, Mar 2, 2012 at 5:39 AM, Sean Carey <carey at basho.com>wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Keep up the great work Shuhao!
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Mar 1, 2012, at 7:36 PM, Shuhao Wu <admin at thekks.net> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> Hey guys,
>>>>>>>>>>>>
>>>>>>>>>>>> I started writing riakkit approximately 3 months ago and
>>>>>>>>>>>> announced it here. In case you missed that, Riakkit is basically an object
>>>>>>>>>>>> mapper for python and riak, which seemed to be lacking in at the time. Now
>>>>>>>>>>>> that 3 months has passed and I have developed this project into much more
>>>>>>>>>>>> than what I had envisioned to begin with. It's currently also being used
>>>>>>>>>>>> and tested in a project of mine (web application) as well and functioning
>>>>>>>>>>>> quite nicely. I would say that currently this is the only object mapper
>>>>>>>>>>>> that can compete with their counterparts in other languages for Riak in
>>>>>>>>>>>> Python, although a lot of work still needs to be done before it could fully
>>>>>>>>>>>> compete.
>>>>>>>>>>>>
>>>>>>>>>>>> So today I felt that riakkit is "stable" and "clean" enough to
>>>>>>>>>>>> deserve the first beta release. So I whipped up a website and it's released
>>>>>>>>>>>> to pypi. Here's (again), a demo (the full tutorial can be found on the
>>>>>>>>>>>> links below)
>>>>>>>>>>>>
>>>>>>>>>>>> class User(Document):
>>>>>>>>>>>>     client = riak.RiakClient()
>>>>>>>>>>>>     bucket_name = "users"
>>>>>>>>>>>>
>>>>>>>>>>>>     name = StringProperty(required=True)
>>>>>>>>>>>>
>>>>>>>>>>>> >>> user = User(name="foo")
>>>>>>>>>>>> >>> user.save()
>>>>>>>>>>>> >>> print user.name
>>>>>>>>>>>> foo
>>>>>>>>>>>>
>>>>>>>>>>>> If anyone is interested, you can find the website here:
>>>>>>>>>>>> http://ultimatebuster.github.com/riakkit and the code at
>>>>>>>>>>>> http://github.com/ultimatebuster/riakkit
>>>>>>>>>>>>
>>>>>>>>>>>> Cheers!
>>>>>>>>>>>>
>>>>>>>>>>>> Shuhao
>>>>>>>>>>>>
>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>> riak-users mailing list
>>>>>>>>>>>> riak-users at lists.basho.com
>>>>>>>>>>>>
>>>>>>>>>>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>> riak-users mailing list
>>>>>>>>>>>> riak-users at lists.basho.com
>>>>>>>>>>>>
>>>>>>>>>>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> _______________________________________________
>>>>>>>>>>> riak-users mailing list
>>>>>>>>>>> riak-users at lists.basho.com
>>>>>>>>>>>
>>>>>>>>>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> riak-users mailing list
>>>>> riak-users at lists.basho.com
>>>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>>>>
>>>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.basho.com/pipermail/riak-users_lists.basho.com/attachments/20120302/cd53622c/attachment.html>


More information about the riak-users mailing list