riakkit, a python riak object mapper, has hit beta!

Michael Clemmons glassresistor at gmail.com
Fri Mar 2 19:08:30 EST 2012


Andrey / Shuhao,
I had been working on a similar project to riakkit.  Its not near as
complete but had some nice feature like lazy loading and basically the same
bucket/client defined class structure.
I have similar concerns as Andrey about riakkit's current beta pep8
compliance being a big one, but that could be fixed without much fuss just
time.
I think having a pythonista friendly client and object mapper is of key
importance and something I'd like to see handled in a standards oriented
way.  If your interested in having a discussion about designs for riakkit
let me know, probably better off the list though.

Shuhao,
"""
Another thing I was considering is forking riak-python-client. Some of the
stuff in there could use some fixing as well, and maybe just take out the
core transports and such and plug that right into riakkit and skip the
notion of RiakObject all together as those API is also somewhat weird.
(such as add_link and remove_link)
"""
I tried something similar in my obj mapper, I baically subclasses the
RiakBucket and bypased the RiakObject for lazy loading and better control.
This was nice at friend but became tedious and seems like itll have issues
in the future since i was having to access internal methods.  If I were to
do it again I think I will probably use the basho api and abstract it in
the mapper.

On Fri, Mar 2, 2012 at 8:26 AM, Andrey V. Martyanov <realduke at gmail.com>wrote:

> Basically, I meant sibling reconciliation.
>
> Best regards,
> Andrey Martyanov
>
>
> On Fri, Mar 2, 2012 at 8:01 PM, Shuhao Wu <admin at thekks.net> wrote:
>
>> Also, what do you mean by conflict resolution? Synchronization or
>> conflicts between references such as deleting references to a document when
>> that document is deleted?
>>
>> Shuhao
>>
>>
>>
>> On Fri, Mar 2, 2012 at 9:56 AM, Shuhao Wu <admin at thekks.net> wrote:
>>
>>> Actually it's okay for no pull request for now.. I'm in the process of
>>> rethinking the whole framework as well as I'm having some of the issues
>>> that Andrey is describing as well.
>>>
>>> Shuhao
>>>
>>>
>>>
>>> On Fri, Mar 2, 2012 at 9:51 AM, Jared Morrow <jared at basho.com> wrote:
>>>
>>>> Andrey,
>>>>
>>>> First off, you are indeed sending two emails to the Riak Users list
>>>> each time you email, not just one, so I think gmail is doing the right
>>>> thing the first time.
>>>>
>>>> Second, I'm sure Shuhao would love some pull requests to help with
>>>> Riakkit since you seem to have a lot of ideas to make it better.
>>>>
>>>> -Jared
>>>>
>>>>
>>>> On Mar 2, 2012, at 6:39 AM, Andrey V. Martyanov wrote:
>>>>
>>>> Also, I think we can add RoR-like migrations on top of that with
>>>> automatic schema versioning and all the stuff.
>>>>
>>>> Best regards,
>>>> Andrey Martyanov
>>>>
>>>> On Fri, Mar 2, 2012 at 6:22 PM, Andrey V. Martyanov <realduke at gmail.com
>>>> > wrote:
>>>>
>>>>> I'm sorry for the repost, but my gmail client by default sends an
>>>>> email to a specific subscriber, not to riak-users.
>>>>>
>>>>>
>>>>> Hi Shuhao,
>>>>>
>>>>> I don't have enough time to write my own implementation and to see how
>>>>> it works in practice. But I share my thoughts about that.
>>>>> It's possible to save the current API, it's common now and used in
>>>>> SQLAlchemy and Django ORM. But underlying implementation I see a bit
>>>>> different. The simple workflow:
>>>>> 1. Define the models with specific fields and properties;
>>>>> 2. Generate the JSON Schema [1] based on these fields;
>>>>> 3. Create an instances of our models;
>>>>> 4. Validate against the schema;
>>>>> 5. Generate the optimal sequence of actions we need to do to store the
>>>>> data, based on the meta information of our models;
>>>>> 6. Store the data, resolve the conflicts if occur.
>>>>>
>>>>> Also it should be possible to deserialize the JSON objects into python
>>>>> objects.
>>>>>
>>>>> [1] - http://json-schema.org/
>>>>>
>>>>> Best regards,
>>>>> Andrey Martyanov
>>>>>
>>>>> On Fri, Mar 2, 2012 at 6:17 PM, Andrey V. Martyanov <
>>>>> realduke at gmail.com> wrote:
>>>>>
>>>>>> Hi Shuhao,
>>>>>>
>>>>>> I don't have enough time to write my own implementation and to see
>>>>>> how it works in practice. But I share my thoughts about that.
>>>>>> It's possible to save the current API, it's common now and used in
>>>>>> SQLAlchemy and Django ORM. But underlying implementation I see a bit
>>>>>> different. The simple workflow:
>>>>>> 1. Define the models with specific fields and properties;
>>>>>> 2. Generate the JSON Schema [1] based on these fields;
>>>>>> 3. Create an instances of our models;
>>>>>> 4. Validate against the schema;
>>>>>> 5. Generate the optimal sequence of actions we need to do to store
>>>>>> the data, based on the meta information of our models;
>>>>>> 6. Store the data, resolve the conflicts if occur.
>>>>>>
>>>>>> Also it should be possible to deserialize the JSON objects into
>>>>>> python objects.
>>>>>>
>>>>>> [1] - http://json-schema.org/
>>>>>>
>>>>>> Best regards,
>>>>>> Andrey Martyanov
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Fri, Mar 2, 2012 at 5:47 PM, Shuhao Wu <admin at thekks.net> wrote:
>>>>>>>
>>>>>>>> Yeah. You're looking at something like dictshield. As riakkit is
>>>>>>>> more a RAD tool in its current state. You bring in an excellent point,
>>>>>>>> nonetheless, which gave me some intuition and ideas about the next step.
>>>>>>>>
>>>>>>>> Cheers,
>>>>>>>> Shuhao
>>>>>>>> Sent from my phone.
>>>>>>>> On Mar 1, 2012 11:41 PM, "Andrey V. Martyanov" <realduke at gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Riakkit in its current implementation has a  lot of overhead. But
>>>>>>>>> what I get for this price? Almost nothing. Only field validation make
>>>>>>>>> sense. Simple key-value mapping is about 50-70 lines of code, not a big
>>>>>>>>> deal, even for small project. Riak used mostly in web-based projects.
>>>>>>>>> Almost all modern projects use JSON. The simple workflow is get JSON from
>>>>>>>>> the client, validate, store in the database, query the database, send JSON
>>>>>>>>> to the client. If our mapper doesn't understand the JSON and cannot operate
>>>>>>>>> on it then it's not useful. We can use jsonpickle with a lot more profit.
>>>>>>>>> Just jsonpickle.encode the object and store it in Riak. I'm sorry, this is
>>>>>>>>> my position!
>>>>>>>>>
>>>>>>>>> Best regards,
>>>>>>>>> Andrey Martyanov
>>>>>>>>>
>>>>>>>>> On Fri, Mar 2, 2012 at 9:01 AM, Andrey V. Martyanov <
>>>>>>>>> realduke at gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> I'm just saying that to efficiently store and get the data user
>>>>>>>>>> should have ability to customize the relationship. Riak has several options
>>>>>>>>>> to get related data: secondary indexes, map-reduce etc. I think that the
>>>>>>>>>> most important task of mapper is to validate the data and to track these
>>>>>>>>>> dependencies. When you have enough meta information about objects storing
>>>>>>>>>> these objects is not a big problem.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Fri, Mar 2, 2012 at 8:39 AM, Shuhao Wu <admin at thekks.net>wrote:
>>>>>>>>>>
>>>>>>>>>>> The thing about riak-python-client is that it's already pretty
>>>>>>>>>>> high level, but not high level enough to be that simple.
>>>>>>>>>>>
>>>>>>>>>>> From what I understand, tracking relationship between models
>>>>>>>>>>> will still involve a certain level of magic. Right now the magic is that
>>>>>>>>>>> when you save any document, all the related document will be saved into the
>>>>>>>>>>> database as well. You're saying that just tracking those relationship
>>>>>>>>>>> change is good enough and saving will be up to the user?
>>>>>>>>>>>
>>>>>>>>>>> Shuhao
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Thu, Mar 1, 2012 at 10:20 PM, Andrey V. Martyanov <
>>>>>>>>>>> realduke at gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> I'm not sure about forking the riak-python-client. The the
>>>>>>>>>>>> whole idea of separation the low-level client library from the high-level
>>>>>>>>>>>> mapper and the like is quite good. Some sort of SQLAlchemy does.
>>>>>>>>>>>>
>>>>>>>>>>>> Best regards,
>>>>>>>>>>>> Andrey Martyanov
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Fri, Mar 2, 2012 at 8:15 AM, Shuhao Wu <admin at thekks.net>wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Another thing I was considering is forking riak-python-client.
>>>>>>>>>>>>> Some of the stuff in there could use some fixing as well, and maybe just
>>>>>>>>>>>>> take out the core transports and such and plug that right into riakkit and
>>>>>>>>>>>>> skip the notion of RiakObject all together as those API is also somewhat
>>>>>>>>>>>>> weird. (such as add_link and remove_link)
>>>>>>>>>>>>>
>>>>>>>>>>>>> Shuhao
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Thu, Mar 1, 2012 at 10:02 PM, Shuhao Wu <admin at thekks.net>wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi!
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> First off, the code is *LGPL* licensed, not GPL. So you're
>>>>>>>>>>>>>> free to put this into any code you want, it's just that when you want to
>>>>>>>>>>>>>> modify riakkit, you have to release your modified version.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I just went over your idea and it would be a valid solution
>>>>>>>>>>>>>> to some of the problems I'm having right now with my own projects with
>>>>>>>>>>>>>> riakkit. When I initially started writing riakkit, I needed something to do
>>>>>>>>>>>>>> RAD with, which would make sense to have an all-in-one, very simple to use
>>>>>>>>>>>>>> framework, rather than working with serialization and what not. However,
>>>>>>>>>>>>>> now I'm beginning to see problems in this approach when it comes to
>>>>>>>>>>>>>> scalability as I'm slowing moving out of the RAD phase. I haven't done
>>>>>>>>>>>>>> anything as I have not encountered any major issues yet (though I probably
>>>>>>>>>>>>>> will).
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> So here's what I'm thinking for the next branch ( I still
>>>>>>>>>>>>>> have to maintain the master branch due to some code i've written and need
>>>>>>>>>>>>>> ): I'm going to break riakkit into 2 pieces. The first piece is the
>>>>>>>>>>>>>> underlying piece, which will do what you said, serialize/deserialize
>>>>>>>>>>>>>> objects, validation, track relationships and what not, however, it doesn't
>>>>>>>>>>>>>> load it those references up nor does it try to accomplish everything all in
>>>>>>>>>>>>>> one shot, and a RAD layer that's compatible with the current API that will
>>>>>>>>>>>>>> allow RAD to work with it, but still reserve the flexibility of having that
>>>>>>>>>>>>>> underlying API that's relatively easy (as oppose to dead simple) to work
>>>>>>>>>>>>>> with and not as messy as what's going on right now.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I do agree that the code is messy.. Most of the time it's me
>>>>>>>>>>>>>> fixing bugs I didn't anticipate as I go along and hacking everything into
>>>>>>>>>>>>>> place..
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> One thing I'm not too clear about is what you mean by "Track
>>>>>>>>>>>>>> dependencies between models". I'm not sure how you think I should go about
>>>>>>>>>>>>>> approaching this (which right now is a pretty big disaster.. though
>>>>>>>>>>>>>> relatively stable)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Shuhao
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> P.S. I don't want to go into the whole PEP8 discussion and 4
>>>>>>>>>>>>>> spaces vs whatever.. I just like the 2 spaces and the style guide that
>>>>>>>>>>>>>> Google uses. Is that really a big deal if you're using it as a library?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Thu, Mar 1, 2012 at 9:35 PM, Andrey V. Martyanov <
>>>>>>>>>>>>>> realduke at gmail.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi!
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Two things force me to avoid using Riakkit: the code is GPL
>>>>>>>>>>>>>>> licensed and non pep8 compliant. So, I cannot fork it and make changes. The
>>>>>>>>>>>>>>> code needs clean up, for example, json module is imported in document.py,
>>>>>>>>>>>>>>> but not used. Riakkit has no conflict resolution logic, it's basically
>>>>>>>>>>>>>>> delegate the work to underlying python client. If I'd like to implement my
>>>>>>>>>>>>>>> save method I need to copy and rewrite more than 150 lines of code mess.
>>>>>>>>>>>>>>> What I think the mapper should do:
>>>>>>>>>>>>>>> 1. Serialize the objects to valid json;
>>>>>>>>>>>>>>> 2. Deserialize the data back to class instances;
>>>>>>>>>>>>>>> 3. Validate the fields data;
>>>>>>>>>>>>>>> 4. Track dependencies between models;
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Simple example using Riakkit-like syntax;
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> class Post(Document):
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>     bucket_name  = 'posts'
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>     title = types.StringProperty(required=True)
>>>>>>>>>>>>>>>     content = types.StringProperty(required=True)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> class Comment(Document):
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>     bucket_name = 'comments'
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>     text = types.StringProperty(required=True)
>>>>>>>>>>>>>>>     post =
>>>>>>>>>>>>>>> types.IndexReferenceProperty(reference_class=Post,
>>>>>>>>>>>>>>> collection_name='comments')
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> >>> post = Post('title', 'content')
>>>>>>>>>>>>>>> >>> comment1 = ('comment1', post=post)
>>>>>>>>>>>>>>> >>> comment2 = ('comment2', post=post)
>>>>>>>>>>>>>>> >>> post.is_valid()
>>>>>>>>>>>>>>> True
>>>>>>>>>>>>>>> >> post.serialize()
>>>>>>>>>>>>>>> {
>>>>>>>>>>>>>>>     "key": "post_key",
>>>>>>>>>>>>>>>     "title": "title",
>>>>>>>>>>>>>>>     "content": "content",
>>>>>>>>>>>>>>>     "comments": [
>>>>>>>>>>>>>>>         {
>>>>>>>>>>>>>>>             "key": "comment1_key",
>>>>>>>>>>>>>>>             "text": "comment1"
>>>>>>>>>>>>>>>         },
>>>>>>>>>>>>>>>         {
>>>>>>>>>>>>>>>             "key": "comment2_key",
>>>>>>>>>>>>>>>             "text": "comment2"
>>>>>>>>>>>>>>>         },
>>>>>>>>>>>>>>>     ]
>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>> >>> post.store()
>>>>>>>>>>>>>>> True
>>>>>>>>>>>>>>> >>> post = Post.get_with_key('post_key')
>>>>>>>>>>>>>>> >>> post.comments
>>>>>>>>>>>>>>> ['Post comment1', 'Post comment2']
>>>>>>>>>>>>>>> >>> post_json_data = get_json_data_from_request()
>>>>>>>>>>>>>>> >>> post = Post.deserialize(post_json_data)
>>>>>>>>>>>>>>> >>> post.title
>>>>>>>>>>>>>>> 'title'
>>>>>>>>>>>>>>> >>> post.comments
>>>>>>>>>>>>>>> ['Post comment1', 'Post comment2']
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Something like that.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Best regards,
>>>>>>>>>>>>>>> Andrey Martyanov
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Fri, Mar 2, 2012 at 5:39 AM, Sean Carey <carey at basho.com>wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Keep up the great work Shuhao!
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Mar 1, 2012, at 7:36 PM, Shuhao Wu <admin at thekks.net>
>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hey guys,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I started writing riakkit approximately 3 months ago and
>>>>>>>>>>>>>>>> announced it here. In case you missed that, Riakkit is basically an object
>>>>>>>>>>>>>>>> mapper for python and riak, which seemed to be lacking in at the time. Now
>>>>>>>>>>>>>>>> that 3 months has passed and I have developed this project into much more
>>>>>>>>>>>>>>>> than what I had envisioned to begin with. It's currently also being used
>>>>>>>>>>>>>>>> and tested in a project of mine (web application) as well and functioning
>>>>>>>>>>>>>>>> quite nicely. I would say that currently this is the only object mapper
>>>>>>>>>>>>>>>> that can compete with their counterparts in other languages for Riak in
>>>>>>>>>>>>>>>> Python, although a lot of work still needs to be done before it could fully
>>>>>>>>>>>>>>>> compete.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> So today I felt that riakkit is "stable" and "clean" enough
>>>>>>>>>>>>>>>> to deserve the first beta release. So I whipped up a website and it's
>>>>>>>>>>>>>>>> released to pypi. Here's (again), a demo (the full tutorial can be found on
>>>>>>>>>>>>>>>> the links below)
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> class User(Document):
>>>>>>>>>>>>>>>>     client = riak.RiakClient()
>>>>>>>>>>>>>>>>     bucket_name = "users"
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>     name = StringProperty(required=True)
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> >>> user = User(name="foo")
>>>>>>>>>>>>>>>> >>> user.save()
>>>>>>>>>>>>>>>> >>> print user.name
>>>>>>>>>>>>>>>> foo
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> If anyone is interested, you can find the website here:
>>>>>>>>>>>>>>>> http://ultimatebuster.github.com/riakkit and the code at
>>>>>>>>>>>>>>>> http://github.com/ultimatebuster/riakkit
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Cheers!
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Shuhao
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>>>>> riak-users mailing list
>>>>>>>>>>>>>>>> riak-users at lists.basho.com
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>>>>> riak-users mailing list
>>>>>>>>>>>>>>>> riak-users at lists.basho.com
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>>>> riak-users mailing list
>>>>>>>>>>>>>>> riak-users at lists.basho.com
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> riak-users mailing list
>>>>>>>>> riak-users at lists.basho.com
>>>>>>>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>> _______________________________________________
>>>> riak-users mailing list
>>>> riak-users at lists.basho.com
>>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> riak-users mailing list
>>>> riak-users at lists.basho.com
>>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>>>
>>>>
>>>
>>
>
> _______________________________________________
> riak-users mailing list
> riak-users at lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
>


-- 
-Michael
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.basho.com/pipermail/riak-users_lists.basho.com/attachments/20120302/c7b7388f/attachment.html>


More information about the riak-users mailing list