riakkit, a python riak object mapper, has hit beta!

Shuhao Wu admin at thekks.net
Fri Mar 2 10:01:21 EST 2012


Also, what do you mean by conflict resolution? Synchronization or conflicts
between references such as deleting references to a document when that
document is deleted?

Shuhao


On Fri, Mar 2, 2012 at 9:56 AM, Shuhao Wu <admin at thekks.net> wrote:

> Actually it's okay for no pull request for now.. I'm in the process of
> rethinking the whole framework as well as I'm having some of the issues
> that Andrey is describing as well.
>
> Shuhao
>
>
>
> On Fri, Mar 2, 2012 at 9:51 AM, Jared Morrow <jared at basho.com> wrote:
>
>> Andrey,
>>
>> First off, you are indeed sending two emails to the Riak Users list each
>> time you email, not just one, so I think gmail is doing the right thing the
>> first time.
>>
>> Second, I'm sure Shuhao would love some pull requests to help with
>> Riakkit since you seem to have a lot of ideas to make it better.
>>
>> -Jared
>>
>>
>> On Mar 2, 2012, at 6:39 AM, Andrey V. Martyanov wrote:
>>
>> Also, I think we can add RoR-like migrations on top of that with
>> automatic schema versioning and all the stuff.
>>
>> Best regards,
>> Andrey Martyanov
>>
>> On Fri, Mar 2, 2012 at 6:22 PM, Andrey V. Martyanov <realduke at gmail.com>wrote:
>>
>>> I'm sorry for the repost, but my gmail client by default sends an email
>>> to a specific subscriber, not to riak-users.
>>>
>>>
>>> Hi Shuhao,
>>>
>>> I don't have enough time to write my own implementation and to see how
>>> it works in practice. But I share my thoughts about that.
>>> It's possible to save the current API, it's common now and used in
>>> SQLAlchemy and Django ORM. But underlying implementation I see a bit
>>> different. The simple workflow:
>>> 1. Define the models with specific fields and properties;
>>> 2. Generate the JSON Schema [1] based on these fields;
>>> 3. Create an instances of our models;
>>> 4. Validate against the schema;
>>> 5. Generate the optimal sequence of actions we need to do to store the
>>> data, based on the meta information of our models;
>>> 6. Store the data, resolve the conflicts if occur.
>>>
>>> Also it should be possible to deserialize the JSON objects into python
>>> objects.
>>>
>>> [1] - http://json-schema.org/
>>>
>>> Best regards,
>>> Andrey Martyanov
>>>
>>> On Fri, Mar 2, 2012 at 6:17 PM, Andrey V. Martyanov <realduke at gmail.com>wrote:
>>>
>>>> Hi Shuhao,
>>>>
>>>> I don't have enough time to write my own implementation and to see how
>>>> it works in practice. But I share my thoughts about that.
>>>> It's possible to save the current API, it's common now and used in
>>>> SQLAlchemy and Django ORM. But underlying implementation I see a bit
>>>> different. The simple workflow:
>>>> 1. Define the models with specific fields and properties;
>>>> 2. Generate the JSON Schema [1] based on these fields;
>>>> 3. Create an instances of our models;
>>>> 4. Validate against the schema;
>>>> 5. Generate the optimal sequence of actions we need to do to store the
>>>> data, based on the meta information of our models;
>>>> 6. Store the data, resolve the conflicts if occur.
>>>>
>>>> Also it should be possible to deserialize the JSON objects into python
>>>> objects.
>>>>
>>>> [1] - http://json-schema.org/
>>>>
>>>> Best regards,
>>>> Andrey Martyanov
>>>>
>>>>>
>>>>>
>>>>> On Fri, Mar 2, 2012 at 5:47 PM, Shuhao Wu <admin at thekks.net> wrote:
>>>>>
>>>>>> Yeah. You're looking at something like dictshield. As riakkit is more
>>>>>> a RAD tool in its current state. You bring in an excellent point,
>>>>>> nonetheless, which gave me some intuition and ideas about the next step.
>>>>>>
>>>>>> Cheers,
>>>>>> Shuhao
>>>>>> Sent from my phone.
>>>>>> On Mar 1, 2012 11:41 PM, "Andrey V. Martyanov" <realduke at gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Riakkit in its current implementation has a  lot of overhead. But
>>>>>>> what I get for this price? Almost nothing. Only field validation make
>>>>>>> sense. Simple key-value mapping is about 50-70 lines of code, not a big
>>>>>>> deal, even for small project. Riak used mostly in web-based projects.
>>>>>>> Almost all modern projects use JSON. The simple workflow is get JSON from
>>>>>>> the client, validate, store in the database, query the database, send JSON
>>>>>>> to the client. If our mapper doesn't understand the JSON and cannot operate
>>>>>>> on it then it's not useful. We can use jsonpickle with a lot more profit.
>>>>>>> Just jsonpickle.encode the object and store it in Riak. I'm sorry, this is
>>>>>>> my position!
>>>>>>>
>>>>>>> Best regards,
>>>>>>> Andrey Martyanov
>>>>>>>
>>>>>>> On Fri, Mar 2, 2012 at 9:01 AM, Andrey V. Martyanov <
>>>>>>> realduke at gmail.com> wrote:
>>>>>>>
>>>>>>>> I'm just saying that to efficiently store and get the data user
>>>>>>>> should have ability to customize the relationship. Riak has several options
>>>>>>>> to get related data: secondary indexes, map-reduce etc. I think that the
>>>>>>>> most important task of mapper is to validate the data and to track these
>>>>>>>> dependencies. When you have enough meta information about objects storing
>>>>>>>> these objects is not a big problem.
>>>>>>>>
>>>>>>>>
>>>>>>>> On Fri, Mar 2, 2012 at 8:39 AM, Shuhao Wu <admin at thekks.net> wrote:
>>>>>>>>
>>>>>>>>> The thing about riak-python-client is that it's already pretty
>>>>>>>>> high level, but not high level enough to be that simple.
>>>>>>>>>
>>>>>>>>> From what I understand, tracking relationship between models will
>>>>>>>>> still involve a certain level of magic. Right now the magic is that when
>>>>>>>>> you save any document, all the related document will be saved into the
>>>>>>>>> database as well. You're saying that just tracking those relationship
>>>>>>>>> change is good enough and saving will be up to the user?
>>>>>>>>>
>>>>>>>>> Shuhao
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Thu, Mar 1, 2012 at 10:20 PM, Andrey V. Martyanov <
>>>>>>>>> realduke at gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> I'm not sure about forking the riak-python-client. The the whole
>>>>>>>>>> idea of separation the low-level client library from the high-level mapper
>>>>>>>>>> and the like is quite good. Some sort of SQLAlchemy does.
>>>>>>>>>>
>>>>>>>>>> Best regards,
>>>>>>>>>> Andrey Martyanov
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Fri, Mar 2, 2012 at 8:15 AM, Shuhao Wu <admin at thekks.net>wrote:
>>>>>>>>>>
>>>>>>>>>>> Another thing I was considering is forking riak-python-client.
>>>>>>>>>>> Some of the stuff in there could use some fixing as well, and maybe just
>>>>>>>>>>> take out the core transports and such and plug that right into riakkit and
>>>>>>>>>>> skip the notion of RiakObject all together as those API is also somewhat
>>>>>>>>>>> weird. (such as add_link and remove_link)
>>>>>>>>>>>
>>>>>>>>>>> Shuhao
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Thu, Mar 1, 2012 at 10:02 PM, Shuhao Wu <admin at thekks.net>wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi!
>>>>>>>>>>>>
>>>>>>>>>>>> First off, the code is *LGPL* licensed, not GPL. So you're
>>>>>>>>>>>> free to put this into any code you want, it's just that when you want to
>>>>>>>>>>>> modify riakkit, you have to release your modified version.
>>>>>>>>>>>>
>>>>>>>>>>>> I just went over your idea and it would be a valid solution to
>>>>>>>>>>>> some of the problems I'm having right now with my own projects with
>>>>>>>>>>>> riakkit. When I initially started writing riakkit, I needed something to do
>>>>>>>>>>>> RAD with, which would make sense to have an all-in-one, very simple to use
>>>>>>>>>>>> framework, rather than working with serialization and what not. However,
>>>>>>>>>>>> now I'm beginning to see problems in this approach when it comes to
>>>>>>>>>>>> scalability as I'm slowing moving out of the RAD phase. I haven't done
>>>>>>>>>>>> anything as I have not encountered any major issues yet (though I probably
>>>>>>>>>>>> will).
>>>>>>>>>>>>
>>>>>>>>>>>> So here's what I'm thinking for the next branch ( I still have
>>>>>>>>>>>> to maintain the master branch due to some code i've written and need ): I'm
>>>>>>>>>>>> going to break riakkit into 2 pieces. The first piece is the underlying
>>>>>>>>>>>> piece, which will do what you said, serialize/deserialize objects,
>>>>>>>>>>>> validation, track relationships and what not, however, it doesn't load it
>>>>>>>>>>>> those references up nor does it try to accomplish everything all in one
>>>>>>>>>>>> shot, and a RAD layer that's compatible with the current API that will
>>>>>>>>>>>> allow RAD to work with it, but still reserve the flexibility of having that
>>>>>>>>>>>> underlying API that's relatively easy (as oppose to dead simple) to work
>>>>>>>>>>>> with and not as messy as what's going on right now.
>>>>>>>>>>>>
>>>>>>>>>>>> I do agree that the code is messy.. Most of the time it's me
>>>>>>>>>>>> fixing bugs I didn't anticipate as I go along and hacking everything into
>>>>>>>>>>>> place..
>>>>>>>>>>>>
>>>>>>>>>>>> One thing I'm not too clear about is what you mean by "Track
>>>>>>>>>>>> dependencies between models". I'm not sure how you think I should go about
>>>>>>>>>>>> approaching this (which right now is a pretty big disaster.. though
>>>>>>>>>>>> relatively stable)
>>>>>>>>>>>>
>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>
>>>>>>>>>>>> Shuhao
>>>>>>>>>>>>
>>>>>>>>>>>> P.S. I don't want to go into the whole PEP8 discussion and 4
>>>>>>>>>>>> spaces vs whatever.. I just like the 2 spaces and the style guide that
>>>>>>>>>>>> Google uses. Is that really a big deal if you're using it as a library?
>>>>>>>>>>>>
>>>>>>>>>>>> On Thu, Mar 1, 2012 at 9:35 PM, Andrey V. Martyanov <
>>>>>>>>>>>> realduke at gmail.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi!
>>>>>>>>>>>>>
>>>>>>>>>>>>> Two things force me to avoid using Riakkit: the code is GPL
>>>>>>>>>>>>> licensed and non pep8 compliant. So, I cannot fork it and make changes. The
>>>>>>>>>>>>> code needs clean up, for example, json module is imported in document.py,
>>>>>>>>>>>>> but not used. Riakkit has no conflict resolution logic, it's basically
>>>>>>>>>>>>> delegate the work to underlying python client. If I'd like to implement my
>>>>>>>>>>>>> save method I need to copy and rewrite more than 150 lines of code mess.
>>>>>>>>>>>>> What I think the mapper should do:
>>>>>>>>>>>>> 1. Serialize the objects to valid json;
>>>>>>>>>>>>> 2. Deserialize the data back to class instances;
>>>>>>>>>>>>> 3. Validate the fields data;
>>>>>>>>>>>>> 4. Track dependencies between models;
>>>>>>>>>>>>>
>>>>>>>>>>>>> Simple example using Riakkit-like syntax;
>>>>>>>>>>>>>
>>>>>>>>>>>>> class Post(Document):
>>>>>>>>>>>>>
>>>>>>>>>>>>>     bucket_name  = 'posts'
>>>>>>>>>>>>>
>>>>>>>>>>>>>     title = types.StringProperty(required=True)
>>>>>>>>>>>>>     content = types.StringProperty(required=True)
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> class Comment(Document):
>>>>>>>>>>>>>
>>>>>>>>>>>>>     bucket_name = 'comments'
>>>>>>>>>>>>>
>>>>>>>>>>>>>     text = types.StringProperty(required=True)
>>>>>>>>>>>>>     post = types.IndexReferenceProperty(reference_class=Post,
>>>>>>>>>>>>> collection_name='comments')
>>>>>>>>>>>>>
>>>>>>>>>>>>> >>> post = Post('title', 'content')
>>>>>>>>>>>>> >>> comment1 = ('comment1', post=post)
>>>>>>>>>>>>> >>> comment2 = ('comment2', post=post)
>>>>>>>>>>>>> >>> post.is_valid()
>>>>>>>>>>>>> True
>>>>>>>>>>>>> >> post.serialize()
>>>>>>>>>>>>> {
>>>>>>>>>>>>>     "key": "post_key",
>>>>>>>>>>>>>     "title": "title",
>>>>>>>>>>>>>     "content": "content",
>>>>>>>>>>>>>     "comments": [
>>>>>>>>>>>>>         {
>>>>>>>>>>>>>             "key": "comment1_key",
>>>>>>>>>>>>>             "text": "comment1"
>>>>>>>>>>>>>         },
>>>>>>>>>>>>>         {
>>>>>>>>>>>>>             "key": "comment2_key",
>>>>>>>>>>>>>             "text": "comment2"
>>>>>>>>>>>>>         },
>>>>>>>>>>>>>     ]
>>>>>>>>>>>>> }
>>>>>>>>>>>>> >>> post.store()
>>>>>>>>>>>>> True
>>>>>>>>>>>>> >>> post = Post.get_with_key('post_key')
>>>>>>>>>>>>> >>> post.comments
>>>>>>>>>>>>> ['Post comment1', 'Post comment2']
>>>>>>>>>>>>> >>> post_json_data = get_json_data_from_request()
>>>>>>>>>>>>> >>> post = Post.deserialize(post_json_data)
>>>>>>>>>>>>> >>> post.title
>>>>>>>>>>>>> 'title'
>>>>>>>>>>>>> >>> post.comments
>>>>>>>>>>>>> ['Post comment1', 'Post comment2']
>>>>>>>>>>>>>
>>>>>>>>>>>>> Something like that.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Best regards,
>>>>>>>>>>>>> Andrey Martyanov
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Fri, Mar 2, 2012 at 5:39 AM, Sean Carey <carey at basho.com>wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Keep up the great work Shuhao!
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Mar 1, 2012, at 7:36 PM, Shuhao Wu <admin at thekks.net>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hey guys,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I started writing riakkit approximately 3 months ago and
>>>>>>>>>>>>>> announced it here. In case you missed that, Riakkit is basically an object
>>>>>>>>>>>>>> mapper for python and riak, which seemed to be lacking in at the time. Now
>>>>>>>>>>>>>> that 3 months has passed and I have developed this project into much more
>>>>>>>>>>>>>> than what I had envisioned to begin with. It's currently also being used
>>>>>>>>>>>>>> and tested in a project of mine (web application) as well and functioning
>>>>>>>>>>>>>> quite nicely. I would say that currently this is the only object mapper
>>>>>>>>>>>>>> that can compete with their counterparts in other languages for Riak in
>>>>>>>>>>>>>> Python, although a lot of work still needs to be done before it could fully
>>>>>>>>>>>>>> compete.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> So today I felt that riakkit is "stable" and "clean" enough
>>>>>>>>>>>>>> to deserve the first beta release. So I whipped up a website and it's
>>>>>>>>>>>>>> released to pypi. Here's (again), a demo (the full tutorial can be found on
>>>>>>>>>>>>>> the links below)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> class User(Document):
>>>>>>>>>>>>>>     client = riak.RiakClient()
>>>>>>>>>>>>>>     bucket_name = "users"
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>     name = StringProperty(required=True)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> >>> user = User(name="foo")
>>>>>>>>>>>>>> >>> user.save()
>>>>>>>>>>>>>> >>> print user.name
>>>>>>>>>>>>>> foo
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> If anyone is interested, you can find the website here:
>>>>>>>>>>>>>> http://ultimatebuster.github.com/riakkit and the code at
>>>>>>>>>>>>>> http://github.com/ultimatebuster/riakkit
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Cheers!
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Shuhao
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>>> riak-users mailing list
>>>>>>>>>>>>>> riak-users at lists.basho.com
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>>> riak-users mailing list
>>>>>>>>>>>>>> riak-users at lists.basho.com
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>> riak-users mailing list
>>>>>>>>>>>>> riak-users at lists.basho.com
>>>>>>>>>>>>>
>>>>>>>>>>>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> riak-users mailing list
>>>>>>> riak-users at lists.basho.com
>>>>>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>>>>>>
>>>>>>>
>>>>>
>>>>
>>>
>> _______________________________________________
>> riak-users mailing list
>> riak-users at lists.basho.com
>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>
>>
>>
>> _______________________________________________
>> riak-users mailing list
>> riak-users at lists.basho.com
>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.basho.com/pipermail/riak-users_lists.basho.com/attachments/20120302/b9d2a764/attachment.html>


More information about the riak-users mailing list