riakkit, a python riak object mapper, has hit beta!

Andrey V. Martyanov realduke at gmail.com
Fri Mar 2 11:26:25 EST 2012


Basically, I meant sibling reconciliation.

Best regards,
Andrey Martyanov

On Fri, Mar 2, 2012 at 8:01 PM, Shuhao Wu <admin at thekks.net> wrote:

> Also, what do you mean by conflict resolution? Synchronization or
> conflicts between references such as deleting references to a document when
> that document is deleted?
>
> Shuhao
>
>
>
> On Fri, Mar 2, 2012 at 9:56 AM, Shuhao Wu <admin at thekks.net> wrote:
>
>> Actually it's okay for no pull request for now.. I'm in the process of
>> rethinking the whole framework as well as I'm having some of the issues
>> that Andrey is describing as well.
>>
>> Shuhao
>>
>>
>>
>> On Fri, Mar 2, 2012 at 9:51 AM, Jared Morrow <jared at basho.com> wrote:
>>
>>> Andrey,
>>>
>>> First off, you are indeed sending two emails to the Riak Users list each
>>> time you email, not just one, so I think gmail is doing the right thing the
>>> first time.
>>>
>>> Second, I'm sure Shuhao would love some pull requests to help with
>>> Riakkit since you seem to have a lot of ideas to make it better.
>>>
>>> -Jared
>>>
>>>
>>> On Mar 2, 2012, at 6:39 AM, Andrey V. Martyanov wrote:
>>>
>>> Also, I think we can add RoR-like migrations on top of that with
>>> automatic schema versioning and all the stuff.
>>>
>>> Best regards,
>>> Andrey Martyanov
>>>
>>> On Fri, Mar 2, 2012 at 6:22 PM, Andrey V. Martyanov <realduke at gmail.com>wrote:
>>>
>>>> I'm sorry for the repost, but my gmail client by default sends an email
>>>> to a specific subscriber, not to riak-users.
>>>>
>>>>
>>>> Hi Shuhao,
>>>>
>>>> I don't have enough time to write my own implementation and to see how
>>>> it works in practice. But I share my thoughts about that.
>>>> It's possible to save the current API, it's common now and used in
>>>> SQLAlchemy and Django ORM. But underlying implementation I see a bit
>>>> different. The simple workflow:
>>>> 1. Define the models with specific fields and properties;
>>>> 2. Generate the JSON Schema [1] based on these fields;
>>>> 3. Create an instances of our models;
>>>> 4. Validate against the schema;
>>>> 5. Generate the optimal sequence of actions we need to do to store the
>>>> data, based on the meta information of our models;
>>>> 6. Store the data, resolve the conflicts if occur.
>>>>
>>>> Also it should be possible to deserialize the JSON objects into python
>>>> objects.
>>>>
>>>> [1] - http://json-schema.org/
>>>>
>>>> Best regards,
>>>> Andrey Martyanov
>>>>
>>>> On Fri, Mar 2, 2012 at 6:17 PM, Andrey V. Martyanov <realduke at gmail.com
>>>> > wrote:
>>>>
>>>>> Hi Shuhao,
>>>>>
>>>>> I don't have enough time to write my own implementation and to see how
>>>>> it works in practice. But I share my thoughts about that.
>>>>> It's possible to save the current API, it's common now and used in
>>>>> SQLAlchemy and Django ORM. But underlying implementation I see a bit
>>>>> different. The simple workflow:
>>>>> 1. Define the models with specific fields and properties;
>>>>> 2. Generate the JSON Schema [1] based on these fields;
>>>>> 3. Create an instances of our models;
>>>>> 4. Validate against the schema;
>>>>> 5. Generate the optimal sequence of actions we need to do to store the
>>>>> data, based on the meta information of our models;
>>>>> 6. Store the data, resolve the conflicts if occur.
>>>>>
>>>>> Also it should be possible to deserialize the JSON objects into python
>>>>> objects.
>>>>>
>>>>> [1] - http://json-schema.org/
>>>>>
>>>>> Best regards,
>>>>> Andrey Martyanov
>>>>>
>>>>>>
>>>>>>
>>>>>> On Fri, Mar 2, 2012 at 5:47 PM, Shuhao Wu <admin at thekks.net> wrote:
>>>>>>
>>>>>>> Yeah. You're looking at something like dictshield. As riakkit is
>>>>>>> more a RAD tool in its current state. You bring in an excellent point,
>>>>>>> nonetheless, which gave me some intuition and ideas about the next step.
>>>>>>>
>>>>>>> Cheers,
>>>>>>> Shuhao
>>>>>>> Sent from my phone.
>>>>>>> On Mar 1, 2012 11:41 PM, "Andrey V. Martyanov" <realduke at gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Riakkit in its current implementation has a  lot of overhead. But
>>>>>>>> what I get for this price? Almost nothing. Only field validation make
>>>>>>>> sense. Simple key-value mapping is about 50-70 lines of code, not a big
>>>>>>>> deal, even for small project. Riak used mostly in web-based projects.
>>>>>>>> Almost all modern projects use JSON. The simple workflow is get JSON from
>>>>>>>> the client, validate, store in the database, query the database, send JSON
>>>>>>>> to the client. If our mapper doesn't understand the JSON and cannot operate
>>>>>>>> on it then it's not useful. We can use jsonpickle with a lot more profit.
>>>>>>>> Just jsonpickle.encode the object and store it in Riak. I'm sorry, this is
>>>>>>>> my position!
>>>>>>>>
>>>>>>>> Best regards,
>>>>>>>> Andrey Martyanov
>>>>>>>>
>>>>>>>> On Fri, Mar 2, 2012 at 9:01 AM, Andrey V. Martyanov <
>>>>>>>> realduke at gmail.com> wrote:
>>>>>>>>
>>>>>>>>> I'm just saying that to efficiently store and get the data user
>>>>>>>>> should have ability to customize the relationship. Riak has several options
>>>>>>>>> to get related data: secondary indexes, map-reduce etc. I think that the
>>>>>>>>> most important task of mapper is to validate the data and to track these
>>>>>>>>> dependencies. When you have enough meta information about objects storing
>>>>>>>>> these objects is not a big problem.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Fri, Mar 2, 2012 at 8:39 AM, Shuhao Wu <admin at thekks.net>wrote:
>>>>>>>>>
>>>>>>>>>> The thing about riak-python-client is that it's already pretty
>>>>>>>>>> high level, but not high level enough to be that simple.
>>>>>>>>>>
>>>>>>>>>> From what I understand, tracking relationship between models will
>>>>>>>>>> still involve a certain level of magic. Right now the magic is that when
>>>>>>>>>> you save any document, all the related document will be saved into the
>>>>>>>>>> database as well. You're saying that just tracking those relationship
>>>>>>>>>> change is good enough and saving will be up to the user?
>>>>>>>>>>
>>>>>>>>>> Shuhao
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Thu, Mar 1, 2012 at 10:20 PM, Andrey V. Martyanov <
>>>>>>>>>> realduke at gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> I'm not sure about forking the riak-python-client. The the whole
>>>>>>>>>>> idea of separation the low-level client library from the high-level mapper
>>>>>>>>>>> and the like is quite good. Some sort of SQLAlchemy does.
>>>>>>>>>>>
>>>>>>>>>>> Best regards,
>>>>>>>>>>> Andrey Martyanov
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Fri, Mar 2, 2012 at 8:15 AM, Shuhao Wu <admin at thekks.net>wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Another thing I was considering is forking riak-python-client.
>>>>>>>>>>>> Some of the stuff in there could use some fixing as well, and maybe just
>>>>>>>>>>>> take out the core transports and such and plug that right into riakkit and
>>>>>>>>>>>> skip the notion of RiakObject all together as those API is also somewhat
>>>>>>>>>>>> weird. (such as add_link and remove_link)
>>>>>>>>>>>>
>>>>>>>>>>>> Shuhao
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Thu, Mar 1, 2012 at 10:02 PM, Shuhao Wu <admin at thekks.net>wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi!
>>>>>>>>>>>>>
>>>>>>>>>>>>> First off, the code is *LGPL* licensed, not GPL. So you're
>>>>>>>>>>>>> free to put this into any code you want, it's just that when you want to
>>>>>>>>>>>>> modify riakkit, you have to release your modified version.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I just went over your idea and it would be a valid solution to
>>>>>>>>>>>>> some of the problems I'm having right now with my own projects with
>>>>>>>>>>>>> riakkit. When I initially started writing riakkit, I needed something to do
>>>>>>>>>>>>> RAD with, which would make sense to have an all-in-one, very simple to use
>>>>>>>>>>>>> framework, rather than working with serialization and what not. However,
>>>>>>>>>>>>> now I'm beginning to see problems in this approach when it comes to
>>>>>>>>>>>>> scalability as I'm slowing moving out of the RAD phase. I haven't done
>>>>>>>>>>>>> anything as I have not encountered any major issues yet (though I probably
>>>>>>>>>>>>> will).
>>>>>>>>>>>>>
>>>>>>>>>>>>> So here's what I'm thinking for the next branch ( I still have
>>>>>>>>>>>>> to maintain the master branch due to some code i've written and need ): I'm
>>>>>>>>>>>>> going to break riakkit into 2 pieces. The first piece is the underlying
>>>>>>>>>>>>> piece, which will do what you said, serialize/deserialize objects,
>>>>>>>>>>>>> validation, track relationships and what not, however, it doesn't load it
>>>>>>>>>>>>> those references up nor does it try to accomplish everything all in one
>>>>>>>>>>>>> shot, and a RAD layer that's compatible with the current API that will
>>>>>>>>>>>>> allow RAD to work with it, but still reserve the flexibility of having that
>>>>>>>>>>>>> underlying API that's relatively easy (as oppose to dead simple) to work
>>>>>>>>>>>>> with and not as messy as what's going on right now.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I do agree that the code is messy.. Most of the time it's me
>>>>>>>>>>>>> fixing bugs I didn't anticipate as I go along and hacking everything into
>>>>>>>>>>>>> place..
>>>>>>>>>>>>>
>>>>>>>>>>>>> One thing I'm not too clear about is what you mean by "Track
>>>>>>>>>>>>> dependencies between models". I'm not sure how you think I should go about
>>>>>>>>>>>>> approaching this (which right now is a pretty big disaster.. though
>>>>>>>>>>>>> relatively stable)
>>>>>>>>>>>>>
>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Shuhao
>>>>>>>>>>>>>
>>>>>>>>>>>>> P.S. I don't want to go into the whole PEP8 discussion and 4
>>>>>>>>>>>>> spaces vs whatever.. I just like the 2 spaces and the style guide that
>>>>>>>>>>>>> Google uses. Is that really a big deal if you're using it as a library?
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Thu, Mar 1, 2012 at 9:35 PM, Andrey V. Martyanov <
>>>>>>>>>>>>> realduke at gmail.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi!
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Two things force me to avoid using Riakkit: the code is GPL
>>>>>>>>>>>>>> licensed and non pep8 compliant. So, I cannot fork it and make changes. The
>>>>>>>>>>>>>> code needs clean up, for example, json module is imported in document.py,
>>>>>>>>>>>>>> but not used. Riakkit has no conflict resolution logic, it's basically
>>>>>>>>>>>>>> delegate the work to underlying python client. If I'd like to implement my
>>>>>>>>>>>>>> save method I need to copy and rewrite more than 150 lines of code mess.
>>>>>>>>>>>>>> What I think the mapper should do:
>>>>>>>>>>>>>> 1. Serialize the objects to valid json;
>>>>>>>>>>>>>> 2. Deserialize the data back to class instances;
>>>>>>>>>>>>>> 3. Validate the fields data;
>>>>>>>>>>>>>> 4. Track dependencies between models;
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Simple example using Riakkit-like syntax;
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> class Post(Document):
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>     bucket_name  = 'posts'
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>     title = types.StringProperty(required=True)
>>>>>>>>>>>>>>     content = types.StringProperty(required=True)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> class Comment(Document):
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>     bucket_name = 'comments'
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>     text = types.StringProperty(required=True)
>>>>>>>>>>>>>>     post = types.IndexReferenceProperty(reference_class=Post,
>>>>>>>>>>>>>> collection_name='comments')
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> >>> post = Post('title', 'content')
>>>>>>>>>>>>>> >>> comment1 = ('comment1', post=post)
>>>>>>>>>>>>>> >>> comment2 = ('comment2', post=post)
>>>>>>>>>>>>>> >>> post.is_valid()
>>>>>>>>>>>>>> True
>>>>>>>>>>>>>> >> post.serialize()
>>>>>>>>>>>>>> {
>>>>>>>>>>>>>>     "key": "post_key",
>>>>>>>>>>>>>>     "title": "title",
>>>>>>>>>>>>>>     "content": "content",
>>>>>>>>>>>>>>     "comments": [
>>>>>>>>>>>>>>         {
>>>>>>>>>>>>>>             "key": "comment1_key",
>>>>>>>>>>>>>>             "text": "comment1"
>>>>>>>>>>>>>>         },
>>>>>>>>>>>>>>         {
>>>>>>>>>>>>>>             "key": "comment2_key",
>>>>>>>>>>>>>>             "text": "comment2"
>>>>>>>>>>>>>>         },
>>>>>>>>>>>>>>     ]
>>>>>>>>>>>>>> }
>>>>>>>>>>>>>> >>> post.store()
>>>>>>>>>>>>>> True
>>>>>>>>>>>>>> >>> post = Post.get_with_key('post_key')
>>>>>>>>>>>>>> >>> post.comments
>>>>>>>>>>>>>> ['Post comment1', 'Post comment2']
>>>>>>>>>>>>>> >>> post_json_data = get_json_data_from_request()
>>>>>>>>>>>>>> >>> post = Post.deserialize(post_json_data)
>>>>>>>>>>>>>> >>> post.title
>>>>>>>>>>>>>> 'title'
>>>>>>>>>>>>>> >>> post.comments
>>>>>>>>>>>>>> ['Post comment1', 'Post comment2']
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Something like that.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Best regards,
>>>>>>>>>>>>>> Andrey Martyanov
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Fri, Mar 2, 2012 at 5:39 AM, Sean Carey <carey at basho.com>wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Keep up the great work Shuhao!
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Mar 1, 2012, at 7:36 PM, Shuhao Wu <admin at thekks.net>
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hey guys,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I started writing riakkit approximately 3 months ago and
>>>>>>>>>>>>>>> announced it here. In case you missed that, Riakkit is basically an object
>>>>>>>>>>>>>>> mapper for python and riak, which seemed to be lacking in at the time. Now
>>>>>>>>>>>>>>> that 3 months has passed and I have developed this project into much more
>>>>>>>>>>>>>>> than what I had envisioned to begin with. It's currently also being used
>>>>>>>>>>>>>>> and tested in a project of mine (web application) as well and functioning
>>>>>>>>>>>>>>> quite nicely. I would say that currently this is the only object mapper
>>>>>>>>>>>>>>> that can compete with their counterparts in other languages for Riak in
>>>>>>>>>>>>>>> Python, although a lot of work still needs to be done before it could fully
>>>>>>>>>>>>>>> compete.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> So today I felt that riakkit is "stable" and "clean" enough
>>>>>>>>>>>>>>> to deserve the first beta release. So I whipped up a website and it's
>>>>>>>>>>>>>>> released to pypi. Here's (again), a demo (the full tutorial can be found on
>>>>>>>>>>>>>>> the links below)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> class User(Document):
>>>>>>>>>>>>>>>     client = riak.RiakClient()
>>>>>>>>>>>>>>>     bucket_name = "users"
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>     name = StringProperty(required=True)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> >>> user = User(name="foo")
>>>>>>>>>>>>>>> >>> user.save()
>>>>>>>>>>>>>>> >>> print user.name
>>>>>>>>>>>>>>> foo
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> If anyone is interested, you can find the website here:
>>>>>>>>>>>>>>> http://ultimatebuster.github.com/riakkit and the code at
>>>>>>>>>>>>>>> http://github.com/ultimatebuster/riakkit
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Cheers!
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Shuhao
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>>>> riak-users mailing list
>>>>>>>>>>>>>>> riak-users at lists.basho.com
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>>>> riak-users mailing list
>>>>>>>>>>>>>>> riak-users at lists.basho.com
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>>> riak-users mailing list
>>>>>>>>>>>>>> riak-users at lists.basho.com
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> riak-users mailing list
>>>>>>>> riak-users at lists.basho.com
>>>>>>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>>>>>>>
>>>>>>>>
>>>>>>
>>>>>
>>>>
>>> _______________________________________________
>>> riak-users mailing list
>>> riak-users at lists.basho.com
>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>>
>>>
>>>
>>> _______________________________________________
>>> riak-users mailing list
>>> riak-users at lists.basho.com
>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.basho.com/pipermail/riak-users_lists.basho.com/attachments/20120302/980b2c64/attachment.html>


More information about the riak-users mailing list