Python client performance issue

Nico Meyer nico.meyer at adition.com
Wed Feb 16 05:34:09 EST 2011


Ah, I see it now. The thing is, we don't usually install libraries on
each machine manually but build Debian packages for them. In our case we
missed the automatically installed protobuf-2.3. I have to check the
difference in performance.

But back to the original problem. The other obvious culprit is the json
encoder/decoder. By default riak-client uses the built in json module,
which is also notoriously slow, especially at decoding as some sources
suggest (e.g.
http://shane.willowrise.com/archives/python-json-performance/), but I'm
to lazy to check myself right now ;-).

Since Mike uses the default of return_body=True for the store(), his
date has to be encoded AND decoded on every put.

Mike: can you first try to replace 'entry.store()' with
'entry.store(return_body=False)'? That should already be much faster. If
it is still not fast enough you can replace the default encoder/decoder
by using the client.set_encoder/set_decoder methods.

Cheers,
Nico


Am Dienstag, den 15.02.2011, 10:05 -0800 schrieb Andy Gross:
> 
> 
> 
> 
> Sorry, I should have been more clear.  The Python client depends on
> "protobuf>=2.3.0" in setup.py, so people are already most likely using
> protobufs-2.3.0.
> 
> 
> - Andy
> 
> On Tuesday, February 15, 2011 at 3:09 AM, Nico Meyer wrote:
> 
> > Hi Andy.
> > 
> > I am not quite sure what you mean, is the protobuf library included
> > with
> > riak-python-client? Or are you talking about the version of the
> > protobuf
> > compiler that was used to create riakclient_pb2.py from 
> > riakclient.proto?
> > 
> > Cheers,
> > Nico
> > 
> > Am Dienstag, den 15.02.2011, 02:23 -0800 schrieb Andy Gross:
> > > python-riak-client already uses version 2.3.0. Adventurous types
> > > might want to check out https://github.com/Greplin/fast-python-pb,
> > > which wraps the C/C++ protocol buffers library. 
> > > 
> > > -- 
> > > Andy Gross
> > > Principal Architect
> > > Basho Technologies, Inc.







More information about the riak-users mailing list