C++ Riak client?

David Leimbach leimy2k at gmail.com
Thu May 5 21:53:34 EDT 2011


On Thu, May 5, 2011 at 4:14 PM, David Leimbach <leimy2k at gmail.com> wrote:

> Thanks!
>
> On Thu, May 5, 2011 at 3:11 PM, Mike Oxford <moxford at gmail.com> wrote:
>
>> Protobuf does not do inter-message delimiters.  You have to handle that
>> aspect yourself.
>>
>> In the streambuf, put in the length of your protobuf message (in network
>> format), then the stream the message into it.
>> On the other end they should extract the length, convert to host format
>> and then read the data in and ship to protobuf.
>>
>
> Ah this seems like some part I hadn't read up to yet...  It's difficult to
> do this stuff without contiguous blocks of time :-).
>
> Some of this makes me want to also investigate using BERT or BERT-RPC
> straight to erlang's "native" client for C++ bindings and just skip protocol
> buffers altogether :-).
>
> Yes I know protocol buffers are trendy and all, but even Google's Go
> programming language doesn't support them natively (without external
> packages).  They did their own encoding as well (gob).
>
> https://github.com/blog/531-introducing-bert-and-bert-rpc
>
> http://bert-rpc.org/
>
> Dave
>

But I suppose that would be pretty wasteful given that I think the erlang
riak client is all protocol buffers too?

Dave


>
>
>>
>> That's the 10,000 foot view.
>>
>> http://code.google.com/apis/protocolbuffers/docs/techniques.html
>>
>> --snip
>> If you want to write multiple messages to a single file or stream, it is
>> up to you to keep track of where one message ends and the next begins. The
>> Protocol Buffer wire format is not self-delimiting, so protocol buffer
>> parsers cannot determine where a message ends on their own. The easiest way
>> to solve this problem is to write the size of each message before you write
>> the message itself. When you read the messages back in, you read the size,
>> then read the bytes into a separate buffer, then parse from that buffer. (If
>> you want to avoid copying bytes to a separate buffer, check out the
>> CodedInputStream class (in both C++ and Java) which can be told to limit
>> reads to a certain number of bytes.)
>>
>> --end snip
>>
>> You do have to parse a message in its entirety; there is no incremental
>> parsing.  If you have a message which wraps 10,000 messages you'll have to
>> load the whole thing in before you can parse.
>>
>> HTH.
>>
>> -mox
>>
>>
>>
>>
>>
>> On Thu, May 5, 2011 at 3:02 PM, David Leimbach <leimy2k at gmail.com> wrote:
>>
>>> I haven't completely abandoned mine yet either, but I've been finding
>>> that some of the  I wanted to use with streambufs aren't going to pan out
>>> due to the nature of a socket backed stream and the way protocol buffers
>>> expects it to work.  Protocol Buffers expects the stream to be terminated
>>> when done receiving a message *THEN* parse it rather than building the
>>> parsing in (I think that's what happened anyway... I really need to check my
>>> notes I took when I ran into this).  This does appear to make a simple
>>> socket fd backed streambuf way to plug into any protocol buffers stream a
>>> bit more difficult than I was hoping.
>>>
>>> Then again, I may just not be overriding a particular member function
>>> that I should be either.
>>>
>>> Might get some time on the weekend to poke at this some more.
>>>
>>> On the plus side I definitely had some Riak RPC binary protocol bits
>>> working.  I was encoding/decoding Ping/Pong just fine (and they don't need
>>> any Protocol Buffer anything to work).
>>>
>>> Dave
>>>
>>> On Sat, Apr 9, 2011 at 8:47 AM, Sean Cribbs <sean at basho.com> wrote:
>>>
>>>> I didn't permanently abandon it, but it was much more fiddly than doing
>>>> the same thing in pure Ruby.  I have plans to deliver separate "native"
>>>> Protocol Buffers libraries for MRI and JRuby (at least) in 1.0 of the Ruby
>>>> client.
>>>>
>>>> Because it's being confused in this conversation, I think it merits
>>>> clarification -- the "protocol" that is used to talk to Riak and Google's
>>>> Protocol Buffers are NOT the same thing. Riak uses a simple length- and
>>>> message-code-prefixed binary protocol, in which the complex messages (ones
>>>> that have bodies and not just the message code) are serialized via Google's
>>>> Protocol Buffers.  So, while we don't use the RPC facilities in Google's
>>>> library, the *serialization format* DOES use Protocol Buffers.
>>>>
>>>> Sorry for the confusion, we'll work to make that clearer in the wiki.
>>>>
>>>>  Sean Cribbs <sean at basho.com>
>>>> Developer Advocate
>>>> Basho Technologies, Inc.
>>>> http://basho.com/
>>>>
>>>> On Apr 8, 2011, at 9:17 PM, Scott Gonyea wrote:
>>>>
>>>> They are the same and you can actually see me plugging into the C++ code
>>>> here:
>>>>
>>>> https://github.com/sgonyea/pabst/tree/master/ext
>>>>
>>>> But as part of an Objective-C library (called ObjFW).  So, the code is
>>>> actually an Objective-C++ wrapper around the C++ PB code, that exchanges
>>>> messages with Objective-C code (that hooks into Ruby).
>>>>
>>>> I believe Sean Cribbs has some initial C++-wrapper code in his Ripple
>>>> repo...  Though he eventually abandoned it after C++ left him permanently
>>>> cross-eyed (I think that's why).
>>>>
>>>> Scott
>>>>
>>>> On Apr 8, 2011, at 5:20 PM, Mike Oxford wrote:
>>>>
>>>> Be careful here..
>>>>
>>>> I do not thing Riak's "protocol buffers" are the same as Google's
>>>> protocol buffers.
>>>>  Google's does bit-level packing and some other tricks that Riak does
>>>> not do, even though they both use the ".proto" file extension and very very
>>>> similar proto semantics.
>>>>
>>>> That said, if they ARE the same, then you can take the .proto files and
>>>> generate C++ classes, and use the secondary library "protobuf-c" to generate
>>>> C structs for the wire format.
>>>>
>>>> -mox
>>>>
>>>> On Fri, Apr 8, 2011 at 4:43 PM, David Leimbach <leimy2k at gmail.com>wrote:
>>>>
>>>>> Spent a little time poking at this today... Kind of surprised that
>>>>> there was no message defined for PingReq or for listing buckets.
>>>>>
>>>>> I realize these messages really have no usable payload, and just sort
>>>>> of have a tag and length, but for completeness it kind of feels like they
>>>>> should be there.
>>>>>
>>>>> Of course I'm not a Protocol Buffers expert in any sense, so I can't
>>>>> say whether this is a normal kind of choice or not.
>>>>>
>>>>> Dave
>>>>>
>>>>>
>>>>> On Fri, Apr 8, 2011 at 2:49 PM, Scott Gonyea <scott at aitrus.org> wrote:
>>>>>
>>>>>> If we had this then a C-wrapper would be that much more attainable.
>>>>>> So, the author of such a lib would be a superstar in my book :).
>>>>>>
>>>>>> Sent from my iPhone
>>>>>>
>>>>>> On Apr 8, 2011, at 1:46 PM, David Leimbach <leimy2k at gmail.com> wrote:
>>>>>>
>>>>>> > I've been writing a bit of code in Haskell to push data to Riak, and
>>>>>> the bindings are pretty easy to use (Thanks Brian!), but getting penetration
>>>>>> at my company for Haskell is going to take a little time.
>>>>>> >
>>>>>> > As such I'm just wondering if anyone knows of anyone working on a
>>>>>> protocol buffers version of a Riak client in C++, or if this is going to be
>>>>>> something I'll have to take on.
>>>>>> >
>>>>>> > I've found a few generic looking C++ projects that use Boost's
>>>>>> asynchronous IO stuff with protocol buffers to make an RPC system, but I'm
>>>>>> not sure if any of those are implicitly compatible.
>>>>>> >
>>>>>> > Guess I'm just looking for a pointer...
>>>>>> >
>>>>>> > Dave
>>>>>> > _______________________________________________
>>>>>> > riak-users mailing list
>>>>>> > riak-users at lists.basho.com
>>>>>> > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> riak-users mailing list
>>>>> riak-users at lists.basho.com
>>>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>>>>
>>>>>
>>>> _______________________________________________
>>>> riak-users mailing list
>>>> riak-users at lists.basho.com
>>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>>>
>>>>
>>>> _______________________________________________
>>>> riak-users mailing list
>>>> riak-users at lists.basho.com
>>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> riak-users mailing list
>>>> riak-users at lists.basho.com
>>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>>>
>>>>
>>>
>>> _______________________________________________
>>> riak-users mailing list
>>> riak-users at lists.basho.com
>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.basho.com/pipermail/riak-users_lists.basho.com/attachments/20110505/d71652a5/attachment.html>


More information about the riak-users mailing list