C++ Riak client?

David Leimbach leimy2k at gmail.com
Thu May 5 19:14:26 EDT 2011


Thanks!

On Thu, May 5, 2011 at 3:11 PM, Mike Oxford <moxford at gmail.com> wrote:

> Protobuf does not do inter-message delimiters.  You have to handle that
> aspect yourself.
>
> In the streambuf, put in the length of your protobuf message (in network
> format), then the stream the message into it.
> On the other end they should extract the length, convert to host format and
> then read the data in and ship to protobuf.
>

Ah this seems like some part I hadn't read up to yet...  It's difficult to
do this stuff without contiguous blocks of time :-).

Some of this makes me want to also investigate using BERT or BERT-RPC
straight to erlang's "native" client for C++ bindings and just skip protocol
buffers altogether :-).

Yes I know protocol buffers are trendy and all, but even Google's Go
programming language doesn't support them natively (without external
packages).  They did their own encoding as well (gob).

https://github.com/blog/531-introducing-bert-and-bert-rpc

http://bert-rpc.org/

Dave


>
> That's the 10,000 foot view.
>
> http://code.google.com/apis/protocolbuffers/docs/techniques.html
>
> --snip
> If you want to write multiple messages to a single file or stream, it is up
> to you to keep track of where one message ends and the next begins. The
> Protocol Buffer wire format is not self-delimiting, so protocol buffer
> parsers cannot determine where a message ends on their own. The easiest way
> to solve this problem is to write the size of each message before you write
> the message itself. When you read the messages back in, you read the size,
> then read the bytes into a separate buffer, then parse from that buffer. (If
> you want to avoid copying bytes to a separate buffer, check out the
> CodedInputStream class (in both C++ and Java) which can be told to limit
> reads to a certain number of bytes.)
>
> --end snip
>
> You do have to parse a message in its entirety; there is no incremental
> parsing.  If you have a message which wraps 10,000 messages you'll have to
> load the whole thing in before you can parse.
>
> HTH.
>
> -mox
>
>
>
>
>
> On Thu, May 5, 2011 at 3:02 PM, David Leimbach <leimy2k at gmail.com> wrote:
>
>> I haven't completely abandoned mine yet either, but I've been finding that
>> some of the  I wanted to use with streambufs aren't going to pan out due to
>> the nature of a socket backed stream and the way protocol buffers expects it
>> to work.  Protocol Buffers expects the stream to be terminated when done
>> receiving a message *THEN* parse it rather than building the parsing in (I
>> think that's what happened anyway... I really need to check my notes I took
>> when I ran into this).  This does appear to make a simple socket fd backed
>> streambuf way to plug into any protocol buffers stream a bit more difficult
>> than I was hoping.
>>
>> Then again, I may just not be overriding a particular member function that
>> I should be either.
>>
>> Might get some time on the weekend to poke at this some more.
>>
>> On the plus side I definitely had some Riak RPC binary protocol bits
>> working.  I was encoding/decoding Ping/Pong just fine (and they don't need
>> any Protocol Buffer anything to work).
>>
>> Dave
>>
>> On Sat, Apr 9, 2011 at 8:47 AM, Sean Cribbs <sean at basho.com> wrote:
>>
>>> I didn't permanently abandon it, but it was much more fiddly than doing
>>> the same thing in pure Ruby.  I have plans to deliver separate "native"
>>> Protocol Buffers libraries for MRI and JRuby (at least) in 1.0 of the Ruby
>>> client.
>>>
>>> Because it's being confused in this conversation, I think it merits
>>> clarification -- the "protocol" that is used to talk to Riak and Google's
>>> Protocol Buffers are NOT the same thing. Riak uses a simple length- and
>>> message-code-prefixed binary protocol, in which the complex messages (ones
>>> that have bodies and not just the message code) are serialized via Google's
>>> Protocol Buffers.  So, while we don't use the RPC facilities in Google's
>>> library, the *serialization format* DOES use Protocol Buffers.
>>>
>>> Sorry for the confusion, we'll work to make that clearer in the wiki.
>>>
>>>  Sean Cribbs <sean at basho.com>
>>> Developer Advocate
>>> Basho Technologies, Inc.
>>> http://basho.com/
>>>
>>> On Apr 8, 2011, at 9:17 PM, Scott Gonyea wrote:
>>>
>>> They are the same and you can actually see me plugging into the C++ code
>>> here:
>>>
>>> https://github.com/sgonyea/pabst/tree/master/ext
>>>
>>> But as part of an Objective-C library (called ObjFW).  So, the code is
>>> actually an Objective-C++ wrapper around the C++ PB code, that exchanges
>>> messages with Objective-C code (that hooks into Ruby).
>>>
>>> I believe Sean Cribbs has some initial C++-wrapper code in his Ripple
>>> repo...  Though he eventually abandoned it after C++ left him permanently
>>> cross-eyed (I think that's why).
>>>
>>> Scott
>>>
>>> On Apr 8, 2011, at 5:20 PM, Mike Oxford wrote:
>>>
>>> Be careful here..
>>>
>>> I do not thing Riak's "protocol buffers" are the same as Google's
>>> protocol buffers.
>>>  Google's does bit-level packing and some other tricks that Riak does
>>> not do, even though they both use the ".proto" file extension and very very
>>> similar proto semantics.
>>>
>>> That said, if they ARE the same, then you can take the .proto files and
>>> generate C++ classes, and use the secondary library "protobuf-c" to generate
>>> C structs for the wire format.
>>>
>>> -mox
>>>
>>> On Fri, Apr 8, 2011 at 4:43 PM, David Leimbach <leimy2k at gmail.com>wrote:
>>>
>>>> Spent a little time poking at this today... Kind of surprised that there
>>>> was no message defined for PingReq or for listing buckets.
>>>>
>>>> I realize these messages really have no usable payload, and just sort of
>>>> have a tag and length, but for completeness it kind of feels like they
>>>> should be there.
>>>>
>>>> Of course I'm not a Protocol Buffers expert in any sense, so I can't say
>>>> whether this is a normal kind of choice or not.
>>>>
>>>> Dave
>>>>
>>>>
>>>> On Fri, Apr 8, 2011 at 2:49 PM, Scott Gonyea <scott at aitrus.org> wrote:
>>>>
>>>>> If we had this then a C-wrapper would be that much more attainable. So,
>>>>> the author of such a lib would be a superstar in my book :).
>>>>>
>>>>> Sent from my iPhone
>>>>>
>>>>> On Apr 8, 2011, at 1:46 PM, David Leimbach <leimy2k at gmail.com> wrote:
>>>>>
>>>>> > I've been writing a bit of code in Haskell to push data to Riak, and
>>>>> the bindings are pretty easy to use (Thanks Brian!), but getting penetration
>>>>> at my company for Haskell is going to take a little time.
>>>>> >
>>>>> > As such I'm just wondering if anyone knows of anyone working on a
>>>>> protocol buffers version of a Riak client in C++, or if this is going to be
>>>>> something I'll have to take on.
>>>>> >
>>>>> > I've found a few generic looking C++ projects that use Boost's
>>>>> asynchronous IO stuff with protocol buffers to make an RPC system, but I'm
>>>>> not sure if any of those are implicitly compatible.
>>>>> >
>>>>> > Guess I'm just looking for a pointer...
>>>>> >
>>>>> > Dave
>>>>> > _______________________________________________
>>>>> > riak-users mailing list
>>>>> > riak-users at lists.basho.com
>>>>> > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> riak-users mailing list
>>>> riak-users at lists.basho.com
>>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>>>
>>>>
>>> _______________________________________________
>>> riak-users mailing list
>>> riak-users at lists.basho.com
>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>>
>>>
>>> _______________________________________________
>>> riak-users mailing list
>>> riak-users at lists.basho.com
>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>>
>>>
>>>
>>> _______________________________________________
>>> riak-users mailing list
>>> riak-users at lists.basho.com
>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>>
>>>
>>
>> _______________________________________________
>> riak-users mailing list
>> riak-users at lists.basho.com
>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.basho.com/pipermail/riak-users_lists.basho.com/attachments/20110505/7ab3d6e1/attachment.html>


More information about the riak-users mailing list