native erlang client on non-Riak node?

Alexander Sicular siculars at gmail.com
Thu Apr 5 16:21:21 EDT 2012


Ya, that is exactly what I had in mind. A non data storage cluster member. Specifically to act as additional read/computation asset.

-Alexander Sicular

@siculars

On Apr 5, 2012, at 4:12 PM, John E. Vincent wrote:

> ElasticSearch has this feature as well.
> 
> When using the Java transport client (as opposed to the HTTP/JSON
> interface), you actually become a cluster member as a non-data node.
> You can also add any number of nodes to the cluster as non-data nodes
> that then participate in general processing.
> 
> http://www.elasticsearch.org/guide/reference/modules/node.html
> http://www.elasticsearch.org/guide/reference/java-api/client.html
> 
> On Thu, Apr 5, 2012 at 3:52 PM, Alexander Sicular <siculars at gmail.com> wrote:
>> I think that being able to have a native erlang interface to a riak cluster
>> would be a "Good Thing." I can imagine some compute applications that could
>> benefit from having access to data at the speed of network but not actually
>> be a member of the storage pool.
>> 
>> Just watched Rich Hickey discussing similar concepts employed in his new
>> system, Datomic, here, http://www.infoq.com/interviews/hickey-datomic .
>> Worth the 30min.
>> 
>> -Alexander Sicular
>> 
>> @siculars
>> 
>> On Apr 5, 2012, at 3:30 PM, Ryan Zezeski wrote:
>> 
>> Mike,
>> 
>> The native Erlang client was never designed to be used in this fashion, as
>> you are finding out first hand.  You are correct in that some operations
>> assume they are running on the Riak VM, e.g. mapred_dynamic_inputs_stream.
>> 
>> Normally I would say not to establish a Erlang dist connection to the Riak
>> cluster but given you need a work around for the performance issue it seems
>> to be your only good choice ATM.  You can use `rpc:call` to invoke the
>> mapred queries.
>> 
>> -Z
>> 
>> On Wed, Apr 4, 2012 at 2:40 PM, Michael Radford <mrad at blorf.com> wrote:
>>> 
>>> In case anyone else is wondering, the answer is no, the native client
>>> isn't fully usable from a non-Riak node.
>>> 
>>> I started a hidden node and connected to one of my Riak nodes using
>>> riak:client_connect. The returned client is able to successfully
>>> execute Client:get(Bucket, Key) (and probably other simple things like
>>> put/delete), but it fails on mapred with a {modfun, ...} input.
>>> 
>>> 
>>> The problem is that the native client tries to run the input function
>>> on the local node, rather than the node the client points to, which of
>>> course won't work unless the local node is participating in the Riak
>>> cluster.
>>> 
>>> Mike
>>> 
>>> On Tue, Apr 3, 2012 at 9:43 AM, Michael Radford <mrad at blorf.com> wrote:
>>>> Is it OK to use Riak's native erlang client from an erlang node that's
>>>> connected to a Riak cluster via normal erlang distribution
>>>> (net_adm:ping etc.), but isn't participating in the Riak cluster?  I'm
>>>> wondering if e.g. Riak makes any assumptions about nodes listed in
>>>> nodes(), or the nodes of processes running client code, that would
>>>> break if a node isn't running Riak.
>>>> 
>>>> I'm trying to figure out a fallback strategy in case I can't resolve
>>>> performance issues I'm seeing with the protocol buffers client (see
>>>> other thread about slow searches).  The scariest option would be to
>>>> run my application on the same nodes as Riak, which seems like it
>>>> would complicate operations too much.  Or if
>>>> connected-but-not-participating nodes aren't a good idea, maybe my
>>>> application nodes could each start a slave node with -hidden, which
>>>> would then connect to Riak and act as a proxy?
>>>> 
>>>> Mike
>>> 
>>> _______________________________________________
>>> riak-users mailing list
>>> riak-users at lists.basho.com
>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>> 
>> 
>> _______________________________________________
>> riak-users mailing list
>> riak-users at lists.basho.com
>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>> 
>> 
>> 
>> _______________________________________________
>> riak-users mailing list
>> riak-users at lists.basho.com
>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>> 





More information about the riak-users mailing list