Lager and Scribe

Bip Thelin bip.thelin at evolope.se
Fri Nov 18 05:01:26 EST 2011


Interesting,

 one idea we've tossed around is to use zeromq as a transport layer. Implement a lager_zeromq_backend and a simple zeromq receiver feeding messages into HDFS. Will look at your amqp backend!

--
Bip Thelin
 
Evolope AB | Lugnets Allé 1 | 120 33 Stockholm
Tel 08-533 335 37 | Mob 0735-18 18 90
www.evolope.se

On 17 nov 2011, at 21:10, Jon Brisbin wrote:

> Not to hijack your thread or anything, but I just pushed to Github a lager backend for sending stuff to AMQP. You could have a subscriber in Java or whatever that fed those messages directly into HDFS and wouldn't have to worry about Thrift directly in the backend...
> 
> https://github.com/jbrisbin/lager_amqp_backend
> 
> 
> Thanks!
> 
> Jon Brisbin
> http//jbrisbin.com
> 
> From: "Bip Thelin" <bip.thelin at evolope.se>
> To: riak-users at lists.basho.com
> Sent: Thursday, November 17, 2011 9:52:58 AM
> Subject: Lager and Scribe
> 
> Hi,
> 
>  I've been thinking about using lager to push logs to our Big Data Hadoop backend for later analysis using Scribe. Looking at the lager_syslog backend it looks fairly simple to create such a backend. But thrift works kind of different and some implementation questions occur when going through the lager code.
> 
> The way thrift works is that you create a connection:
> {ok, Client} = thrift_client_util:new()
> 
> then you can use the Client to do subsequent calls like
> {Client2, _Result} = thrift_client:call(Client, ...)
> {Client3, _Result} = thrift_client:call(Client2, ...)
> [...]
> 
> The problem is that you get a new Client descriptor every time and as I've understood it you shouldn't use the same Client twice but instead use the new one with each call. One could of course create a client and do the call within each handle_event() and pay that penalty hit but there must be a smarter way. I was going through the lager_file_backend and saw that you use a FD that get's passed around, is that a good approach to use something similar for a Scribe backend?
> 
> --
> Bip Thelin
>  
> Evolope AB | Lugnets Allé 1 | 120 33 Stockholm
> Tel 08-533 335 37 | Mob 0735-18 18 90
> www.evolope.se
> 
> 
> _______________________________________________
> riak-users mailing list
> riak-users at lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.basho.com/pipermail/riak-users_lists.basho.com/attachments/20111118/6c8db3a4/attachment.html>


More information about the riak-users mailing list