Lager and Scribe

Jon Brisbin jon at jbrisbin.com
Thu Nov 17 15:10:28 EST 2011


Not to hijack your thread or anything, but I just pushed to Github a lager backend for sending stuff to AMQP. You could have a subscriber in Java or whatever that fed those messages directly into HDFS and wouldn't have to worry about Thrift directly in the backend... 

https://github.com/jbrisbin/lager_amqp_backend 

Thanks! 

Jon Brisbin 
http//jbrisbin.com 

----- Original Message -----

> From: "Bip Thelin" <bip.thelin at evolope.se>
> To: riak-users at lists.basho.com
> Sent: Thursday, November 17, 2011 9:52:58 AM
> Subject: Lager and Scribe

> Hi,

> I've been thinking about using lager to push logs to our Big Data
> Hadoop backend for later analysis using Scribe. Looking at the
> lager_syslog backend it looks fairly simple to create such a
> backend. But thrift works kind of different and some implementation
> questions occur when going through the lager code.

> The way thrift works is that you create a connection:
> {ok, Client} = thrift_client_util:new()

> then you can use the Client to do subsequent calls like
> {Client2, _Result} = thrift_client:call(Client, ...)

> {Client3, _Result} = thrift_client:call(Client2, ...)
> [...]

> The problem is that you get a new Client descriptor every time and as
> I've understood it you shouldn't use the same Client twice but
> instead use the new one with each call. One could of course create a
> client and do the call within each handle_event() and pay that
> penalty hit but there must be a smarter way. I was going through the
> lager_file_backend and saw that you use a FD that get's passed
> around, is that a good approach to use something similar for a
> Scribe backend?

> --
> Bip Thelin

> Evolope AB | Lugnets Allé 1 | 120 33 Stockholm
> Tel 08-533 335 37 | Mob 0735-18 18 90
> www. evolope.se

> _______________________________________________
> riak-users mailing list
> riak-users at lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.basho.com/pipermail/riak-users_lists.basho.com/attachments/20111117/7ad46bde/attachment.html>


More information about the riak-users mailing list