Feedback for GSoC project - RIak Destination for Syslog-ng

Christopher Meiklejohn cmeiklejohn at basho.com
Sun May 3 17:47:53 EDT 2015


> On May 2, 2015, at 2:48 PM, Parth Oberoi <htrapdev at gmail.com> wrote:
> 
> Hello Everyone,
> 
> My proposal[0] for adding Riak as a destination to Syslog-ng was accepted for GSoC2015, so before my coding period begins i.e. 25th May, I wanted to do some ground work and take some feedback from the potential users.
> The proposal[0] contains all the initial implementation details which might require improvement .
> Therefore if some of you could take a look and give me some feedback on it, would  help me greatly. 
> Below are a few links to important snippets of the proposal.
> 
> 
> [0] - https://github.com/balabit/syslog-ng/wiki/GSoC2015-Proposal:-Riak-destination-(htrap)
> 
> Goals for the project :-  https://github.com/balabit/syslog-ng/wiki/GSoC2015-Proposal:-Riak-destination-(htrap)#goals-for-the-project
> 
> Development Tools :-  https://github.com/balabit/syslog-ng/wiki/GSoC2015-Proposal:-Riak-destination-(htrap)#development-tools
> 
> Deliverables :-  https://github.com/balabit/syslog-ng/wiki/GSoC2015-Proposal:-Riak-destination-(htrap)#deliverables
> 
> My scheduled Plan:-   https://github.com/balabit/syslog-ng/wiki/GSoC2015-Proposal:-Riak-destination-(htrap)#scheduled-plan---

Hi Parth,

I’ve only had time to briefly read the overview and the goals.  That said, this sounds great!  I’m happy to assist in any way possible.

I’m a bit concerned with your use of the set embedded in the map.  Large objects have traditionally been a big problem in Riak due to the use of distributed Erlang and head of line blocking.  I’m curious if you could elaborate on what type of data you will be storing in the set: how big you expect each item to be, how big you expect the map to be, and the overall layout of data inside of the data structure.  I’ve provided some references on why these things might be problematic below.

Thanks,
- Chris

[1] http://stackoverflow.com/questions/24389183/riak-returns-an-error-reading-large-object-of-size-when-a-mapreduce
[2] http://stackoverflow.com/questions/10491156/is-riak-recommended-for-storing-and-streaming-large-binary-files
[3] http://basho.com/riak-development-anti-patterns/
[4] http://dl.acm.org/citation.cfm?id=2596635

Christopher Meiklejohn
Senior Software Engineer
Basho Technologies, Inc.
cmeiklejohn at basho.com





More information about the riak-users mailing list