40 million dataset load(2.1 GB) strauk up

Sangeetha.PattabiRaman2 at cognizant.com Sangeetha.PattabiRaman2 at cognizant.com
Wed Oct 3 00:11:15 EDT 2012


Dear team,


I am trying to load a 2.1 GB of sample csv customercall(40 million ) dataset .using erlang script to load the same

escript: exception error: no match of right hand side value {error,enoent}

escript: exception error: no match of right hand side value {error,enomem}


keep on getting these erros.could you please help me out with the same in moving forward.
On giving  ./load_data40m CustCalls40m.csv....  as a means for performance analysis.

Vi load_data40m :


#!/usr/local/bin/escript
main([Filename]) ->
    {ok, Data} = file:read_file(Filename),
    Lines = tl(re:split(Data, "\r?\n", [{return, binary},trim])),
    lists:foreach(fun(L) -> LS = re:split(L, ","), format_and_insert(LS) end, Lines).

format_and_insert(Line) ->
    JSON = io_lib:format("{\"id\":\"~s\",\"phonenumber\":~s,\"callednumber\":~s,\"starttime\":~s,\"endtime\":~s,\"status\":~s}", Line),
    Command = io_lib:format("curl -X PUT http://10.232.5.170:8098/riak/CustCalls40m/~s -d '~s' -H 'content-type: application/json'", [hd(Line),JSON]),
    io:format("Inserting: ~s~n", [hd(Line)]),
    os:cmd(Command).

Sample CustCalls40m.csv

[hadoop at CTSINGMRGTO1 data]$ head -5 CustCalls40m.csv
id,phonenumber,callednumber,starttime,endtime,status
1,9292320,9894970,1290740408,1290790220,inactive
2,9369618,9971350,1290731959,1290792354,inactive
3,9528924,9599589,1290744994,1290806657,active
4,9861270,9517988,1290754694,1290787091,inactive







Riak Log is as follows

2012-10-01 02:10:46.610 [error] <0.70.0> gen_server memsup terminated with reason: {port_died,normal}
2012-10-01 02:10:46.723 [error] <0.70.0> CRASH REPORT Process memsup with 0 neighbours exited with reason: {port_died,normal} in gen_server:terminate/6 line 747
2012-10-01 02:10:46.725 [info] <0.42.0> alarm_handler: {clear,system_memory_high_watermark}
2012-10-01 02:10:46.732 [error] <0.68.0> Supervisor os_mon_sup had child memsup started with memsup:start_link() at <0.70.0> exit with reason {port_died,normal} in context child_terminated
2012-10-01 02:10:46.733 [info] <0.42.0> alarm_handler: {set,{system_memory_high_watermark,[]}}
~




012-10-01 09:02:31.273 [info] <0.405.0>@riak_kv_js_vm:terminate:240 Spidermonkey VM (pool: riak_kv_js_map) host stopping (<0.405.0>)
2012-10-01 09:02:43.003 [error] <0.1056.0> gen_fsm <0.1056.0> in state ready terminated with reason: {timeout,{gen_fsm,sync_send_all_state_event,[<0.1057.0>,stop]}}
2012-10-01 09:02:50.426 [error] <0.1056.0> CRASH REPORT Process <0.1056.0> with 0 neighbours exited with reason: {timeout,{gen_fsm,sync_send_all_state_event,[<0.1057.0>,stop]}} in gen_fsm:handle_msg/7 line 511
2012-10-01 09:02:52.937 [error] <0.1071.0> gen_fsm <0.1071.0> in state ready terminated with reason: {timeout,{gen_fsm,sync_send_all_state_event,[<0.1072.0>,stop]}}
2012-10-01 09:02:54.292 [error] <0.1071.0> CRASH REPORT Process <0.1071.0> with 0 neighbours exited with reason: {timeout,{gen_fsm,sync_send_all_state_event,[<0.1072.0>,stop]}} in gen_fsm:handle_msg/7 line 511


escript: exception error: no match of right hand side value {error,enoent}

escript: exception error: no match of right hand side value {error,enomem}


so changing to

15360 mem size since no enomem error from 12288(1024*12)

{js_thread_stack, 16}, changed from 16 mb


to 1024

Thanks & regards
sangeetha

This e-mail and any files transmitted with it are for the sole use of the intended recipient(s) and may contain confidential and privileged information. If you are not the intended recipient(s), please reply to the sender and destroy all copies of the original message. Any unauthorized review, use, disclosure, dissemination, forwarding, printing or copying of this email, and/or any action taken in reliance on the contents of this e-mail is strictly prohibited and may be unlawful.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.basho.com/pipermail/riak-users_lists.basho.com/attachments/20121003/81d1b4fe/attachment.html>


More information about the riak-users mailing list