<html>
  <head>
    <meta content="text/html; charset=ISO-8859-1"
      http-equiv="Content-Type">
  </head>
  <body bgcolor="#FFFFFF" text="#000000">
    Also, which I forgot on my reply, make sure your Riak client is
    connected to each node and not only to a single node (cluster config
    doesn't work that well, so try haproxy and make sure you are using
    protocol buffers)<br>
    <br>
    <i>HA proxy sample config:</i> <a class="moz-txt-link-freetext" href="https://gist.github.com/gburd/1507077">https://gist.github.com/gburd/1507077</a><br>
    <br>
    And a single PB config like this one which will connect HA proxy
    load balancer assuming it is running on localhost and it is
    connected to each node:<br>
    <i><br>
      final PBClientConfig clientConfig=new
      PBClientConfig.Builder().withHost("127.0.0.1").withPort(8087).withPoolSize(N).build();</i><br>
    <br>
    Guido.<br>
    <br>
    <div class="moz-cite-prefix">On 13/02/13 10:29, Guido Medina wrote:<br>
    </div>
    <blockquote cite="mid:511B6B15.1090307@temetra.com" type="cite">
      <meta content="text/html; charset=ISO-8859-1"
        http-equiv="Content-Type">
      Are you transferring using a single thread? If so, I would
      recommend you to use a ThreaPoolExecutor and schedule each write
      as you, control the failures (if any) using either an
      AtomicInteger or a concurrent/synchronized list where you can
      track the keys that failed.<br>
      <br>
      No matter how much you do, a single threaded transfer won't help
      you at all. We have done transfers many times and depending on the
      size of the DB table, we use single thread or thread pool service.
      Try 8 threads and see the difference, assuming you have N
      connections in your Riak client where N>max thread pool size.<br>
      <br>
      You might want to remove pw=1 when using multi-threading so Riak
      doesn't fallback behind too much (elevel db catch up? whatever
      that's called), pw=1 will add more risk than the benefit you gain.<br>
      <br>
      Hope that helps,<br>
      <br>
      Guido.<br>
      <br>
      <div class="moz-cite-prefix">On 13/02/13 09:44, Bogdan Flueras
        wrote:<br>
      </div>
      <blockquote
cite="mid:CADKC62oXvFwh1N6ib4JKMuejdQc3Pe4Hw0Qi3=hs8wC4rU1D5A@mail.gmail.com"
        type="cite">
        <div dir="ltr">
          <div>Ok, so I've done something like this:<br>
          </div>
          Bucket bucket = client.createBucket("foo"); //
          lastWriteWins(true) doesn't work for Protobuf<br>
          <div>
            <div>
              <div><br>
                when I insert I have:<br>
              </div>
              <div>bucket.store(someKey,
                someValue).withoutFetch().pw(1).execute();<br>
                <br>
              </div>
              <div>It looks like it's 20% faster than before. Is there
                something I could further tweak ?<br>
              </div>
            </div>
          </div>
        </div>
        <div class="gmail_extra"> <br clear="all">
          <div>ing. Bogdan Flueras<br>
            <br>
          </div>
          <br>
          <br>
          <div class="gmail_quote">On Wed, Feb 13, 2013 at 10:19 AM,
            Bogdan Flueras <span dir="ltr"><<a
                moz-do-not-send="true"
                href="mailto:flueras.bogdan@gmail.com" target="_blank">flueras.bogdan@gmail.com</a>></span>
            wrote:<br>
            <blockquote class="gmail_quote" style="margin:0 0 0
              .8ex;border-left:1px #ccc solid;padding-left:1ex">
              <div dir="ltr">
                <div>Each thread has it's own bucket instance (pointing
                  to the same location) and I don't re-fetch the bucket
                  per insert.<br>
                </div>
                Thank you very much!<br>
              </div>
              <div class="gmail_extra"><br clear="all">
                <div>ing. Bogdan Flueras<br>
                  <br>
                </div>
                <div>
                  <div class="h5"> <br>
                    <br>
                    <div class="gmail_quote">On Wed, Feb 13, 2013 at
                      10:14 AM, Russell Brown <span dir="ltr"><<a
                          moz-do-not-send="true"
                          href="mailto:russell.brown@me.com"
                          target="_blank">russell.brown@me.com</a>></span>
                      wrote:<br>
                      <blockquote class="gmail_quote" style="margin:0 0
                        0 .8ex;border-left:1px #ccc
                        solid;padding-left:1ex">
                        <div><br>
                          On 13 Feb 2013, at 08:07, Bogdan Flueras <<a
                            moz-do-not-send="true"
                            href="mailto:flueras.bogdan@gmail.com"
                            target="_blank">flueras.bogdan@gmail.com</a>>

                          wrote:<br>
                          <br>
                          > How to set the bucket to last write? Is
                          it in the builder?<br>
                          <br>
                        </div>
                        Something like:<br>
                        <br>
                            Bucket b =  
                        client.createBucket("my_bucket").lastWriteWins(true);<br>
                        <br>
                        Also, after you've created the bucket, do you
                        use it from all threads? You don't re-fetch the
                        bucket per-insert operation, do you?<br>
                        <br>
                        But  the "withoutFecth()" option is probably
                        going to be the biggest performance increase,
                        and safe if you are only doing inserts.<br>
                        <br>
                        Cheers<br>
                        <span><font color="#888888"><br>
                            Russell<br>
                          </font></span>
                        <div>
                          <div><br>
                            > I'll have a look..<br>
                            > Yes, I use more threads and the bucket
                            is configured to spread the load across all
                            nodes.<br>
                            ><br>
                            > Thanks, I'll have a deeper look into
                            the API and let you know about my results.<br>
                            ><br>
                            > ing. Bogdan Flueras<br>
                            ><br>
                            ><br>
                            ><br>
                            > On Wed, Feb 13, 2013 at 10:02 AM,
                            Russell Brown <<a moz-do-not-send="true"
                              href="mailto:russell.brown@me.com"
                              target="_blank">russell.brown@me.com</a>>

                            wrote:<br>
                            > Hi,<br>
                            ><br>
                            > On 13 Feb 2013, at 07:37, Bogdan
                            Flueras <<a moz-do-not-send="true"
                              href="mailto:flueras.bogdan@gmail.com"
                              target="_blank">flueras.bogdan@gmail.com</a>>

                            wrote:<br>
                            ><br>
                            > > Hello all,<br>
                            > > I've got a 5 node cluster with
                            Riak 1.2.1, all machines are multicore,<br>
                            > > with min 4GB RAM.<br>
                            > ><br>
                            > > I want to insert something like 50
                            million records in Riak with the java client
                            (Protobuf used) with default settings.  I've
                            tried also with HTTP protocol and set w = 1
                            but got some problems.<br>
                            > ><br>
                            > > However the process is very slow:
                            it doesn't write more than 6GB/ hour or
                            aprox. 280 KB/second.<br>
                            > > To have all my data filled in, it
                            would take aprox 2 days !!<br>
                            > ><br>
                            > > What can I do to have the data
                            filled into Riak ASAP?<br>
                            > > How should I configure the cluster
                            ? (vm.args/ app.config) I don't care so much
                            about consistency at this point.<br>
                            ><br>
                            > If you are certain to be only inserting
                            new data setting your bucket(s) to last
                            write wins will speed things up. Also, are
                            you using multiple threads for the Java
                            client insert? Spreading the load across all
                            five nodes? Are you using the
                            "withoutFetch()" option on the java client?<br>
                            ><br>
                            > Cheers<br>
                            ><br>
                            > Russell<br>
                            ><br>
                            > ><br>
                            > > Thank you,<br>
                            > > ing. Bogdan Flueras<br>
                            > ><br>
                            > >
                            _______________________________________________<br>
                            > > riak-users mailing list<br>
                            > > <a moz-do-not-send="true"
                              href="mailto:riak-users@lists.basho.com"
                              target="_blank">riak-users@lists.basho.com</a><br>
                            > > <a moz-do-not-send="true"
href="http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com"
                              target="_blank">http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com</a><br>
                            ><br>
                            ><br>
                            <br>
                          </div>
                        </div>
                      </blockquote>
                    </div>
                    <br>
                  </div>
                </div>
              </div>
            </blockquote>
          </div>
          <br>
        </div>
        <br>
        <fieldset class="mimeAttachmentHeader"></fieldset>
        <br>
        <pre wrap="">_______________________________________________
riak-users mailing list
<a moz-do-not-send="true" class="moz-txt-link-abbreviated" href="mailto:riak-users@lists.basho.com">riak-users@lists.basho.com</a>
<a moz-do-not-send="true" class="moz-txt-link-freetext" href="http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com">http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com</a>
</pre>
      </blockquote>
      <br>
    </blockquote>
    <br>
  </body>
</html>