Link walking with a java client

Deepak Balasubramanyam deepak.balu at gmail.com
Sat Jun 2 02:09:57 EDT 2012


Created bug 1341 <https://issues.basho.com/show_bug.cgi?id=1341> to track
this.

Can you help me with the questions below ? I am trying to gauge how
expensive linking is in comparison to using a secondary index to track a
node.

Thanks
Deepak

On Wed, May 30, 2012 at 9:42 AM, Deepak Balasubramanyam <
deepak.balu at gmail.com> wrote:

> Brian,
>
> Yes I read about the hack somewhere in your documentation. My
> understanding is that the link walking operation will cease to work via
> HTTP after links to a node grow beyond a particular number. This happens
> because a HTTP header is used to send link related data and there are
> limits around how much data a header can hold.
>
> I was interested in exploring if the PB client can overcome this
> limitation. The use case I am interested in is this - If a single node
> links to 10k nodes, how much time would it take a link walker to visit all
> nodes via the PB client ? What sort of client would you recommend for
> something like this ?
>
> I know that linking is only meant to be used as a lightweight feature.
> Would 10k links to a node be considered lightweight ?
>
> Thanks
> Deepak
>
> On Tue, May 29, 2012 at 9:33 PM, Brian Roach <roach at basho.com> wrote:
>
>> Deepak -
>>
>> I'll take a look at it this week, but more than likely it's a bug.
>>
>> Link walking is a REST only operation as far as Riak’s interfaces are
>> concerned. Link Walking in the protocol buffers Java client is a hack that
>> issues two m/r jobs to the protocol buffers interface (the first constructs
>> the inputs to the second by walking the links, the second returns the data).
>>
>> Thanks,
>> Brian Roach
>>
>> On May 27, 2012, at 7:19 AM, Deepak Balasubramanyam wrote:
>>
>> > This looks like a bug. The code to walk links via a HTTP client works
>> perfectly. The same code fails when the PB client is used. The POJO
>> attached in this email reproduces the problem.
>> >
>> > I searched the email archives and existing issues and found no trace of
>> this problem. Please run the POJO by swapping the clients returned from the
>> getClient() method to reproduce the problem. I can create a bug report once
>> someone from the dev team confirms this really is a bug.
>> >
>> > Riak client pom:
>> >       <dependency>
>> >               <groupId>com.basho.riak</groupId>
>> >               <artifactId>riak-client</artifactId>
>> >               <version>1.0.5</version>
>> >       </dependency>
>> >
>> > Riak server version - 1.1.2. Built from source.
>> > 4 nodes running on 1 machine. OS - Linux mint.
>> >
>> > On Sun, May 27, 2012 at 10:05 AM, Deepak Balasubramanyam <
>> deepak.balu at gmail.com> wrote:
>> > Hi,
>> >
>> > I have a cluster that contains 2 buckets. A bucket named 'usersMine'
>> contains the key 'user2', which is linked to several keys (about 10) under
>> a bucket named userPreferences. The relationship exists under the name
>> 'myPref'. A user and a preference have String values.
>> >
>> > I can successfully traverse the link over HTTP using the following URL -
>> >
>> > curl -v localhost:8091/riak/usersMine/user2/_,myPref,1
>> >
>> > ------------------------ ------------------------
>> ------------------------
>> > > User-Agent: curl/7.21.6 (i686-pc-linux-gnu) libcurl/7.21.6
>> OpenSSL/1.0.0e zlib/1.2.3.4 libidn/1.22 librtmp/2.3
>> > > Host: localhost:8091
>> > > Accept: */*
>> > >
>> > < HTTP/1.1 200 OK
>> > < Server: MochiWeb/1.1 WebMachine/1.9.0 (someone had painted it blue)
>> > < Expires: Sun, 27 May 2012 04:12:39 GMT
>> > < Date: Sun, 27 May 2012 04:02:39 GMT
>> > < Content-Type: multipart/mixed; boundary=IYGfKNqjGdco9ddfyjRP1Utzfi2
>> > < Content-Length: 3271
>> > <
>> >
>> > --IYGfKNqjGdco9ddfyjRP1Utzfi2
>> > Content-Type: multipart/mixed; boundary=3YVES0x2tFnUDOdTzfn1OGS6uMt
>> >
>> > --3YVES0x2tFnUDOdTzfn1OGS6uMt
>> > X-Riak-Vclock: a85hYGBgzGDKBVIcMRuuc/nvy7mSwZTImMfKUKpodpIvCwA=
>> > Location: /riak/userPreferences/preference3004
>> > Content-Type: text/plain; charset=UTF-8
>> > Link: </riak/userPreferences>; rel="up"
>> > Etag: 5GucnGSk4TjQc8BO1eNLyI
>> > Last-Modified: Sun, 27 May 2012 03:54:29 GMT
>> >
>> > junk
>> >
>> > << ------------------------ Truncated ------------------------ >>
>> > junk
>> > --3YVES0x2tFnUDOdTzfn1OGS6uMt--
>> >
>> > --IYGfKNqjGdco9ddfyjRP1Utzfi2--
>> > ------------------------ ------------------------
>> ------------------------
>> >
>> > However when I use the java client to walk the link, I get a
>> ClassCastException.
>> >
>> > ------------------------ ------------------------
>> ------------------------
>> > Java code:
>> > ------------------------ ------------------------
>> ------------------------
>> >     private void getAllLinks()
>> >     {
>> >         String user="user2";
>> >         IRiakClient riakClient = null;
>> >         try
>> >         {
>> >             long past = System.currentTimeMillis();
>> >             riakClient = RiakFactory.pbcClient("localhost",8081);
>> >             Bucket userBucket =
>> riakClient.fetchBucket("usersMine").execute();
>> >             DefaultRiakObject user1 =(DefaultRiakObject)
>> userBucket.fetch(user).execute();
>> >             List links = user1.getLinks();
>> >             System.out.println(links.size());
>> >             WalkResult execute =
>> riakClient.walk(user1).addStep("userPreferences", "myPref",true).execute();
>> >             Iterator iterator = execute.iterator();
>> >             while(iterator.hasNext())
>> >             {
>> >                 Object next = iterator.next();
>> >                 System.out.println(next);
>> >             }
>> >             long now = System.currentTimeMillis();
>> >             System.out.println("Retrieval in " + (now-past) + " ms");
>> >         }
>> >         catch (Exception e)
>> >         {
>> >             e.printStackTrace();
>> >         }
>> >         finally
>> >         {
>> >             if(riakClient != null)
>> >             {
>> >                 riakClient.shutdown();
>> >             }
>> >         }
>> >     }
>> >
>> > ------------------------ ------------------------
>> ------------------------
>> > Stack:
>> > ------------------------ ------------------------
>> ------------------------
>> > java.lang.ClassCastException: java.lang.String cannot be cast to
>> java.util.List
>> >       at
>> com.basho.riak.client.raw.pbc.PBClientAdapter.linkWalkSecondPhase(PBClientAdapter.java:380)
>> >       at
>> com.basho.riak.client.raw.pbc.PBClientAdapter.linkWalk(PBClientAdapter.java:325)
>> >       at com.basho.riak.client.query.LinkWalk.execute(LinkWalk.java:63)
>> >       at
>> com.chatterbox.persistence.riak.RiakTest.getAllLinks(RiakTest.java:81)
>> >       at com.chatterbox.persistence.riak.RiakTest.main(RiakTest.java:25)
>> > ------------------------ ------------------------
>> ------------------------
>> >
>> > At line 380 on PBClientAdapter, an enhanced for loop attempts to
>> iterate over a Collection<List<String>> while the actual value is a
>> List<String>. This causes the exception. The value of the variable 'step'
>> at the time of the exception is [userPreferences, preference3000, myPref].
>> >
>> > Am i doing something wrong ? The java code retrieves links in a fashion
>> that is similar to the HTTP retrieval, so I am puzzled by the ClassCast.
>> >
>> > Thanks
>> > Deepak
>> >
>> > <RiakTest.java>_______________________________________________
>> > riak-users mailing list
>> > riak-users at lists.basho.com
>> > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.basho.com/pipermail/riak-users_lists.basho.com/attachments/20120602/8f5ddd20/attachment.html>


More information about the riak-users mailing list