Implicit calls to ListKeys?

Kevin Burton rkevinburton at charter.net
Sun Feb 10 09:59:42 EST 2013


I can see that but there is something I am missing. I have code that
basically takes a subset of one bucket and copies it into another. Something
that Admittedly we will not plan to do in production. 

 

        #region GenerateJsonSubsetFromRiak

        private static void GenerateJsonSubsetFromRiak(int take = 1750, int
skip = 33250, string sourceBucketName = " products", string
destinationBucketName = "test", string path = @"C:\Users\Kevin\Test.txt")

        {

            using(IRiakEndPoint cluster =
RiakCluster.FromConfig("riakConfig"))

            {

                IRiakClient riakClient = cluster.CreateClient();

                RiakResult pingResult = riakClient.Ping();

                List<Shared.Types.Riak.Product> productList = new
List<Shared.Types.Riak.Product>();

                if (pingResult.IsSuccess)

                {

                    Debug.WriteLine(string.Format("Listing {0} bucket",
sourceBucketName));

                    RiakResult<IEnumerable<string>> listKeysResult =
riakClient.ListKeys(sourceBucketName);

                    if (!listKeysResult.IsSuccess) return;

                    int totalCount = listKeysResult.Value.Count();

                    if (skip > totalCount)

                    {

                        skip = (totalCount - take) > 0 ? (totalCount - take)
: 0;

                    }

                    Debug.WriteLine(string.Format("Extracting from {0}
bucket {1}/{2}/{3}", sourceBucketName, skip, take, totalCount));

                    foreach (var sku in listKeysResult.Value.OrderBy(s =>
s).Skip(skip).Take(take))

                    {

                        RiakResult<RiakObject> getSkuResult =
riakClient.Get(sourceBucketName, sku);

                        if (!getSkuResult.IsSuccess)

                        {

                            Debug.WriteLine(string.Format("SKU {0} not
found", sku));

                        }

                        else

                        {

                            RiakObject riakObject = getSkuResult.Value;

                            productList.Add(

 
JsonConvert.DeserializeObject<Shared.Types.Riak.Product>(

 
Encoding.UTF8.GetString(riakObject.Value)));

                        }

                    }

                    Debug.WriteLine(string.Format("Deleting {0} bucket",
destinationBucketName));

                    int count = 0;

                    foreach (RiakResult deleteResult in
riakClient.DeleteBucket(destinationBucketName))

                    {

                        if (!deleteResult.IsSuccess)

                        {

                            Debug.WriteLine(string.Format("{0} '{1}'",
deleteResult.ResultCode,

 
deleteResult.ErrorMessage));

                        }

                        else

                        {

                            count++;

                        }

                    }

                    Debug.WriteLine(string.Format("Deleted {0} bucket
containing {1} documents", destinationBucketName, count));

                    Debug.WriteLine(string.Format("Setting {0} bucket
properties", destinationBucketName));

                    RiakResult<RiakBucketProperties> getPropertyResult =
riakClient.GetBucketProperties(

                        sourceBucketName,

                       true);

                    if (getPropertyResult.IsSuccess)

                    {

                        //Copy source bucket properties to destination
bucket

                        RiakResult setPropertyResult =
riakClient.SetBucketProperties(destinationBucketName,

 
getPropertyResult.Value);

                        if (!setPropertyResult.IsSuccess)

                        {

                            Debug.WriteLine(string.Format("Setting bucket
properties failed - {0} - {1}",

 
setPropertyResult.ErrorMessage, setPropertyResult.ResultCode));

                            return;

                        }

                    }

                    else

                    {

                        Debug.WriteLine(string.Format("Getting bucket
properties failed - {0} - {1}",

 
getPropertyResult.ErrorMessage, getPropertyResult.ResultCode));

                        return;

                    }

                    Debug.WriteLine(string.Format("Copying from {0}/{1}/{2}
to {3}", sourceBucketName, skip, take, destinationBucketName));

                    List<RiakObject> copyDocumentlist = new
List<RiakObject>();

                    foreach (Shared.Types.Riak.Product product in
productList)

                    {

                        string keySku =
int.Parse(product.SKU).ToString("D8");

 

                        RiakObject value = new
RiakObject(destinationBucketName, keySku);

                        value.SetObject(product,
RiakConstants.ContentTypes.ApplicationJson);

                        if (!string.IsNullOrEmpty(product.ProductID))

 
value.BinIndex("ProductId").Set(product.ProductID);

                        if (!string.IsNullOrEmpty(product.ActiveInactive))

 
value.BinIndex("Active").Set(product.ActiveInactive);

                        if (!string.IsNullOrEmpty(product.DefinitionName))

 
value.BinIndex("Definition").Set(product.DefinitionName);

                        if (!string.IsNullOrEmpty(product.Department))

 
value.BinIndex("Department").Set(product.Department);

                        copyDocumentlist.Add(value);

                    }

                    var responses = riakClient.Put(copyDocumentlist, new
RiakPutOptions {W = 2});

                    if (responses.Any(r => !r.IsSuccess))

                    {

                        foreach (var response in responses)

                        {

                            if (!response.IsSuccess)

                            {

                                Debug.WriteLine(string.Format("Write to new
bucket of {0} items failed with '{1}'",

 
copyDocumentlist.Count, response.ErrorMessage));

                            }

                        }

                        return;

                    }

                }

                if (!string.IsNullOrEmpty(path))

                {

                    using (StreamWriter sw = new StreamWriter(path, false))

                    {

 
sw.WriteLine(JsonConvert.SerializeObject(productList.ToArray()));

                    }

                    Debug.WriteLine(string.Format("Wrote {0} objects to
'{1}'", productList.Count, path));

                }

            }

        }

        #endregion        

 

But you can see from this code that there is a Debug print statement between
each of the potential 'ListKeys' operations. Is what I am seeing is that
warning message multiple times after the ListKeys and DeleteBucket calls.
Yet each of these calls is only made once.

 

Kevin

 

From: OJ Reeves [mailto:oj at buffered.io] 
Sent: Sunday, February 10, 2013 4:06 AM
To: Kevin Burton
Cc: Alexander Moore; riak-users at lists.basho.com
Subject: Re: Implicit calls to ListKeys?

 

 

There must be another.

 

If you look at the code you'll see the ListKeys and StreamingListKeys are
the only two functions that make the call to write this debug output.
DeleteBucket has to list keys to know what to delete, so it calls ListKeys
first.

 

If you can provide a code sample that generates this output twice where
there is only one visible call I would be happy to fix the problem.

 

OJ

 

 

 

I am only calling ListKeys and DeleteBucket. Each once, so I expect to see
this warning twice. Which I do most of the time. But occasionally I see this
warning 2 or 3 more times.  Is it possible that there is some kind of
recursion going on that triggers this call?

 

From: Alexander Moore [mailto:moore.alex at gmail.com] 
Sent: Saturday, February 09, 2013 10:41 PM
To: Kevin Burton
Cc: riak-users at lists.basho.com
Subject: Re: Implicit calls to ListKeys?

 

Hey Kevin,

 

It looks like only RiakClient.ListKeys(), RiakClient.ListKeysStream(), and
RiakClient.DeleteBucket() will give you that warning. 
You shouldn't see it popup from anywhere else in CorrugatedIron. 


Thanks,

Alex

 

 

On Sat, Feb 9, 2013 at 11:19 PM, Kevin Burton <rkevinburton at charter.net>
wrote:

One more question I have some code that specifically lists the keys in a
bucket. I am using Corrugated Iron so I get the warning:

 

*** [CI] -> ListKeys is an expensive operation and should not be used in
Production scenarios. ***

 

Then because again this is a test/development environment I am explicitly
removing the bucket. I get this same warning again and from what I
understand this is to be expected.

So I run this code with different data sets and every once and a while I get
two or three of these warnings again. Other than ListKeys and DeleteBucket
what other calls in CorrugatedIron implicitly call ListKeys and would
trigger generating this warning again?

 


_______________________________________________
riak-users mailing list
riak-users at lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

 


_______________________________________________
riak-users mailing list
riak-users at lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com





 

-- 

OJ Reeves

+61 431 952 586
http://buffered.io/

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.basho.com/pipermail/riak-users_lists.basho.com/attachments/20130210/d71a725e/attachment.html>


More information about the riak-users mailing list