Slow performance on search-cmd set-schema

David Smith dizzyd at basho.com
Wed Jul 13 00:08:51 EDT 2011


On Tue, Jul 12, 2011 at 7:51 PM, Ken Perkins <ken at clipboard.com> wrote:
> What could be causing the delay on the rackspace VM? Is it purely a VM
> performance issue, or is there something else going on.

The blessing of the cloud: It hides the Hard Stuff.
The curse of the cloud: It hides the Hard Stuff. :)

First order of business -- how easy is it to reproduce? If you dump
the fs cache and spin up pre-populated nodes, do you get predictable
(if poor) latency? If you run a series of random queries, or even a
series of repeating queries, does it have the variance you expect?
Does it get better/worse?

Based on the limited info you stated, same data set, same
configuration, etc -- the only variable present seems to be the
virtualized system. 30-40 second delay is pretty crazy, though. Have
you confirmed that both deployments have similar CPU and I/O profiles?
My hunch is that you will see significant I/O wait on the slow one.
You might also consider doing a strace (or equivalent) on the slow
nodes and see what (if anything) is happening while that search is
going on.

I realize this is really just a bunch of questions, not an answer. :)
Diagnosing performance problems in a virtualized environment is
tricky, at best. Add to this the possibility that you may have a bunch
of nodes running on a single piece of hardware (competing for
resources and reliability) and things get really, really, hard to
predict.

Hope that helps...

D.

-- 
Dave Smith
Director, Engineering
Basho Technologies, Inc.
dizzyd at basho.com




More information about the riak-users mailing list