Slow performance on search-cmd set-schema

David Smith dizzyd at
Wed Jul 13 00:08:51 EDT 2011

On Tue, Jul 12, 2011 at 7:51 PM, Ken Perkins <ken at> wrote:
> What could be causing the delay on the rackspace VM? Is it purely a VM
> performance issue, or is there something else going on.

The blessing of the cloud: It hides the Hard Stuff.
The curse of the cloud: It hides the Hard Stuff. :)

First order of business -- how easy is it to reproduce? If you dump
the fs cache and spin up pre-populated nodes, do you get predictable
(if poor) latency? If you run a series of random queries, or even a
series of repeating queries, does it have the variance you expect?
Does it get better/worse?

Based on the limited info you stated, same data set, same
configuration, etc -- the only variable present seems to be the
virtualized system. 30-40 second delay is pretty crazy, though. Have
you confirmed that both deployments have similar CPU and I/O profiles?
My hunch is that you will see significant I/O wait on the slow one.
You might also consider doing a strace (or equivalent) on the slow
nodes and see what (if anything) is happening while that search is
going on.

I realize this is really just a bunch of questions, not an answer. :)
Diagnosing performance problems in a virtualized environment is
tricky, at best. Add to this the possibility that you may have a bunch
of nodes running on a single piece of hardware (competing for
resources and reliability) and things get really, really, hard to

Hope that helps...


Dave Smith
Director, Engineering
Basho Technologies, Inc.
dizzyd at

More information about the riak-users mailing list