Ok, I am stumped. Losing data or riak stop

Joe Olson technology at nododos.com
Fri Feb 26 11:19:34 EST 2016



Negative. 

I have ring size set to 8, leveldb split across two sets of drives ("fast" and "slow", but meaningless on the test Vagrant box...just two separate directories). I checked all of the ../leveldb/* directories. All LOG files are identical, and no errors in any of them. 

I will try to build another Vagrant machine with the default riak.conf and see if I can get this to repeat. It is almost as if the KV pairs are not persisting to disk at all. 



From: "Matthew Von-Maszewski" <matthewv at basho.com> 
To: "Joe Olson" <technology at nododos.com> 
Cc: "riak-users" <riak-users at lists.basho.com>, "cmancini" <cmancini at basho.com> 
Sent: Friday, February 26, 2016 10:12:15 AM 
Subject: Re: Ok, I am stumped. Losing data or riak stop 

Joe, 

Are there any error messages in the leveldb LOG and/or LOG.old files? These files are located within each vnode's directory, likely /var/lib/riak/data/leveldb/*/LOG* on your machine. 

The LOG files are not to be confused with 000xxx.log files. The lower case *.log files are the recovery files that should contain the keys you are missing. If they are not loading properly, the LOG files should have clues. 

Matthew 




On Feb 26, 2016, at 11:04 AM, Christopher Mancini < cmancini at basho.com > wrote: 

Hey Joe, 

I will do my best to help, but I am not the most experienced with Riak operations. Your best bet to get to a solution as fast as possible is to include the full users group, which I have added to the recipients of this message. 

1. Are the Riak data directories within Vagrant shared directories between the host and guest? I have had issues with OS file system caching before when working with web server files. 

2. What version of Ubuntu are you using? 

3. How did you install Riak on Ubuntu? 

4. Have you tried restoring the original distribution riak.conf file and seen if the issue persists? This would help you determine if the issue is your config or something with your environment. 

Chris 

On Fri, Feb 26, 2016 at 10:55 AM Joe Olson < technology at nododos.com > wrote: 

BQ_BEGIN


Chris - 

I cannot figure out what is going on. Here is my test case. Configuration file attached. I am running a single node of Riak on a vagrant box with a level DB back end. I don't even have to bring the box down, merely stopping and restarting riak '(riak stop' and 'riak start' or 'risk restart) causes all the keys to be lost. The riak node is set up on a Vagrant box. But again....I do not have to bring the machine up or down to get this error. 

I've also deleted the ring info in /var/lib/riak/ring, and deleted all the leveldb files. In this case, the bucket type is just n_val = 1, and the ring size is the minimum of 8. 

Is it possible Riak is not flushing RAM to disk after write? The keys only reside in RAM? 

My test procedure: 

====On a remote machine===== 

riak01 at ubuntu:/etc$ curl -i http:// <ip>:8098/types/n1/buckets/test/keys?keys=true 
HTTP/1.1 200 OK 
Vary: Accept-Encoding 
Server: MochiWeb/1.1 WebMachine/1.10.8 (that head fake, tho) 
Date: Fri, 26 Feb 2016 13:14:59 GMT 
Content-Type: application/json 
Content-Length: 17 

{"keys":["test"]} 

riak01 at ubuntu:/etc$ 



====On the single Riak node itself==== 

[vagrant at i- 2016022519 -9bb5c84f riak]$ sudo riak stop 
ok 
[vagrant at i- 2016022519 -9bb5c84f riak]$ sudo riak start 
[vagrant at i- 2016022519 -9bb5c84f riak]$ sudo riak ping 
pong 



====Back to the remote machine==== 

riak01 at ubuntu:/etc$ curl -i http:// <ip>:8098/types/n1/buckets/test/keys?keys=true 
HTTP/1.1 200 OK 
Vary: Accept-Encoding 
Server: MochiWeb/1.1 WebMachine/1.10.8 (that head fake, tho) 
Date: Fri, 26 Feb 2016 13:16:34 GMT 
Content-Type: application/json 
Content-Length: 11 

{"keys":[]} 

riak01 at ubuntu:/etc$ 






-- 
Sincerely, 

Christopher Mancini 
----------------------------- 

employee = { 
purpose: solve problems with code, 
phone: 7164625591, 
email: cmancini at basho.com , 
github: http://www.github.com/christophermancini 
} 
_______________________________________________ 
riak-users mailing list 
riak-users at lists.basho.com 
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com 

BQ_END



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.basho.com/pipermail/riak-users_lists.basho.com/attachments/20160226/83b63403/attachment-0002.html>


More information about the riak-users mailing list