Crash on Riak with Riak CS

Anita Wong anita.wong at talkboxapp.com
Tue Jan 28 05:26:24 EST 2014


I got this case happen again after a reboot.
I’ve installed Riak-CS-Control and it gave me “There’s nothing here."

Here’s the RiakCS error.log:
2014-01-28 15:18:07.165 [error] <0.278.0> Supervisor {<0.278.0>,poolboy_sup} had child riak_cs_riakc_pool_worker started with riak_cs_riakc_pool_worker:start_link([{name,{local,bucket_list_pool}},{worker_module,riak_cs_riakc_pool_worker},{size,5},{max_overflow,...},...]) at undefined exit with reason shutdown in context shutdown_error
2014-01-28 15:40:59.116 [error] <0.5693.0>@riak_cs_wm_common:maybe_create_user:223 Retrieval of user record for s3 failed. Reason: disconnected
2014-01-28 15:40:59.118 [error] <0.1717.0>@riak_cs_wm_common:maybe_create_user:223 Retrieval of user record for s3 failed. Reason: disconnected
2014-01-28 15:41:28.876 [error] <0.5799.0>@riak_cs_wm_common:maybe_create_user:223 Retrieval of user record for s3 failed. Reason: disconnected
2014-01-28 15:41:39.110 [error] <0.5808.0>@riak_cs_wm_common:maybe_create_user:223 Retrieval of user record for s3 failed. Reason: disconnected
2014-01-28 15:43:08.552 [error] <0.5946.0>@riak_cs_wm_common:maybe_create_user:223 Retrieval of user record for s3 failed. Reason: disconnected
2014-01-28 15:43:09.095 [error] <0.5949.0>@riak_cs_wm_common:maybe_create_user:223 Retrieval of user record for s3 failed. Reason: disconnected
2014-01-28 15:43:09.109 [error] <0.5804.0>@riak_cs_wm_common:maybe_create_user:223 Retrieval of user record for s3 failed. Reason: disconnected
2014-01-28 15:43:12.857 [error] <0.1897.0>@riak_cs_wm_common:maybe_create_user:223 Retrieval of user record for s3 failed. Reason: disconnected
2014-01-28 16:22:24.576 [error] <0.5809.0>@riak_cs_wm_common:maybe_create_user:223 Retrieval of user record for s3 failed. Reason: disconnected
2014-01-28 16:22:25.179 [error] <0.1706.0>@riak_cs_wm_common:maybe_create_user:223 Retrieval of user record for s3 failed. Reason: disconnected
2014-01-28 16:22:25.665 [error] <0.6721.0>@riak_cs_wm_common:maybe_create_user:223 Retrieval of user record for s3 failed. Reason: disconnected
2014-01-28 16:24:00.806 [error] <0.6806.0>@riak_cs_wm_common:maybe_create_user:223 Retrieval of user record for s3 failed. Reason: disconnected
2014-01-28 16:30:20.111 [error] <0.1225.0>@riak_cs_wm_common:maybe_create_user:223 Retrieval of user record for s3 failed. Reason: disconnected
2014-01-28 16:30:34.803 [error] <0.13918.0>@riak_cs_wm_common:maybe_create_user:223 Retrieval of user record for s3 failed. Reason: disconnected
2014-01-28 16:30:43.276 [error] <0.14230.0>@riak_cs_wm_common:maybe_create_user:223 Retrieval of user record for s3 failed. Reason: disconnected
2014-01-28 16:30:43.464 [error] <0.13882.0>@riak_cs_wm_common:maybe_create_user:223 Retrieval of user record for s3 failed. Reason: disconnected
2014-01-28 16:30:45.200 [error] <0.15516.0>@riak_cs_wm_common:maybe_create_user:223 Retrieval of user record for s3 failed. Reason: disconnected
2014-01-28 16:30:47.274 [error] <0.5618.0>@riak_cs_wm_common:maybe_create_user:223 Retrieval of user record for s3 failed. Reason: disconnected
2014-01-28 16:30:48.967 [error] <0.15536.0>@riak_cs_wm_common:maybe_create_user:223 Retrieval of user record for s3 failed. Reason: disconnected
2014-01-28 16:32:39.880 [error] <0.13889.0>@riak_cs_wm_common:maybe_create_user:223 Retrieval of user record for s3 failed. Reason: disconnected
2014-01-28 16:37:20.765 [error] <0.6859.0>@riak_cs_wm_common:maybe_create_user:223 Retrieval of user record for s3 failed. Reason: disconnected
2014-01-28 16:38:17.226 [error] <0.15498.0>@riak_cs_wm_common:maybe_create_user:223 Retrieval of user record for s3 failed. Reason: disconnected
2014-01-28 16:40:12.524 [error] <0.15910.0>@riak_cs_wm_common:maybe_create_user:223 Retrieval of user record for s3 failed. Reason: disconnected
2014-01-28 16:40:54.307 [error] <0.1197.0> Supervisor {<0.1197.0>,poolboy_sup} had child riak_cs_riakc_pool_worker started with riak_cs_riakc_pool_worker:start_link([{name,{local,bucket_list_pool}},{worker_module,riak_cs_riakc_pool_worker},{size,5},{max_overflow,...},...]) at undefined exit with reason shutdown in context shutdown_error
2014-01-28 17:45:12.476 [error] <0.8154.0>@riak_cs_wm_common:maybe_create_user:223 Retrieval of user record for s3 failed. Reason: disconnected
2014-01-28 17:45:25.109 [error] <0.8154.0>@riak_cs_wm_common:maybe_create_user:223 Retrieval of user record for s3 failed. Reason: disconnected
2014-01-28 17:48:59.176 [error] <0.11530.0>@riak_cs_wm_utils:validate_auth_header:215 Retrieval of user record for "admin-key" failed. Reason: disconnected
2014-01-28 17:49:05.062 [error] <0.13507.0>@riak_cs_wm_utils:validate_auth_header:215 Retrieval of user record for "admin-key" failed. Reason: disconnected
2014-01-28 17:49:25.780 [error] <0.268.0> Supervisor {<0.268.0>,poolboy_sup} had child riak_cs_riakc_pool_worker started with riak_cs_riakc_pool_worker:start_link([{name,{local,bucket_list_pool}},{worker_module,riak_cs_riakc_pool_worker},{size,5},{max_overflow,...},...]) at undefined exit with reason shutdown in context shutdown_error
2014-01-28 17:50:07.941 [error] <0.288.0>@riak_cs_wm_utils:validate_auth_header:215 Retrieval of user record for "admin-key" failed. Reason: <<"{pr_val_unsatisfied,1,0}">>
2014-01-28 18:02:23.048 [error] <0.286.0>@riak_cs_wm_utils:validate_auth_header:215 Retrieval of user record for "admin-key" failed. Reason: disconnected

Stanchion:
2014-01-28 18:02:18.892 [error] <0.121.0> gen_server <0.121.0> terminated with reason: disconnected
2014-01-28 18:02:18.892 [error] <0.121.0> CRASH REPORT Process <0.121.0> with 0 neighbours exited with reason: disconnected in gen_server:terminate/6 line 747

Anita WONG
Senior System Engineer

let's try TalkBox. my username is "anita"

TalkBox Limited | www.talkboxapp.com
D: +852.3728.2452  | M: +852.9821.5819 
E: anita.wong at talkboxapp.com
Facebook: TalkBox App | Twitter: TalkBoxApp | Weibo: TalkBox

On 31 Dec, 2013, at 12:46 am, Seth Thomas <sthomas at basho.com> wrote:

> Anita,
> 
> It sounds like there are two potentially separate issues here. 
> 
> For the first item related to Riak CS - was there already an admin user created? are all (mb, rb, put, get)  operations returning the 403 "The AWS Access Key Id you provided does not exist in our records." error? The console logs from Riak CS or Stanchion may reveal more relevant detail. I'd be happy to look over the configs and logs for you if you want to zip them all up and send them directly.
> 
> The invalid hint files should not prevent Riak from returning requests so might be a red herring with regard to the above error.
> 
> Cheers,
> 
> 
> 
> 
> On Mon, Dec 30, 2013 at 2:03 AM, Anita Wong <anita.wong at talkboxapp.com> wrote:
> Hi,
> 
> I’m running RIak with Riak-CS for some time already. 
> 
> Today when I tried to access it, I found that the it returns 
> "The AWS Access Key Id you provided does not exist in our records."
> 
> I tried to use Riak-CS Control to look at the user list, but it returned that there’s no data, even if I called 
> curl -H 'Content-Type: application/json' \
>   -X POST http://localhost:8080/riak-cs/user \
>   --data '{"email":"foobar at example.com", "name":"admin user"}'
> as suggested from the website.
> 
> The objects uploaded directly to Riak (without Riak-CS) can still be accessed, only those through RIak-CS failed.
> 
> I also found crash.log in RIak, Stanchion and Riak-CS, the one in Riak seems to be the problem:
> 2013-12-30 15:36:38 =ERROR REPORT====
> Hintfile '/vol1/riak/bitcask/799258707915337533392640142891717276374338109440/1.bitcask.hint' invalid
> 2013-12-30 15:36:39 =ERROR REPORT====
> Hintfile '/vol1/riak/bitcask/0/1.bitcask.hint' invalid
> 2013-12-30 15:36:39 =ERROR REPORT====
> Hintfile '/vol1/riak/bitcask/91343852333181432387730302044767688728495783936/1.bitcask.hint' invalid
> 2013-12-30 15:36:39 =ERROR REPORT====
> Hintfile '/vol1/riak/bitcask/319703483166135013357056057156686910549735243776/1.bitcask.hint' invalid
> 2013-12-30 15:36:39 =ERROR REPORT====
> Hintfile '/vol1/riak/bitcask/182687704666362864775460604089535377456991567872/1.bitcask.hint' invalid
> 2013-12-30 15:36:39 =ERROR REPORT====
> Hintfile '/vol1/riak/bitcask/45671926166590716193865151022383844364247891968/1.bitcask.hint' invalid
> 2013-12-30 15:36:39 =ERROR REPORT====
> Hintfile '/vol1/riak/bitcask/296867520082839655260123481645494988367611297792/1.bitcask.hint' invalid
> 2013-12-30 15:36:39 =ERROR REPORT====
> Hintfile '/vol1/riak/bitcask/365375409332725729550921208179070754913983135744/1.bitcask.hint' invalid
> 2013-12-30 15:36:39 =ERROR REPORT====
> Hintfile '/vol1/riak/bitcask/159851741583067506678528028578343455274867621888/1.bitcask.hint' invalid
> 2013-12-30 15:36:39 =ERROR REPORT====
> ….
> 
> Any suggestion on how to solve or prevent this? Thanks a lot.
> 
> Anita WONG
> 
> 
> _______________________________________________
> riak-users mailing list
> riak-users at lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
> 
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.basho.com/pipermail/riak-users_lists.basho.com/attachments/20140128/e58c4596/attachment.html>


More information about the riak-users mailing list