Innostore crash - force recovery not working via innodump

Rudy X. Desjardins rudy at sandbenders.ca
Fri Nov 19 16:26:44 EST 2010


Hello,

I'm using Riak in a fairly high-volume scenario and one of my nodes
has crashed... I finally found a half-decent reference to innodb crash
recovery and what the force_recovery levels mean (here, if anyone's
interested: http://www.trickykid.org/article/mysql-forcing-recovery
... dev.mysql.com also has a decent explanation, although it's in the
context of using the force_recovery option to allow re-starting a
mysql server vs. fixing the actual innodb tables/files)...

I've tried to run innodump (from /priv in the innostore src/build dir)
to dump the data with no luck - I get the same error message (first
one, below) as when I just try to re-start the Riak node, unless I use
-force_recovery 6 (all other levels produce original error), which
produces the second error below.

Any ideas how I might go about fixing this, or should I assume that
node/data is dead/lost and just re-initialize the innostore for this
node from scratch?

Thanks! (errors follow)

--- first error:

InnoDB: Mutexes and rw_locks use GCC atomic builtins
101119 21:24:14  InnoDB: highest supported file format is Barracuda.
InnoDB: Log scan progressed past the checkpoint lsn 3119545955
101119 21:24:14 InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
InnoDB: Error: tried to read 16384 bytes at offset 0 14680064.
InnoDB: Was only able to read 12288.
InnoDB: Fatal error: cannot read from file. OS error number 17.
101119 21:24:14  InnoDB: Assertion failure in thread 3030391696 in
file os/os0file.c line 2520
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report, check the InnoDB website for details
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the server startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: the InnoDB website for details
InnoDB: about forcing recovery.

--- second error:

InnoDB: Mutexes and rw_locks use GCC atomic builtins
101119 21:24:46  InnoDB: highest supported file format is Barracuda.
InnoDB: The user has set IB_RECOVERY_NO_LOG_REDO on
InnoDB: Skipping log redo
101119 21:24:46 Embedded InnoDB 1.0.6.6750 started; log sequence number 0
InnoDB: !!! force_recovery is set to 6 !!!
101119 21:24:46  InnoDB: error: space object of
table'innokeystore/click-conversions_0',
InnoDB: space id 50 did not exist in memory. Retrying an open.
101119 21:24:46  InnoDB: Assertion failure in thread 3035970448 in
file fil/fil0fil.c line 3057
InnoDB: Failing assertion: flags != DICT_TF_COMPACT
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report, check the InnoDB website for details
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the server startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: the InnoDB website for details
InnoDB: about forcing recovery.

-- 
Rudy X. Desjardins
pgp: 96FFA628




More information about the riak-users mailing list