記一次血淋淋的MySQL崩潰修復案例
摘要：今天給大家帶來一篇MySQL數據庫崩潰的修復案例
本文分享自華為雲社區《記一次MySQL崩潰修復案例，再也不用刪庫跑路了》，作者： 冰 河。
問題描述
研究MySQL源代碼，調試並壓測MySQL源代碼時，MySQL崩潰了！問題是它竟然崩潰了！而且還損壞了InnoDB文件！！還好是在調試環境下發生的，趕緊看看如何解決這個問題，經過一系列的查閱資料、驗證、對比、MySQL源碼調試跟蹤、修復損壞的InnoDB文件、總結等流程，整理成此文，如果以後真的發生在線上的生產壞境，也不用擔心是不是要跑路的問題了，可以分分鐘搞定MySQL的崩潰問題了！！查看錯誤日誌，如下：
----------------------------------------- 161108 23:36:45 mysqld_safe Starting mysqld daemon with databases from /usr/local/mysql/var 2022-08-25 23:36:46 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details). 2022-08-25 23:36:46 5497 [Note] Plugin 'FEDERATED' is disabled. 2022-08-25 23:36:46 7f11c48e1720 InnoDB: Warning: Using innodb_additional_mem_pool_size is DEPRECATED. This option may be removed in future releases, together with the option innodb_use_sys_malloc and with the InnoDB's internal memory allocator. 2022-08-25 23:36:46 5497 [Note] InnoDB: Using atomics to ref count buffer pool pages 2022-08-25 23:36:46 5497 [Note] InnoDB: The InnoDB memory heap is disabled 2022-08-25 23:36:46 5497 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2022-08-25 23:36:46 5497 [Note] InnoDB: Memory barrier is not used 2022-08-25 23:36:46 5497 [Note] InnoDB: Compressed tables use zlib 1.2.3 2022-08-25 23:36:46 5497 [Note] InnoDB: Using CPU crc32 instructions 2022-08-25 23:36:46 5497 [Note] InnoDB: Initializing buffer pool, size = 16.0M 2022-08-25 23:36:46 5497 [Note] InnoDB: Completed initialization of buffer pool InnoDB: Database page corruption on disk or a failed InnoDB: file read of page 5. InnoDB: You may have to recover from a backup. 2022-08-25 23:36:46 7f11c48e1720 InnoDB: Page dump in ascii and hex (16384 bytes): len 16384; hex 7478d078000000050000000000000000000000000f271f4d000700000000000000000000000000000000001b4000000000000000000200f20000000000000006000000000000002d000000000000002e000000000000002f0000000000000030000000000（省略很多類似代碼） InnoDB: End of page dump 2022-08-25 23:36:46 7f11c48e1720 InnoDB: uncompressed page, stored checksum in field1 1954074744, calculated checksums for field1: crc32 993334256, innodb 2046145943, none 3735928559, stored checksum in field2 1139795846, calculated checksums for field2: crc32 993334256, innodb 1606613742, none 3735928559, page LSN 0 254222157, low 4 bytes of LSN at page end 254221236, page number (if stored to page already) 5, space id (if created with >= MySQL-4.1.1 and stored already) 0 InnoDB: Page may be a transaction system page InnoDB: Database page corruption on disk or a failed InnoDB: file read of page 5. InnoDB: You may have to recover from a backup. InnoDB: It is also possible that your operating InnoDB: system has corrupted its own file cache InnoDB: and rebooting your computer removes the InnoDB: error. InnoDB: If the corrupt page is an index page InnoDB: you can also try to fix the corruption InnoDB: by dumping, dropping, and reimporting InnoDB: the corrupt table. You can use CHECK InnoDB: TABLE to scan your table for corruption. InnoDB: See also http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html InnoDB: about forcing recovery. InnoDB: Ending processing because of a corrupt database page. 2022-08-25 23:36:46 7f11c48e1720 InnoDB: Assertion failure in thread 139714288817952 in file buf0buf.cc line 4201 InnoDB: We intentionally generate a memory trap. InnoDB: Submit a detailed bug report to http://bugs.mysql.com. InnoDB: If you get repeated assertion failures or crashes, even InnoDB: immediately after the mysqld startup, there may be InnoDB: corruption in the InnoDB tablespace. Please refer to InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html InnoDB: about forcing recovery. 03:36:46 UTC - mysqld got signal 6 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. key_buffer_size=16777216 read_buffer_size=262144 max_used_connections=0 max_threads=1000 thread_count=0 connection_count=0 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 798063 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x0 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0 thread_stack 0x40000 /usr/local/mysql/bin/mysqld(my_print_stacktrace+0x35)[0x8e64b5] /usr/local/mysql/bin/mysqld(handle_fatal_signal+0x41b)[0x652fbb] /lib64/libpthread.so.0(+0xf7e0)[0x7f11c44c77e0] /lib64/libc.so.6(gsignal+0x35)[0x7f11c315d625] /lib64/libc.so.6(abort+0x175)[0x7f11c315ee05] /usr/local/mysql/bin/mysqld[0xa585c5] /usr/local/mysql/bin/mysqld[0xa6c7b4] /usr/local/mysql/bin/mysqld[0xa6cbc7] /usr/local/mysql/bin/mysqld[0xa5bce2] /usr/local/mysql/bin/mysqld[0xa1e2ba] /usr/local/mysql/bin/mysqld[0xa0bf60] /usr/local/mysql/bin/mysqld[0x95a427] /usr/local/mysql/bin/mysqld(_Z24ha_initialize_handlertonP13st_plugin_int+0x48)[0x58f788] /usr/local/mysql/bin/mysqld[0x6e4a36] /usr/local/mysql/bin/mysqld(_Z11plugin_initPiPPci+0xb3e)[0x6e826e] /usr/local/mysql/bin/mysqld[0x582d85] /usr/local/mysql/bin/mysqld(_Z11mysqld_mainiPPc+0x4d8)[0x587d18] /lib64/libc.so.6(__libc_start_main+0xfd)[0x7f11c3149d5d] /usr/local/mysql/bin/mysqld[0x57a019] The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. 161108 23:36:46 mysqld_safe mysqld from pid file /usr/local/mysql/var/VM_241_49_centos.pid ended ------------------------------------------------------------------------------
問題分析
從日誌中可以看出是innodb引擎出了問題。日誌里提示到 //dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html查看強制恢復的方法。在mysql的配置文件my.cnf里找到 [mysqld]字段下，添加 innodb_force_recovery=1：
[mysqld] innodb_force_recovery = 1
如果innodb_force_recovery = 1不生效，則可嘗試2——6幾個數字
然後重啟mysql，重啟成功。然後使用mysqldump或 pma 導出數據，執行修復操作等。修復完成後，把該參數注釋掉，還原默認值0。
配置文件的參數：innodb_force_recovery
innodb_force_recovery影響整個InnoDB存儲引擎的恢復狀況。默認為0，表示當需要恢復時執行所有的恢復操作（即校驗數據頁/purge undo/insert buffer merge/rolling back&forward），當不能進行有效的恢復操作時，mysql有可能無法啟動，並記錄錯誤日誌；
innodb_force_recovery可以設置為1-6,大的數字包含前面所有數字的影響。當設置參數值大於0後，可以對錶進行select,create,drop操作,但insert,update或者delete這類操作是不允許的。
- (SRV_FORCE_IGNORE_CORRUPT):忽略檢查到的corrupt頁。
- (SRV_FORCE_NO_BACKGROUND):阻止主線程的運行，如主線程需要執行full purge操作，會導致crash。
- (SRV_FORCE_NO_TRX_UNDO):不執行事務回滾操作。
- (SRV_FORCE_NO_IBUF_MERGE):不執行插入緩衝的合併操作。
- (SRV_FORCE_NO_UNDO_LOG_SCAN):不查看重做日誌，InnoDB存儲引擎會將未提交的事務視為已提交。
- (SRV_FORCE_NO_LOG_REDO):不執行前滾的操作。
解決方案
一般修復方法參考：
第一種方法
建立一張新表：
create table demo_bak #和原表結構一樣，只是把INNODB改成了MYISAM。
把數據導進去
insert into demo_bak select * from demo;
刪除掉原表：
drop table demo;
注釋掉 innodb_force_recovery 之後，重啟。
重命名：
rename table demo_bak to demo;
最後改回存儲引擎：
alter table demo engine = innodb
第二種方法
另一個方法是使用mysqldump將表格導出，然後再導回到InnoDB表中。這兩種方法的結果是相同的。
備份導出(包括結構和數據)：
mysqldump -uroot -p123 test > test.sql
還原方法1：
use test;
source test.sql
還原方法2（系統命令行）：
mysql -uroot -p123 test < test.sql;
注意，CHECK TABLE命令在InnoDB數據庫中基本上是沒有用的。
第三種方法
1、配置my.cnf
配置innodb_force_recovery = 1或2——6幾個數字，重啟MySQL
2、導出數據腳本
mysqldump -uroot -p123 test > test.sql
導出SQL腳本。或者用Navicat將所有數據庫/表導入到其他服務器的數據庫中。
注意：這裡的數據一定要備份成功。然後刪除原數據庫中的數據。
3、刪除ib_logfile0、ib_logfile1、ibdata1
備份MySQL數據目錄下的ib_logfile0、ib_logfile1、ibdata1三個文件，然後將這三個文件刪除
4、配置my.cnf
將my.cnf中innodb_force_recovery = 1或2——6幾個數字這行配置刪除或者配置為innodb_force_recovery = 0，重啟MySQL服務
5、將數據導入MySQL數據庫
mysql -uroot -p123 test < test.sql; 或者用Navicat將備份的數據導入到數據庫中。
此種方法下要注意的問題：
- ib_logfile0、ib_logfile1、ibdata1這三個文件一定要先備份後刪除；
- 一定要確認原數據導出成功了
- 當數據導出成功後，刪除原數據庫中的數據時，如果提示不能刪除，可在命令行進入MySQL的數據目錄，手動刪除相關數據庫的文件夾或者數據庫文件夾下的數據表文件，前提是數據一定導出或備份成功。
