#amazon-ec2 #mariadb #lamp #amazon-linux-2
Вопрос:
У меня здесь есть стандартный сервер LAMP под Amazon Linux 2 (экземпляр t3.small), где MariaDB выходит из строя каждые 1-2 недели. Конфигурация mariadb вышла из коробки, я не вносил никаких изменений. На сервере работает сайт MediaWiki и ничего больше.
Я предполагаю, что это и есть триггер сбоя: 2021-10-31 1:31:37 139789600806656 [ERROR] [FATAL] InnoDB: Semaphore wait has lasted gt; 600 seconds. We intentionally crash the server because it appears to be hung.
Что может быть причиной этого? Есть какой-нибудь способ отладить это?
Полный журнал сбоев:
[...] 0 lock struct(s), heap size 1128, 0 row lock(s) -------- FILE I/O -------- I/O thread 0 state: waiting for completed aio requests (insert buffer thread) I/O thread 1 state: waiting for completed aio requests (log thread) I/O thread 2 state: waiting for completed aio requests (read thread) I/O thread 3 state: waiting for completed aio requests (read thread) I/O thread 4 state: waiting for completed aio requests (read thread) I/O thread 5 state: waiting for completed aio requests (read thread) I/O thread 6 state: waiting for completed aio requests (write thread) I/O thread 7 state: waiting for completed aio requests (write thread) I/O thread 8 state: complete io for buf page (write thread) I/O thread 9 state: waiting for completed aio requests (write thread) Pending normal aio reads: [0, 0, 0, 0] , aio writes: [0, 0, 0, 0] , ibuf aio reads:, log i/o's:, sync i/o's: Pending flushes (fsync) log: 0; buffer pool: 2 1673169 OS file reads, 3264738 OS file writes, 1549190 OS fsyncs 0.00 reads/s, 0 avg bytes/read, 0.05 writes/s, 0.11 fsyncs/s ------------------------------------- INSERT BUFFER AND ADAPTIVE HASH INDEX ------------------------------------- Ibuf: size 1, free list len 20, seg size 22, 22437 merges merged operations: insert 10086, delete mark 16567, delete 1496 discarded operations: insert 0, delete mark 0, delete 0 InnoDB: ###### Diagnostic info printed to the standard error stream 2021-10-31 1:25:57 139789600806656 [Warning] InnoDB: A long semaphore wait: --Thread 139790080628480 has waited at row0sel.cc line 4775 for 619.00 seconds the semaphore: S-lock on RW-latch at 0x7f235936e940 created in file buf0buf.cc line 1523 a writer (thread id 139790341977856) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: ffffffffe0000000 Last time read locked in file row0ins.cc line 2606 Last time write locked in file btr0cur.cc line 2079 2021-10-31 1:25:57 139789600806656 [Note] InnoDB: A semaphore wait: --Thread 139790118184704 has waited at buf0buf.cc line 3555 for 5.00 seconds the semaphore: Mutex at 0x7f238330e400, Mutex BUF_POOL created buf0buf.cc:1810, lock var 2 2021-10-31 1:25:57 139789600806656 [Note] InnoDB: A semaphore wait: --Thread 139790083086080 has waited at buf0buf.cc line 3555 for 3.00 seconds the semaphore: Mutex at 0x7f238330e400, Mutex BUF_POOL created buf0buf.cc:1810, lock var 2 2021-10-31 1:25:57 139789600806656 [Note] InnoDB: A semaphore wait: --Thread 139790160312064 has waited at buf0buf.cc line 3555 for 3.00 seconds the semaphore: Mutex at 0x7f238330e400, Mutex BUF_POOL created buf0buf.cc:1810, lock var 2 2021-10-31 1:25:57 139789600806656 [Note] InnoDB: A semaphore wait: --Thread 139790341670656 has waited at buf0buf.cc line 3555 for 3.00 seconds the semaphore: Mutex at 0x7f238330e400, Mutex BUF_POOL created buf0buf.cc:1810, lock var 2 2021-10-31 1:25:57 139789600806656 [Note] InnoDB: A semaphore wait: --Thread 139790157547264 has waited at buf0buf.cc line 3555 for 6.00 seconds the semaphore: Mutex at 0x7f238330e400, Mutex BUF_POOL created buf0buf.cc:1810, lock var 2 2021-10-31 1:25:57 139789600806656 [Note] InnoDB: A semaphore wait: --Thread 139789938120448 has waited at buf0buf.cc line 3555 for 6.00 seconds the semaphore: Mutex at 0x7f238330e400, Mutex BUF_POOL created buf0buf.cc:1810, lock var 2 2021-10-31 1:25:57 139789600806656 [Note] InnoDB: A semaphore wait: --Thread 139789628032768 has waited at buf0lru.cc line 939 for 6.00 seconds the semaphore: Mutex at 0x7f238330e400, Mutex BUF_POOL created buf0buf.cc:1810, lock var 2 2021-10-31 1:25:57 139789600806656 [Note] InnoDB: A semaphore wait: --Thread 139790081242880 has waited at buf0buf.cc line 3555 for 6.00 seconds the semaphore: Mutex at 0x7f238330e400, Mutex BUF_POOL created buf0buf.cc:1810, lock var 2 2021-10-31 1:25:57 139789600806656 [Note] InnoDB: A semaphore wait: --Thread 139790159697664 has waited at buf0lru.cc line 939 for 6.00 seconds the semaphore: Mutex at 0x7f238330e400, Mutex BUF_POOL created buf0buf.cc:1810, lock var 2 2021-10-31 1:25:57 139789600806656 [Note] InnoDB: A semaphore wait: --Thread 139789590316800 has waited at btr0sea.ic line 97 for 6.00 seconds the semaphore: X-lock (wait_ex) on RW-latch at 0x7f2358dedba0 created in file btr0sea.cc line 191 a writer (thread id 139789590316800) has reserved it in mode wait exclusive number of readers 1, waiters flag 1, lock_word: ffffffffffffffff Last time read locked in file btr0sea.cc line 960 Last time write locked in file btr0sea.ic line 97 2021-10-31 1:25:57 139789600806656 [Note] InnoDB: A semaphore wait: --Thread 139789937813248 has waited at buf0buf.cc line 3555 for 3.00 seconds the semaphore: Mutex at 0x7f238330e400, Mutex BUF_POOL created buf0buf.cc:1810, lock var 2 2021-10-31 1:25:57 139789600806656 [Note] InnoDB: A semaphore wait: --Thread 139790156318464 has waited at row0sel.cc line 4540 for 5.00 seconds the semaphore: S-lock on RW-latch at 0x7f2358dedba0 created in file btr0sea.cc line 191 a writer (thread id 139789590316800) has reserved it in mode wait exclusive number of readers 1, waiters flag 1, lock_word: ffffffffffffffff Last time read locked in file btr0sea.cc line 960 Last time write locked in file btr0sea.ic line 97 2021-10-31 1:25:57 139789600806656 [Note] InnoDB: A semaphore wait: --Thread 139790082778880 has waited at buf0buf.cc line 3555 for 6.00 seconds the semaphore: Mutex at 0x7f238330e400, Mutex BUF_POOL created buf0buf.cc:1810, lock var 2 2021-10-31 1:25:57 139789600806656 [Note] InnoDB: A semaphore wait: --Thread 139790080628480 has waited at row0sel.cc line 4775 for 619.00 seconds the semaphore: S-lock on RW-latch at 0x7f235936e940 created in file buf0buf.cc line 1523 a writer (thread id 139790341977856) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: ffffffffe0000000 Last time read locked in file row0ins.cc line 2606 Last time write locked in file btr0cur.cc line 2079 InnoDB: ###### Starts InnoDB Monitor for 30 secs to print diagnostic info: InnoDB: Pending reads 0, writes 0 Hash table size 34679, node heap has 54 buffer(s) Hash table size 34679, node heap has 16 buffer(s) Hash table size 34679, node heap has 24 buffer(s) Hash table size 34679, node heap has 17 buffer(s) Hash table size 34679, node heap has 1 buffer(s) Hash table size 34679, node heap has 26 buffer(s) Hash table size 34679, node heap has 8 buffer(s) Hash table size 34679, node heap has 36 buffer(s) 28.12 hash searches/s, 20.00 non-hash searches/s --- LOG --- InnoDB: ###### Diagnostic info printed to the standard error stream 2021-10-31 1:26:28 139789600806656 [Warning] InnoDB: A long semaphore wait: --Thread 139790080628480 has waited at row0sel.cc line 4775 for 650.00 seconds the semaphore: S-lock on RW-latch at 0x7f235936e940 created in file buf0buf.cc line 1523 a writer (thread id 139790341977856) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: ffffffffe0000000 Last time read locked in file row0ins.cc line 2606 Last time write locked in file btr0cur.cc line 2079 2021-10-31 1:26:28 139789600806656 [Note] InnoDB: A semaphore wait: --Thread 139789590316800 has waited at log0log.cc line 2099 for 6.00 seconds the semaphore: Mutex at 0x7f23833b8110, Mutex LOG_SYS created log0log.cc:657, lock var 2 2021-10-31 1:26:28 139789600806656 [Note] InnoDB: A semaphore wait: --Thread 139789579826944 has waited at log0log.cc line 1583 for 6.00 seconds the semaphore: Mutex at 0x7f23833b8110, Mutex LOG_SYS created log0log.cc:657, lock var 2 2021-10-31 1:26:28 139789600806656 [Note] InnoDB: A semaphore wait: --Thread 139790080628480 has waited at row0sel.cc line 4775 for 650.00 seconds the semaphore: S-lock on RW-latch at 0x7f235936e940 created in file buf0buf.cc line 1523 a writer (thread id 139790341977856) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: ffffffffe0000000 Last time read locked in file row0ins.cc line 2606 Last time write locked in file btr0cur.cc line 2079 InnoDB: ###### Starts InnoDB Monitor for 30 secs to print diagnostic info: InnoDB: Pending reads 0, writes 0 InnoDB: ###### Diagnostic info printed to the standard error stream 2021-10-31 1:26:59 139789600806656 [Warning] InnoDB: A long semaphore wait: --Thread 139790080628480 has waited at row0sel.cc line 4775 for 681.00 seconds the semaphore: S-lock on RW-latch at 0x7f235936e940 created in file buf0buf.cc line 1523 a writer (thread id 139790341977856) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: ffffffffe0000000 Last time read locked in file row0ins.cc line 2606 Last time write locked in file btr0cur.cc line 2079 2021-10-31 1:26:59 139789600806656 [Note] InnoDB: A semaphore wait: --Thread 139789590316800 has waited at log0log.cc line 2099 for 37.00 seconds the semaphore: Mutex at 0x7f23833b8110, Mutex LOG_SYS created log0log.cc:657, lock var 2 2021-10-31 1:26:59 139789600806656 [Note] InnoDB: A semaphore wait: --Thread 139789579826944 has waited at log0log.cc line 1583 for 37.00 seconds the semaphore: Mutex at 0x7f23833b8110, Mutex LOG_SYS created log0log.cc:657, lock var 2 2021-10-31 1:26:59 139789600806656 [Note] InnoDB: A semaphore wait: --Thread 139790080628480 has waited at row0sel.cc line 4775 for 681.00 seconds the semaphore: S-lock on RW-latch at 0x7f235936e940 created in file buf0buf.cc line 1523 a writer (thread id 139790341977856) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: ffffffffe0000000 Last time read locked in file row0ins.cc line 2606 Last time write locked in file btr0cur.cc line 2079 InnoDB: ###### Starts InnoDB Monitor for 30 secs to print diagnostic info: InnoDB: Pending reads 0, writes 0 InnoDB: ###### Diagnostic info printed to the standard error stream 2021-10-31 1:27:30 139789600806656 [Warning] InnoDB: A long semaphore wait: --Thread 139790080628480 has waited at row0sel.cc line 4775 for 712.00 seconds the semaphore: S-lock on RW-latch at 0x7f235936e940 created in file buf0buf.cc line 1523 a writer (thread id 139790341977856) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: ffffffffe0000000 Last time read locked in file row0ins.cc line 2606 Last time write locked in file btr0cur.cc line 2079 2021-10-31 1:27:30 139789600806656 [Note] InnoDB: A semaphore wait: --Thread 139789590316800 has waited at log0log.cc line 2099 for 68.00 seconds the semaphore: Mutex at 0x7f23833b8110, Mutex LOG_SYS created log0log.cc:657, lock var 2 2021-10-31 1:27:30 139789600806656 [Note] InnoDB: A semaphore wait: --Thread 139789579826944 has waited at log0log.cc line 1583 for 68.00 seconds the semaphore: Mutex at 0x7f23833b8110, Mutex LOG_SYS created log0log.cc:657, lock var 2 2021-10-31 1:27:30 139789600806656 [Note] InnoDB: A semaphore wait: --Thread 139790080628480 has waited at row0sel.cc line 4775 for 712.00 seconds the semaphore: S-lock on RW-latch at 0x7f235936e940 created in file buf0buf.cc line 1523 a writer (thread id 139790341977856) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: ffffffffe0000000 Last time read locked in file row0ins.cc line 2606 Last time write locked in file btr0cur.cc line 2079 InnoDB: ###### Starts InnoDB Monitor for 30 secs to print diagnostic info: InnoDB: Pending reads 0, writes 0 InnoDB: ###### Diagnostic info printed to the standard error stream 2021-10-31 1:28:01 139789600806656 [Warning] InnoDB: A long semaphore wait: --Thread 139790080628480 has waited at row0sel.cc line 4775 for 743.00 seconds the semaphore: S-lock on RW-latch at 0x7f235936e940 created in file buf0buf.cc line 1523 a writer (thread id 139790341977856) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: ffffffffe0000000 Last time read locked in file row0ins.cc line 2606 Last time write locked in file btr0cur.cc line 2079 2021-10-31 1:28:01 139789600806656 [Note] InnoDB: A semaphore wait: --Thread 139789590316800 has waited at log0log.cc line 2099 for 99.00 seconds the semaphore: Mutex at 0x7f23833b8110, Mutex LOG_SYS created log0log.cc:657, lock var 2 2021-10-31 1:28:01 139789600806656 [Note] InnoDB: A semaphore wait: --Thread 139789579826944 has waited at log0log.cc line 1583 for 99.00 seconds the semaphore: Mutex at 0x7f23833b8110, Mutex LOG_SYS created log0log.cc:657, lock var 2 2021-10-31 1:28:01 139789600806656 [Note] InnoDB: A semaphore wait: --Thread 139790080628480 has waited at row0sel.cc line 4775 for 743.00 seconds the semaphore: S-lock on RW-latch at 0x7f235936e940 created in file buf0buf.cc line 1523 a writer (thread id 139790341977856) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: ffffffffe0000000 Last time read locked in file row0ins.cc line 2606 Last time write locked in file btr0cur.cc line 2079 InnoDB: ###### Starts InnoDB Monitor for 30 secs to print diagnostic info: InnoDB: Pending reads 0, writes 0 InnoDB: ###### Diagnostic info printed to the standard error stream 2021-10-31 1:28:32 139789600806656 [Warning] InnoDB: A long semaphore wait: --Thread 139790080628480 has waited at row0sel.cc line 4775 for 774.00 seconds the semaphore: S-lock on RW-latch at 0x7f235936e940 created in file buf0buf.cc line 1523 a writer (thread id 139790341977856) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: ffffffffe0000000 Last time read locked in file row0ins.cc line 2606 Last time write locked in file btr0cur.cc line 2079 2021-10-31 1:28:32 139789600806656 [Note] InnoDB: A semaphore wait: --Thread 139789590316800 has waited at log0log.cc line 2099 for 130.00 seconds the semaphore: Mutex at 0x7f23833b8110, Mutex LOG_SYS created log0log.cc:657, lock var 2 2021-10-31 1:28:32 139789600806656 [Note] InnoDB: A semaphore wait: --Thread 139789579826944 has waited at log0log.cc line 1583 for 130.00 seconds the semaphore: Mutex at 0x7f23833b8110, Mutex LOG_SYS created log0log.cc:657, lock var 2 2021-10-31 1:28:32 139789600806656 [Note] InnoDB: A semaphore wait: --Thread 139790080628480 has waited at row0sel.cc line 4775 for 774.00 seconds the semaphore: S-lock on RW-latch at 0x7f235936e940 created in file buf0buf.cc line 1523 a writer (thread id 139790341977856) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: ffffffffe0000000 Last time read locked in file row0ins.cc line 2606 Last time write locked in file btr0cur.cc line 2079 InnoDB: ###### Starts InnoDB Monitor for 30 secs to print diagnostic info: InnoDB: Pending reads 0, writes 0 InnoDB: ###### Diagnostic info printed to the standard error stream 2021-10-31 1:29:03 139789600806656 [Warning] InnoDB: A long semaphore wait: --Thread 139790080628480 has waited at row0sel.cc line 4775 for 805.00 seconds the semaphore: S-lock on RW-latch at 0x7f235936e940 created in file buf0buf.cc line 1523 a writer (thread id 139790341977856) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: ffffffffe0000000 Last time read locked in file row0ins.cc line 2606 Last time write locked in file btr0cur.cc line 2079 2021-10-31 1:29:03 139789600806656 [Note] InnoDB: A semaphore wait: --Thread 139789590316800 has waited at log0log.cc line 2099 for 161.00 seconds the semaphore: Mutex at 0x7f23833b8110, Mutex LOG_SYS created log0log.cc:657, lock var 2 2021-10-31 1:29:03 139789600806656 [Note] InnoDB: A semaphore wait: --Thread 139789579826944 has waited at log0log.cc line 1583 for 161.00 seconds the semaphore: Mutex at 0x7f23833b8110, Mutex LOG_SYS created log0log.cc:657, lock var 2 2021-10-31 1:29:03 139789600806656 [Note] InnoDB: A semaphore wait: --Thread 139790080628480 has waited at row0sel.cc line 4775 for 805.00 seconds the semaphore: S-lock on RW-latch at 0x7f235936e940 created in file buf0buf.cc line 1523 a writer (thread id 139790341977856) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: ffffffffe0000000 Last time read locked in file row0ins.cc line 2606 Last time write locked in file btr0cur.cc line 2079 InnoDB: ###### Starts InnoDB Monitor for 30 secs to print diagnostic info: InnoDB: Pending reads 0, writes 0 2021-10-31 1:29:12 139790081857280 [Warning] Aborted connection 2164853 to db: 'wiki' user: 'db127135_7' host: 'localhost' (Got an error writing communication packets) InnoDB: ###### Diagnostic info printed to the standard error stream 2021-10-31 1:29:34 139789600806656 [Warning] InnoDB: A long semaphore wait: --Thread 139790080628480 has waited at row0sel.cc line 4775 for 836.00 seconds the semaphore: S-lock on RW-latch at 0x7f235936e940 created in file buf0buf.cc line 1523 a writer (thread id 139790341977856) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: ffffffffe0000000 Last time read locked in file row0ins.cc line 2606 Last time write locked in file btr0cur.cc line 2079 2021-10-31 1:29:34 139789600806656 [Note] InnoDB: A semaphore wait: --Thread 139789590316800 has waited at log0log.cc line 2099 for 192.00 seconds the semaphore: Mutex at 0x7f23833b8110, Mutex LOG_SYS created log0log.cc:657, lock var 2 2021-10-31 1:29:34 139789600806656 [Note] InnoDB: A semaphore wait: --Thread 139789579826944 has waited at log0log.cc line 1583 for 192.00 seconds the semaphore: Mutex at 0x7f23833b8110, Mutex LOG_SYS created log0log.cc:657, lock var 2 2021-10-31 1:29:34 139789600806656 [Note] InnoDB: A semaphore wait: --Thread 139790080628480 has waited at row0sel.cc line 4775 for 836.00 seconds the semaphore: S-lock on RW-latch at 0x7f235936e940 created in file buf0buf.cc line 1523 a writer (thread id 139790341977856) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: ffffffffe0000000 Last time read locked in file row0ins.cc line 2606 Last time write locked in file btr0cur.cc line 2079 InnoDB: ###### Starts InnoDB Monitor for 30 secs to print diagnostic info: InnoDB: Pending reads 0, writes 0 InnoDB: ###### Diagnostic info printed to the standard error stream 2021-10-31 1:30:05 139789600806656 [Warning] InnoDB: A long semaphore wait: --Thread 139790080628480 has waited at row0sel.cc line 4775 for 867.00 seconds the semaphore: S-lock on RW-latch at 0x7f235936e940 created in file buf0buf.cc line 1523 a writer (thread id 139790341977856) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: ffffffffe0000000 Last time read locked in file row0ins.cc line 2606 Last time write locked in file btr0cur.cc line 2079 2021-10-31 1:30:05 139789600806656 [Note] InnoDB: A semaphore wait: --Thread 139789590316800 has waited at log0log.cc line 2099 for 223.00 seconds the semaphore: Mutex at 0x7f23833b8110, Mutex LOG_SYS created log0log.cc:657, lock var 2 2021-10-31 1:30:05 139789600806656 [Note] InnoDB: A semaphore wait: --Thread 139789579826944 has waited at log0log.cc line 1583 for 223.00 seconds the semaphore: Mutex at 0x7f23833b8110, Mutex LOG_SYS created log0log.cc:657, lock var 2 2021-10-31 1:30:05 139789600806656 [Note] InnoDB: A semaphore wait: --Thread 139790080628480 has waited at row0sel.cc line 4775 for 867.00 seconds the semaphore: S-lock on RW-latch at 0x7f235936e940 created in file buf0buf.cc line 1523 a writer (thread id 139790341977856) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: ffffffffe0000000 Last time read locked in file row0ins.cc line 2606 Last time write locked in file btr0cur.cc line 2079 InnoDB: ###### Starts InnoDB Monitor for 30 secs to print diagnostic info: InnoDB: Pending reads 0, writes 0 InnoDB: ###### Diagnostic info printed to the standard error stream 2021-10-31 1:30:36 139789600806656 [Warning] InnoDB: A long semaphore wait: --Thread 139789590316800 has waited at log0log.cc line 2099 for 254.00 seconds the semaphore: Mutex at 0x7f23833b8110, Mutex LOG_SYS created log0log.cc:657, lock var 2 2021-10-31 1:30:36 139789600806656 [Warning] InnoDB: A long semaphore wait: --Thread 139789579826944 has waited at log0log.cc line 1583 for 254.00 seconds the semaphore: Mutex at 0x7f23833b8110, Mutex LOG_SYS created log0log.cc:657, lock var 2 2021-10-31 1:30:36 139789600806656 [Warning] InnoDB: A long semaphore wait: --Thread 139790080628480 has waited at row0sel.cc line 4775 for 898.00 seconds the semaphore: S-lock on RW-latch at 0x7f235936e940 created in file buf0buf.cc line 1523 a writer (thread id 139790341977856) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: ffffffffe0000000 Last time read locked in file row0ins.cc line 2606 Last time write locked in file btr0cur.cc line 2079 2021-10-31 1:30:36 139789600806656 [Note] InnoDB: A semaphore wait: --Thread 139789590316800 has waited at log0log.cc line 2099 for 254.00 seconds the semaphore: Mutex at 0x7f23833b8110, Mutex LOG_SYS created log0log.cc:657, lock var 2 2021-10-31 1:30:36 139789600806656 [Note] InnoDB: A semaphore wait: --Thread 139789579826944 has waited at log0log.cc line 1583 for 254.00 seconds the semaphore: Mutex at 0x7f23833b8110, Mutex LOG_SYS created log0log.cc:657, lock var 2 2021-10-31 1:30:36 139789600806656 [Note] InnoDB: A semaphore wait: --Thread 139790080628480 has waited at row0sel.cc line 4775 for 898.00 seconds the semaphore: S-lock on RW-latch at 0x7f235936e940 created in file buf0buf.cc line 1523 a writer (thread id 139790341977856) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: ffffffffe0000000 Last time read locked in file row0ins.cc line 2606 Last time write locked in file btr0cur.cc line 2079 InnoDB: ###### Starts InnoDB Monitor for 30 secs to print diagnostic info: InnoDB: Pending reads 0, writes 0 InnoDB: ###### Diagnostic info printed to the standard error stream 2021-10-31 1:31:07 139789600806656 [Warning] InnoDB: A long semaphore wait: --Thread 139789590316800 has waited at log0log.cc line 2099 for 285.00 seconds the semaphore: Mutex at 0x7f23833b8110, Mutex LOG_SYS created log0log.cc:657, lock var 2 2021-10-31 1:31:07 139789600806656 [Warning] InnoDB: A long semaphore wait: --Thread 139789579826944 has waited at log0log.cc line 1583 for 285.00 seconds the semaphore: Mutex at 0x7f23833b8110, Mutex LOG_SYS created log0log.cc:657, lock var 2 2021-10-31 1:31:07 139789600806656 [Warning] InnoDB: A long semaphore wait: --Thread 139790080628480 has waited at row0sel.cc line 4775 for 929.00 seconds the semaphore: S-lock on RW-latch at 0x7f235936e940 created in file buf0buf.cc line 1523 a writer (thread id 139790341977856) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: ffffffffe0000000 Last time read locked in file row0ins.cc line 2606 Last time write locked in file btr0cur.cc line 2079 2021-10-31 1:31:07 139789600806656 [Note] InnoDB: A semaphore wait: --Thread 139789590316800 has waited at log0log.cc line 2099 for 285.00 seconds the semaphore: Mutex at 0x7f23833b8110, Mutex LOG_SYS created log0log.cc:657, lock var 2 2021-10-31 1:31:07 139789600806656 [Note] InnoDB: A semaphore wait: --Thread 139789579826944 has waited at log0log.cc line 1583 for 285.00 seconds the semaphore: Mutex at 0x7f23833b8110, Mutex LOG_SYS created log0log.cc:657, lock var 2 2021-10-31 1:31:07 139789600806656 [Note] InnoDB: A semaphore wait: --Thread 139790080628480 has waited at row0sel.cc line 4775 for 929.00 seconds the semaphore: S-lock on RW-latch at 0x7f235936e940 created in file buf0buf.cc line 1523 a writer (thread id 139790341977856) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: ffffffffe0000000 Last time read locked in file row0ins.cc line 2606 Last time write locked in file btr0cur.cc line 2079 InnoDB: ###### Starts InnoDB Monitor for 30 secs to print diagnostic info: InnoDB: Pending reads 0, writes 0 InnoDB: ###### Diagnostic info printed to the standard error stream 2021-10-31 1:31:37 139789600806656 [ERROR] [FATAL] InnoDB: Semaphore wait has lasted gt; 600 seconds. We intentionally crash the server because it appears to be hung. 211031 1:31:37 [ERROR] mysqld got signal 6 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.2.38-MariaDB key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=104 max_threads=153 thread_count=110 It is possible that mysqld could use up to key_buffer_size (read_buffer_size sort_buffer_size)*max_threads = 467243 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x0 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x0 thread_stack 0x49000 Log sequence number 23513466164 Log flushed up to 23513452869 Pages flushed up to 23513452869 Last checkpoint at 23513452332 1 pending log flushes, 0 pending chkp writes 864275 log i/o's done, 0.00 log i/o's/second ---------------------- BUFFER POOL AND MEMORY ---------------------- Total large memory allocated 137232384 Dictionary memory allocated 331760 Buffer pool size 8192 Free buffers 1017 Database pages 6991 Old database pages 2560 Modified db pages 14 Percent of dirty pages(LRU amp; free pages): 0.175 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 1, flush list 0, single page 0 Pages made young 88918373, not young 10092370 0.54 youngs/s, 0.04 non-youngs/s Pages read 1673053, created 42467, written 2144372 0.01 reads/s, 0.00 creates/s, 0.00 writes/s Buffer pool hit rate 999 / 1000, young-making rate 12 / 1000 not 0 / 1000 Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 6991, unzip_LRU len: 0 I/O sum[25]:cur[8], unzip sum[0]:cur[0] -------------- ROW OPERATIONS -------------- 0 queries inside InnoDB, 0 queries in queue 50 read views open inside InnoDB Process ID=25605, Main thread ID=139789579826944, state: making checkpoint Number of rows inserted 269732, updated 97845, deleted 190647, read 614433742 0.00 inserts/s, 0.00 updates/s, 0.06 deletes/s, 21838.66 reads/s Number of system rows inserted 1765, updated 0, deleted 0, read 3530 0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s ---------------------------- END OF INNODB MONITOR OUTPUT ============================ /usr/libexec/mysqld(my_print_stacktrace 0x2e)[0x5578f86d222e] /usr/libexec/mysqld(handle_fatal_signal 0x4df)[0x5578f818d0bf] /lib64/libpthread.so.0( 0x117e0)[0x7f2387f597e0] /lib64/libc.so.6(gsignal 0x110)[0x7f23863caca0] /lib64/libc.so.6(abort 0x148)[0x7f23863cc148] /usr/libexec/mysqld( 0x986d91)[0x5578f849cd91] /usr/libexec/mysqld( 0x932426)[0x5578f8448426] /lib64/libpthread.so.0( 0x740b)[0x7f2387f4f40b] /lib64/libc.so.6(clone 0x3f)[0x7f238648440f] The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /var/lib/mysql Resource Limits: Fatal signal 11 while backtracing