Вторичная БД Неожиданно получила сбой

#mongodb #database-replication #wiredtiger

Вопрос:

Основной и дополнительный работают в стандарте Windows server 2012 R2. и сегодня Основная Машина вышла из строя. и после этого вторичный экземпляр Mongo отключается из-за ошибки ниже полученного сигнала: 22 (SIGABRT). а затем Основной переходит в режим восстановления. Основной сервер не печатает никаких чистых журналов.

Журнал Основного Сервера:

 2021-07-23T19:13:53.237 0530 I REPL     [replexec-23] Member SecondaryHost:27017 is now in state RS_DOWN 2021-07-23T19:13:53.237 0530 I REPL     [replexec-23] can't see a majority of the set, relinquishing primary 2021-07-23T19:13:53.237 0530 I REPL     [replexec-23] Stepping down from primary in response to heartbeat 2021-07-23T19:13:53.237 0530 I REPL     [replexec-23] transition to RECOVERING from PRIMARY 2021-07-23T19:13:53.238 0530 I NETWORK  [replexec-23] Skip closing connection for connection # 11139
 

Сегодня наш основной сервер UAT внезапно переходит в режим восстановления, а Вторичный сервер не вставал.
Из журналов я получил ниже Ошибки со вторичного сервера. Просто хочу знать, почему это произошло.

Журналы Вторичного Сервера:

 2021-07-23T19:05:26.900 0530 E STORAGE [WTCheckpointThread] WiredTiger error (13) [1627047326:876858][1540:140704437702784], file:ReconDB_UAT/collection/26--5782211086932514962.wt, WT_SESSION.checkpoint: __win_open_file, 543: D:MongoDataDataReconDB_UAT/collection/26--5782211086932514962.wt: handle-open: CreateFileW: Access is denied. 
    : Permission denied Raw: [1627047326:876858][1540:140704437702784], file:ReconDB_UAT/collection/26--5782211086932514962.wt, WT_SESSION.checkpoint: __win_open_file, 543: D:MongoDataDataReconDB_UAT/collection/26--5782211086932514962.wt: handle-open: CreateFileW: Access is denied. 
    : Permission denied
    2021-07-23T19:05:26.900 0530 F - [WTCheckpointThread] Invariant failure: s->checkpoint(s, "use_timestamp=true") resulted in status UnknownError: 13: Permission denied at srcmongodbstoragewiredtigerwiredtiger_kv_engine.cpp 352
    2021-07-23T19:05:26.900 0530 F - [WTCheckpointThread]
    
    ***aborting after invariant() failure
    
    
    2021-07-23T19:05:28.595 0530 I ASIO [Replication] Connecting to ArbHost:27017
    2021-07-23T19:05:28.642 0530 I ASIO [Replication] Failed to connect to ArbHost:27017 - AuthenticationFailed: Authentication failed.
    2021-07-23T19:05:28.642 0530 I CONNPOOL [Replication] Dropping all pooled connections to ArbHost:27017 due to AuthenticationFailed: Authentication failed.
    2021-07-23T19:05:28.642 0530 I REPL_HB [replexec-99] Error in heartbeat (requestId: 1211726) to ArbHost:27017, response status: AuthenticationFailed: Authentication failed.
    2021-07-23T19:05:30.643 0530 I ASIO [Replication] Connecting to ArbHost:27017
    2021-07-23T19:05:30.689 0530 I ASIO [Replication] Failed to connect to ArbHost:27017 - AuthenticationFailed: Authentication failed.
    2021-07-23T19:05:30.689 0530 I CONNPOOL [Replication] Dropping all pooled connections to ArbHost:27017 due to AuthenticationFailed: Authentication failed.
    2021-07-23T19:05:30.690 0530 I REPL_HB [replexec-99] Error in heartbeat (requestId: 1211729) to ArbHost:27017, response status: AuthenticationFailed: Authentication failed.
    2021-07-23T19:05:32.691 0530 I ASIO [Replication] Connecting to ArbHost:27017
    2021-07-23T19:05:32.736 0530 I ASIO [Replication] Failed to connect to ArbHost:27017 - AuthenticationFailed: Authentication failed.
    2021-07-23T19:05:32.736 0530 I CONNPOOL [Replication] Dropping all pooled connections to ArbHost:27017 due to AuthenticationFailed: Authentication failed.
    2021-07-23T19:05:32.736 0530 I REPL_HB [replexec-99] Error in heartbeat (requestId: 1211732) to ArbHost:27017, response status: AuthenticationFailed: Authentication failed.
    2021-07-23T19:05:32.952 0530 F - [WTCheckpointThread] Got signal: 22 (SIGABRT).
    mongod.exe ...srcmongoutilstacktrace_windows.cpp(247) mongo::printStackTrace 0x43
    mongod.exe ...srcmongoutilsignal_handlers_synchronous.cpp(184) mongo::`anonymous namespace'::printSignalAndBacktrace 0x74
    mongod.exe ...srcmongoutilsignal_handlers_synchronous.cpp(240) mongo::`anonymous namespace'::abruptQuit 0x85
    ucrtbase.DLL raise 0x1e7
    ucrtbase.DLL abort 0x31
    mongod.exe ...srcmongoutilassert_util.cpp(129) mongo::invariantOKFailed 0x228
    mongod.exe ...srcmongodbstoragewiredtigerwiredtiger_kv_engine.cpp(352) mongo::WiredTigerKVEngine::WiredTigerCheckpointThread::run 0x661
    mongod.exe ...srcmongoutilbackground.cpp(153) mongo::BackgroundJob::jobBody 0x131
    mongod.exe c:program files (x86)microsoft visual studio 14.0vcincludethrxthread(247) std::_LaunchPad<std::unique_ptr<std::tuple<<lambda_8929d5e2502e4de5883694c1a7a329eb> >,std::default_delete<std::tuple<<lambda_8929d5e2502e4de5883694c1a7a329eb> > > > >::_Run 0x77
    mongod.exe c:program files (x86)microsoft visual studio 14.0vcincludethrxthread(210) std::_Pad::_Call_func 0x9
    ucrtbase.DLL o__realloc_base 0x60
    KERNEL32.DLL BaseThreadInitThunk 0x22
    2021-07-23T19:05:33.006 0530 F CONTROL [WTCheckpointThread] *** unhandled exception 0x0000000E at 0x00007FF84DC27AFC, terminating
    2021-07-23T19:05:33.006 0530 F CONTROL [WTCheckpointThread] *** stack trace for unhandled exception:
    2021-07-23T19:05:33.021 0530 I - [WTCheckpointThread] KERNELBASE.dll RaiseException 0x6c
    mongod.exe ...srcmongoutilsignal_handlers_synchronous.cpp(241) mongo::`anonymous namespace'::abruptQuit 0x9d
    ucrtbase.DLL raise 0x1e7
    ucrtbase.DLL abort 0x31
    mongod.exe ...srcmongoutilassert_util.cpp(129) mongo::invariantOKFailed 0x228
    mongod.exe ...srcmongodbstoragewiredtigerwiredtiger_kv_engine.cpp(352) mongo::WiredTigerKVEngine::WiredTigerCheckpointThread::run 0x661
    mongod.exe ...srcmongoutilbackground.cpp(153) mongo::BackgroundJob::jobBody 0x131
    mongod.exe c:program files (x86)microsoft visual studio 14.0vcincludethrxthread(247) std::_LaunchPad<std::unique_ptr<std::tuple<<lambda_8929d5e2502e4de5883694c1a7a329eb> >,std::default_delete<std::tuple<<lambda_8929d5e2502e4de5883694c1a7a329eb> > > > >::_Run 0x77
    mongod.exe c:program files (x86)microsoft visual studio 14.0vcincludethrxthread(210) std::_Pad::_Call_func 0x9
    ucrtbase.DLL o__realloc_base 0x60
    KERNEL32.DLL BaseThreadInitThunk 0x22
    2021-07-23T19:05:33.022 0530 I CONTROL [WTCheckpointThread] writing minidump diagnostic file C:Program FilesMongoDBServer4.2021-07-23T13-35-33.mdmp
    2021-07-23T19:13:01.047 0530 I NETWORK [listener] connection accepted from PrimaryHost:51006 #182621 (15 connections now open)