Блог

Я получаю эти две ошибки Validator.java:268 - Failed creating a merkle tree for и CassandraDaemon.java:228 - Exception in thread Thread в точное время 0t:00:03 каждый час

 cat debug.log | grep -i java | grep -i err | grep -i -oE '.{0,80}ValidationExecutor.{0,85}'
ERROR [ValidationExecutor:6067] 2021-12-10 06:00:09,358 Validator.java:268 - Failed creating a merkle tree for [repair #6a9c43e0-597e-11ec-954f-0be7ea52f380 on mars/mapping, [(372614885
ERROR [ValidationExecutor:6067] 2021-12-10 06:00:09,359 CassandraDaemon.java:228 - Exception in thread Thread[ValidationExecutor:6067,1,main]
ERROR [ValidationExecutor:6068] 2021-12-10 06:00:15,555 Validator.java:268 - Failed creating a merkle tree for [repair #6ac29090-597e-11ec-bb29-c95c929750e0 on mars/mapping, [(372614885
ERROR [ValidationExecutor:6068] 2021-12-10 06:00:15,556 CassandraDaemon.java:228 - Exception in thread Thread[ValidationExecutor:6068,1,main]
ERROR [ValidationExecutor:6070] 2021-12-10 07:00:03,236 Validator.java:268 - Failed creating a merkle tree for [repair #cc2b9220-5986-11ec-bb29-c95c929750e0 on mars/mapping, [(372614885
ERROR [ValidationExecutor:6071] 2021-12-10 07:00:03,236 Validator.java:268 - Failed creating a merkle tree for [repair #cc2c2e60-5986-11ec-954f-0be7ea52f380 on mars/mapping, [(372614885
ERROR [ValidationExecutor:6070] 2021-12-10 07:00:03,237 CassandraDaemon.java:228 - Exception in thread Thread[ValidationExecutor:6070,1,main]
ERROR [ValidationExecutor:6071] 2021-12-10 07:00:03,237 CassandraDaemon.java:228 - Exception in thread Thread[ValidationExecutor:6071,1,main]
ERROR [ValidationExecutor:6073] 2021-12-10 08:00:03,422 Validator.java:268 - Failed creating a merkle tree for [repair #2e0ad520-598f-11ec-bb29-c95c929750e0 on mars/mapping, [(372614885
ERROR [ValidationExecutor:6073] 2021-12-10 08:00:03,423 CassandraDaemon.java:228 - Exception in thread Thread[ValidationExecutor:6073,1,main]
ERROR [ValidationExecutor:6074] 2021-12-10 08:00:03,424 Validator.java:268 - Failed creating a merkle tree for [repair #2e1a8c90-598f-11ec-a28d-115f730d3c5f on mars/mapping, [(372614885
ERROR [ValidationExecutor:6074] 2021-12-10 08:00:03,425 CassandraDaemon.java:228 - Exception in thread Thread[ValidationExecutor:6074,1,main]
ERROR [ValidationExecutor:6076] 2021-12-10 09:00:03,566 Validator.java:268 - Failed creating a merkle tree for [repair #8fe906b0-5997-11ec-bb29-c95c929750e0 on mars/mapping, [(372614885
ERROR [ValidationExecutor:6076] 2021-12-10 09:00:03,569 CassandraDaemon.java:228 - Exception in thread Thread[ValidationExecutor:6076,1,main]
ERROR [ValidationExecutor:6077] 2021-12-10 09:00:03,655 Validator.java:268 - Failed creating a merkle tree for [repair #8ff032a0-5997-11ec-a28d-115f730d3c5f on mars/mapping, [(372614885
ERROR [ValidationExecutor:6077] 2021-12-10 09:00:03,656 CassandraDaemon.java:228 - Exception in thread Thread[ValidationExecutor:6077,1,main]
ERROR [ValidationExecutor:6079] 2021-12-10 10:00:03,833 Validator.java:268 - Failed creating a merkle tree for [repair #f1bf9720-599f-11ec-bb29-c95c929750e0 on mars/mapping, [(372614885
ERROR [ValidationExecutor:6080] 2021-12-10 10:00:03,833 Validator.java:268 - Failed creating a merkle tree for [repair #f1cbf330-599f-11ec-954f-0be7ea52f380 on mars/mapping, [(372614885
ERROR [ValidationExecutor:6079] 2021-12-10 10:00:03,834 CassandraDaemon.java:228 - Exception in thread Thread[ValidationExecutor:6079,1,main]
ERROR [ValidationExecutor:6080] 2021-12-10 10:00:03,834 CassandraDaemon.java:228 - Exception in thread Thread[ValidationExecutor:6080,1,main]
ERROR [ValidationExecutor:6083] 2021-12-10 11:00:20,044 Validator.java:268 - Failed creating a merkle tree for [repair #53a0d5f0-59a8-11ec-a28d-115f730d3c5f on mars/mapping, [(372614885
ERROR [ValidationExecutor:6082] 2021-12-10 11:00:20,044 Validator.java:268 - Failed creating a merkle tree for [repair #538d9c10-59a8-11ec-bb29-c95c929750e0 on mars/mapping, [(372614885
ERROR [ValidationExecutor:6083] 2021-12-10 11:00:20,045 CassandraDaemon.java:228 - Exception in thread Thread[ValidationExecutor:6083,1,main]
ERROR [ValidationExecutor:6082] 2021-12-10 11:00:20,045 CassandraDaemon.java:228 - Exception in thread Thread[ValidationExecutor:6082,1,main]
ERROR [ValidationExecutor:6085] 2021-12-10 12:00:03,054 Validator.java:268 - Failed creating a merkle tree for [repair #b4ec3d60-59b0-11ec-954f-0be7ea52f380 on mars/mapping, [(372614885
ERROR [ValidationExecutor:6085] 2021-12-10 12:00:03,055 CassandraDaemon.java:228 - Exception in thread Thread[ValidationExecutor:6085,1,main]
ERROR [ValidationExecutor:6085] 2021-12-10 12:00:03,563 ActiveRepairService.java:561 - Cannot start multiple repair sessions over the same sstables
ERROR [ValidationExecutor:6085] 2021-12-10 12:00:03,563 Validator.java:268 - Failed creating a merkle tree for [repair #b533a6a0-59b0-11ec-bb29-c95c929750e0 on mars/mapping, [(372614885
ERROR [ValidationExecutor:6085] 2021-12-10 12:00:03,564 CassandraDaemon.java:228 - Exception in thread Thread[ValidationExecutor:6085,1,main]
 

В чем причина их возникновения и повторения и как их устранить?

Журналы показывают, что Cassandra потерпела неудачу на этапе проверки процесса антиэнтропийного восстановления.

Поскольку «Невозможно запустить несколько сеансов восстановления через одни и те же sstables», это означает, что одновременно выполняется несколько сеансов восстановления в одном и том же диапазоне токенов.

Вам нужно убедиться, что в данный момент в вашем кластере не запущен сеанс восстановления и нет антикомпактирования.

Я предлагаю выполнить перезапуск, чтобы остановить все текущие ремонтные работы. затем попробуйте восстановить узел за узлом.

Последнее предложение — попробовать https://github.com/thelastpickle/cassandra-reaper раньше он запускал автоматический ремонт для Кассандры.