Блог

Я создавал прототипы K8S и RabbitMQ на ванильном локальном экземпляре K8S (т. Е. Не AWS / Azure). Я установил RabbitMQ с помощью скрипта Bitnami Helm, и он работал в первый раз без каких-либо проблем. Затем я решил удалить развертывание (через helm), добавить некоторые плагины сообщества, а затем переустановить с помощью того же скрипта Helm.

Все учетные данные были одинаковыми между обеими установками.

Теперь K8S не запускает модуль: (

Сообщение об ошибке K8S из модуля:

 Readiness probe failed: Error: Failed to connect and authenticate to rabbit@rabbitmq-headless.rabbitmq.svc.cluster.local
 

Я заметил, что внешний IP-адрес, предоставленный службе (через MetalLB), отличается, но я думаю, что имя модуля такое же.

Я думаю, что удаление helm не очистило некоторые внутренние записи DNS K8S…. но я все еще изучаю K8S, поэтому я не уверен, где искать, чтобы проверить это, а также найти и удалить все, что висит вокруг.

В журнале модулей ничего полезного не говорится:

 rabbitmq 22:12:35.68 
rabbitmq 22:12:35.68 Welcome to the Bitnami rabbitmq container
rabbitmq 22:12:35.68 Subscribe to project updates by watching https://github.com/bitnami/bitnami-docker-rabbitmq
rabbitmq 22:12:35.69 Submit issues and feature requests at https://github.com/bitnami/bitnami-docker-rabbitmq/issues
rabbitmq 22:12:35.69 
rabbitmq 22:12:35.69 INFO  ==> ** Starting RabbitMQ setup **
rabbitmq 22:12:35.71 INFO  ==> Validating settings in RABBITMQ_* env vars..
rabbitmq 22:12:35.74 INFO  ==> Initializing RabbitMQ...
 

Я оставил это на некоторое время, и это никогда не менялось.

Затем я попытался переустановить без дополнительного плагина (это был плагин с задержкой сообщения).

Казалось, что все прошло лучше, но модули по-прежнему не запускаются (по той же причине). В этом случае журнал модуля указывает на некоторое движение

 rabbitmq 22:37:46.70 
rabbitmq 22:37:46.71 Welcome to the Bitnami rabbitmq container
rabbitmq 22:37:46.71 Subscribe to project updates by watching https://github.com/bitnami/bitnami-docker-rabbitmq
rabbitmq 22:37:46.71 Submit issues and feature requests at https://github.com/bitnami/bitnami-docker-rabbitmq/issues
rabbitmq 22:37:46.71 
rabbitmq 22:37:46.72 INFO  ==> ** Starting RabbitMQ setup **
rabbitmq 22:37:46.74 INFO  ==> Validating settings in RABBITMQ_* env vars..
rabbitmq 22:37:46.76 INFO  ==> Initializing RabbitMQ...
rabbitmq 22:37:46.80 INFO  ==> Persisted data detected. Restoring...
rabbitmq 22:37:46.81 INFO  ==> ** RabbitMQ setup finished! **

rabbitmq 22:37:46.83 INFO  ==> ** Starting RabbitMQ **
Configuring logger redirection
2021-01-25 22:38:02.925 [debug] <0.286.0> Lager installed handler error_logger_lager_h into error_logger
2021-01-25 22:38:02.954 [debug] <0.289.0> Lager installed handler lager_forwarder_backend into error_logger_lager_event
2021-01-25 22:38:02.954 [debug] <0.292.0> Lager installed handler lager_forwarder_backend into rabbit_log_lager_event
2021-01-25 22:38:02.954 [debug] <0.295.0> Lager installed handler lager_forwarder_backend into rabbit_log_channel_lager_event
2021-01-25 22:38:02.954 [debug] <0.319.0> Lager installed handler lager_forwarder_backend into rabbit_log_ra_lager_event
2021-01-25 22:38:02.954 [debug] <0.322.0> Lager installed handler lager_forwarder_backend into rabbit_log_shovel_lager_event
2021-01-25 22:38:02.954 [debug] <0.298.0> Lager installed handler lager_forwarder_backend into rabbit_log_connection_lager_event
2021-01-25 22:38:02.954 [debug] <0.325.0> Lager installed handler lager_forwarder_backend into rabbit_log_upgrade_lager_event
2021-01-25 22:38:02.954 [debug] <0.301.0> Lager installed handler lager_forwarder_backend into rabbit_log_feature_flags_lager_event
2021-01-25 22:38:02.954 [debug] <0.304.0> Lager installed handler lager_forwarder_backend into rabbit_log_federation_lager_event
2021-01-25 22:38:02.955 [debug] <0.307.0> Lager installed handler lager_forwarder_backend into rabbit_log_ldap_lager_event
2021-01-25 22:38:02.955 [debug] <0.310.0> Lager installed handler lager_forwarder_backend into rabbit_log_mirroring_lager_event
2021-01-25 22:38:02.955 [debug] <0.313.0> Lager installed handler lager_forwarder_backend into rabbit_log_prelaunch_lager_event
2021-01-25 22:38:02.955 [debug] <0.316.0> Lager installed handler lager_forwarder_backend into rabbit_log_queue_lager_event
2021-01-25 22:38:02.972 [info] <0.44.0> Application lager started on node 'rabbit@rabbitmq-0.rabbitmq-headless.rabbitmq.svc.cluster.local'
2021-01-25 22:38:03.291 [info] <0.44.0> Application mnesia started on node 'rabbit@rabbitmq-0.rabbitmq-headless.rabbitmq.svc.cluster.local'
2021-01-25 22:38:03.292 [info] <0.270.0> 
 Starting RabbitMQ 3.8.9 on Erlang 22.3
 Copyright (c) 2007-2020 VMware, Inc. or its affiliates.
 Licensed under the MPL 2.0. Website: https://rabbitmq.com

  ##  ##      RabbitMQ 3.8.9
  ##  ##
  ##########  Copyright (c) 2007-2020 VMware, Inc. or its affiliates.
  ######  ##
  ##########  Licensed under the MPL 2.0. Website: https://rabbitmq.com

  Doc guides: https://rabbitmq.com/documentation.html
  Support:    https://rabbitmq.com/contact.html
  Tutorials:  https://rabbitmq.com/getstarted.html
  Monitoring: https://rabbitmq.com/monitoring.html

  Logs: <stdout>

  Config file(s): /opt/bitnami/rabbitmq/etc/rabbitmq/rabbitmq.conf

  Starting broker...2021-01-25 22:38:03.295 [info] <0.270.0> 
 node           : rabbit@rabbitmq-0.rabbitmq-headless.rabbitmq.svc.cluster.local
 home dir       : /opt/bitnami/rabbitmq/.rabbitmq
 config file(s) : /opt/bitnami/rabbitmq/etc/rabbitmq/rabbitmq.conf
 cookie hash    : ENSyEjlycK3XumHZ0m5A3g==
 log(s)         : <stdout>
 database dir   : /bitnami/rabbitmq/mnesia/rabbit@rabbitmq-0.rabbitmq-headless.rabbitmq.svc.cluster.local
2021-01-25 22:38:03.425 [debug] <0.282.0> Lager installed handler lager_backend_throttle into lager_event
2021-01-25 22:38:07.395 [info] <0.270.0> Running boot step pre_boot defined by app rabbit
2021-01-25 22:38:07.395 [info] <0.270.0> Running boot step rabbit_core_metrics defined by app rabbit
2021-01-25 22:38:07.396 [info] <0.270.0> Running boot step rabbit_alarm defined by app rabbit
2021-01-25 22:38:07.402 [info] <0.351.0> Memory high watermark set to 1574 MiB (1650966528 bytes) of 3936 MiB (4127416320 bytes) total
2021-01-25 22:38:07.409 [info] <0.353.0> Enabling free disk space monitoring
2021-01-25 22:38:07.409 [info] <0.353.0> Disk free limit set to 50MB
2021-01-25 22:38:07.415 [info] <0.270.0> Running boot step code_server_cache defined by app rabbit
2021-01-25 22:38:07.415 [info] <0.270.0> Running boot step file_handle_cache defined by app rabbit
2021-01-25 22:38:07.416 [info] <0.356.0> Limiting to approx 1048479 file handles (943629 sockets)
2021-01-25 22:38:07.416 [info] <0.357.0> FHC read buffering:  OFF
2021-01-25 22:38:07.416 [info] <0.357.0> FHC write buffering: ON
2021-01-25 22:38:07.417 [info] <0.270.0> Running boot step worker_pool defined by app rabbit
2021-01-25 22:38:07.417 [info] <0.344.0> Will use 2 processes for default worker pool
2021-01-25 22:38:07.417 [info] <0.344.0> Starting worker pool 'worker_pool' with 2 processes in it
2021-01-25 22:38:07.418 [info] <0.270.0> Running boot step database defined by app rabbit
2021-01-25 22:38:07.419 [info] <0.270.0> Node database directory at /bitnami/rabbitmq/mnesia/rabbit@rabbitmq-0.rabbitmq-headless.rabbitmq.svc.cluster.local is empty. Assuming we need to join an existing cluster or initialise from scratch...
2021-01-25 22:38:07.419 [info] <0.270.0> Configured peer discovery backend: rabbit_peer_discovery_k8s
2021-01-25 22:38:07.438 [info] <0.270.0> Will try to lock with peer discovery backend rabbit_peer_discovery_k8s
2021-01-25 22:38:07.438 [info] <0.270.0> Peer discovery backend does not support locking, falling back to randomized delay
2021-01-25 22:38:07.439 [info] <0.270.0> Peer discovery backend rabbit_peer_discovery_k8s supports registration.
2021-01-25 22:38:07.439 [info] <0.270.0> Will wait for 90 milliseconds before proceeding with registration...
2021-01-25 22:38:09.789 [error] <0.270.0> Failed to fetch a list of nodes from Kubernetes API: {failed_connect,[{to_address,{"kubernetes.default.svc.cluster.local",443}},
                 {inet,[inet],timeout}]}
2021-01-25 22:38:12.043 [error] <0.270.0> Peer discovery returned an error: "{failed_connect,[{to_address,{"kubernetes.default.svc.cluster.local",443}},n                 {inet,[inet],timeout}]}". Will retry after a delay of 500 ms, 9 retries left...
2021-01-25 22:38:14.796 [error] <0.270.0> Failed to fetch a list of nodes from Kubernetes API: {failed_connect,[{to_address,{"kubernetes.default.svc.cluster.local",443}},
                 {inet,[inet],timeout}]}
2021-01-25 22:38:17.050 [error] <0.270.0> Peer discovery returned an error: "{failed_connect,[{to_address,{"kubernetes.default.svc.cluster.local",443}},n                 {inet,[inet],timeout}]}". Will retry after a delay of 500 ms, 8 retries left...
2021-01-25 22:38:19.803 [error] <0.270.0> Failed to fetch a list of nodes from Kubernetes API: {failed_connect,[{to_address,{"kubernetes.default.svc.cluster.local",443}},
                 {inet,[inet],timeout}]}
2021-01-25 22:38:22.056 [error] <0.270.0> Peer discovery returned an error: "{failed_connect,[{to_address,{"kubernetes.default.svc.cluster.local",443}},n                 {inet,[inet],timeout}]}". Will retry after a delay of 500 ms, 7 retries left...
2021-01-25 22:38:24.812 [error] <0.270.0> Failed to fetch a list of nodes from Kubernetes API: {failed_connect,[{to_address,{"kubernetes.default.svc.cluster.local",443}},                 {inet,[inet],timeout}]}
 

Команда helm для плагина была:

 helm install rabbitmq bitnami/rabbitmq -n rabbitmq --set 
metrics.enabled=true,auth.username=admin,auth.password=xxxxxxxxxx,service.type=LoadBalancer,communityPlugins=https://github.com/rabbitmq/rabbitmq-delayed-message-exchange/releases/download/3.8.9/rabbitmq_delayed_message_exchange-3.8.9-0199d11c.ez,extraPlugins=rabbitmq_delayed_message_exchange,replicaCount=1,metrics.serviceMonitor.enabled=true
 

Тот, у которого нет подключаемого модуля:

 helm install rabbitmq bitnami/rabbitmq -n rabbitmq --set 
metrics.enabled=true,auth.username=admin,auth.password=xxxxxxxxxx,service.type=LoadBalancer
 

Запуск на K8S 1.18.3