#apache-kafka #kafka-consumer-api #acl
#apache-kafka #kafka-consumer-api #acl
Вопрос:
У меня следующая проблема: мое приложение подписывается на многие темы (около 16-20) с одной постоянной группой потребителей.
Если я использую kafka без SSL и ACL — он отлично работает.
Но на kafka с ssl и строго разделенными ACL у меня проблемы. Если большая часть тем не разрешена для клиента, тогда правильные темы (разрешенные ACL) не читаются, потребитель не отправляет запросы на выборку, и ни один раздел не назначен для этого.
2020-11-13 16:18:30.209 [INFO ] [o.a.k.c.c.internals.AbstractCoordinator ] [T:main ] - Successfully joined group PPRBODCommandConsumer with generation 74
2020-11-13 16:18:30.209 [INFO ] [o.a.k.c.c.internals.ConsumerCoordinator ] [T:main ] - Setting newly assigned partitions [] for group PPRBODCommandConsumer
2020-11-13 16:20:54.449 [DEBUG] [o.a.k.c.c.internals.AbstractCoordinator ] [T:PRBODCommandConsumer] - Sending Heartbeat request for group PPRBODCommandConsumer to coordinator vck4-s012-kfk010.vm.mos.cloud.sbrf.ru:9093 (id: 2147483644 rack: null)
2020-11-13 16:20:54.452 [DEBUG] [o.a.k.c.c.internals.AbstractCoordinator ] [T:main ] - Received successful Heartbeat response for group PPRBODCommandConsumer
2020-11-13 16:20:54.741 [DEBUG] [o.a.k.c.c.internals.ConsumerCoordinator ] [T:main ] - Sending asynchronous auto-commit of offsets {} for group PPRBODCommandConsumer
2020-11-13 16:20:54.762 [DEBUG] [o.a.k.c.c.internals.ConsumerCoordinator ] [T:main ] - Completed auto-commit of offsets {} for group PPRBODCommandConsumer
2020-11-13 16:20:55.741 [DEBUG] [o.a.k.c.c.internals.ConsumerCoordinator ] [T:main ] - Sending asynchronous auto-commit of offsets {} for group PPRBODCommandConsumer
2020-11-13 16:20:55.763 [DEBUG] [o.a.k.c.c.internals.ConsumerCoordinator ] [T:main ] - Completed auto-commit of offsets {} for group PPRBODCommandConsumer
2020-11-13 16:20:56.741 [DEBUG] [o.a.k.c.c.internals.ConsumerCoordinator ] [T:main ] - Sending asynchronous auto-commit of offsets {} for group PPRBODCommandConsumer
2020-11-13 16:20:56.764 [DEBUG] [o.a.k.c.c.internals.ConsumerCoordinator ] [T:main ] - Completed auto-commit of offsets {} for group PPRBODCommandConsumer
2020-11-13 16:20:57.449 [DEBUG] [o.a.k.c.c.internals.AbstractCoordinator ] [T:PRBODCommandConsumer] - Sending Heartbeat request for group PPRBODCommandConsumer to coordinator vck4-s012-kfk010.vm.mos.cloud.sbrf.ru:9093 (id: 2147483644 rack: null)
2020-11-13 16:20:57.453 [DEBUG] [o.a.k.c.c.internals.AbstractCoordinator ] [T:main ] - Received successful Heartbeat response for group PPRBODCommandConsumer
2020-11-13 16:20:57.742 [DEBUG] [o.a.k.c.c.internals.ConsumerCoordinator ] [T:main ] - Sending asynchronous auto-commit of offsets {} for group PPRBODCommandConsumer
2020-11-13 16:20:57.765 [DEBUG] [o.a.k.c.c.internals.ConsumerCoordinator ] [T:main ] - Completed auto-commit of offsets {} for group PPRBODCommandConsumer
2020-11-13 16:20:58.743 [DEBUG] [o.a.k.c.c.internals.ConsumerCoordinator ] [T:main ] - Sending asynchronous auto-commit of offsets {} for group PPRBODCommandConsumer
2020-11-13 16:20:58.766 [DEBUG] [o.a.k.c.c.internals.ConsumerCoordinator ] [T:main ] - Completed auto-commit of offsets {} for group PPRBODCommandConsumer
2020-11-13 16:20:59.744 [DEBUG] [o.a.k.c.c.internals.ConsumerCoordinator ] [T:main ] - Sending asynchronous auto-commit of offsets {} for group PPRBODCommandConsumer
2020-11-13 16:20:59.767 [DEBUG] [o.a.k.c.c.internals.ConsumerCoordinator ] [T:main ] - Completed auto-commit of offsets {} for group PPRBODCommandConsumer
2020-11-13 16:21:00.449 [DEBUG] [o.a.k.c.c.internals.AbstractCoordinator ] [T:PRBODCommandConsumer] - Sending Heartbeat request for group PPRBODCommandConsumer to coordinator vck4-s012-kfk010.vm.mos.cloud.sbrf.ru:9093 (id: 2147483644 rack: null)
2020-11-13 16:21:00.453 [DEBUG] [o.a.k.c.c.internals.AbstractCoordinator ] [T:main ] - Received successful Heartbeat response for group PPRBODCommandConsumer
2020-11-13 16:21:00.745 [DEBUG] [o.a.k.c.c.internals.ConsumerCoordinator ] [T:main ] - Sending asynchronous auto-commit of offsets {} for group PPRBODCommandConsumer
2020-11-13 16:21:00.768 [DEBUG] [o.a.k.c.c.internals.ConsumerCoordinator ] [T:main ] - Completed auto-commit of offsets {} for group PPRBODCommandConsumer
2020-11-13 16:21:01.746 [DEBUG] [o.a.k.c.c.internals.ConsumerCoordinator ] [T:main ] - Sending asynchronous auto-commit of offsets {} for group PPRBODCommandConsumer
2020-11-13 16:21:01.769 [DEBUG] [o.a.k.c.c.internals.ConsumerCoordinator ] [T:main ] - Completed auto-commit of offsets {} for group PPRBODCommandConsumer
2020-11-13 16:21:02.746 [DEBUG] [o.a.k.c.c.internals.ConsumerCoordinator ] [T:main ] - Sending asynchronous auto-commit of offsets {} for group PPRBODCommandConsumer
2020-11-13 16:21:02.770 [DEBUG] [o.a.k.c.c.internals.ConsumerCoordinator ] [T:main ] - Completed auto-commit of offsets {} for group PPRBODCommandConsumer
2020-11-13 16:21:03.449 [DEBUG] [o.a.k.c.c.internals.AbstractCoordinator ] [T:PRBODCommandConsumer] - Sending Heartbeat request for group PPRBODCommandConsumer to coordinator vck4-s012-kfk010.vm.mos.cloud.sbrf.ru:9093 (id: 2147483644 rack: null)
2020-11-13 16:21:03.454 [DEBUG] [o.a.k.c.c.internals.AbstractCoordinator ] [T:main ] - Received successful Heartbeat response for group PPRBODCommandConsumer
2020-11-13 16:21:03.747 [DEBUG] [o.a.k.c.c.internals.ConsumerCoordinator ] [T:main ] - Sending asynchronous auto-commit of offsets {} for group PPRBODCommandConsumer
2020-11-13 16:21:03.771 [DEBUG] [o.a.k.c.c.internals.ConsumerCoordinator ] [T:main ] - Completed auto-commit of offsets {} for group PPRBODCommandConsumer
2020-11-13 16:21:04.748 [DEBUG] [o.a.k.c.c.internals.ConsumerCoordinator ] [T:main ] - Sending asynchronous auto-commit of offsets {} for group PPRBODCommandConsumer
2020-11-13 16:21:04.771 [DEBUG] [o.a.k.c.c.internals.ConsumerCoordinator ] [T:main ] - Completed auto-commit of offsets {} for group PPRBODCommandConsumer
2020-11-13 16:21:05.749 [DEBUG] [o.a.k.c.c.internals.ConsumerCoordinator ] [T:main ] - Sending asynchronous auto-commit of offsets {} for group PPRBODCommandConsumer
2020-11-13 16:21:05.772 [DEBUG] [o.a.k.c.c.internals.ConsumerCoordinator ] [T:main ] - Completed auto-commit of offsets {} for group PPRBODCommandConsumer
2020-11-13 16:21:06.450 [DEBUG] [o.a.k.c.c.internals.AbstractCoordinator ] [T:PRBODCommandConsumer] - Sending Heartbeat request for group PPRBODCommandConsumer to coordinator vck4-s012-kfk010.vm.mos.cloud.sbrf.ru:9093 (id: 2147483644 rack: null)
2020-11-13 16:21:06.455 [DEBUG] [o.a.k.c.c.internals.AbstractCoordinator ] [T:main ] - Received successful Heartbeat response for group PPRBODCommandConsumer
2020-11-13 16:21:06.749 [DEBUG] [o.a.k.c.c.internals.ConsumerCoordinator ] [T:main ] - Sending asynchronous auto-commit of offsets {} for group PPRBODCommandConsumer
2020-11-13 16:21:06.773 [DEBUG] [o.a.k.c.c.internals.ConsumerCoordinator ] [T:main ] - Completed auto-commit of offsets {} for group PPRBODCommandConsumer
2020-11-13 16:21:07.750 [DEBUG] [o.a.k.c.c.internals.ConsumerCoordinator ] [T:main ] - Sending asynchronous auto-commit of offsets {} for group PPRBODCommandConsumer
2020-11-13 16:21:07.774 [DEBUG] [o.a.k.c.c.internals.ConsumerCoordinator ] [T:main ] - Completed auto-commit of offsets {} for group PPRBODCommandConsumer
2020-11-13 16:21:08.750 [DEBUG] [o.a.k.c.c.internals.ConsumerCoordinator ] [T:main ] - Sending asynchronous auto-commit of offsets {} for group PPRBODCommandConsumer
2020-11-13 16:21:08.775 [DEBUG] [o.a.k.c.c.internals.ConsumerCoordinator ] [T:main ] - Completed auto-commit of offsets {} for group PPRBODCommandConsumer
2020-11-13 16:21:09.451 [DEBUG] [o.a.k.c.c.internals.AbstractCoordinator ] [T:PRBODCommandConsumer] - Sending Heartbeat request for group PPRBODCommandConsumer to coordinator vck4-s012-kfk010.vm.mos.cloud.sbrf.ru:9093 (id: 2147483644 rack: null)
2020-11-13 16:21:09.455 [DEBUG] [o.a.k.c.c.internals.AbstractCoordinator ] [T:main ] - Received successful Heartbeat response for group PPRBODCommandConsumer
2020-11-13 16:21:09.751 [DEBUG] [o.a.k.c.c.internals.ConsumerCoordinator ] [T:main ] - Sending asynchronous auto-commit of offsets {} for group PPRBODCommandConsumer
2020-11-13 16:21:09.776 [DEBUG] [o.a.k.c.c.internals.ConsumerCoordinator ] [T:main ] - Completed auto-commit of offsets {} for group PPRBODCommandConsumer
2020-11-13 16:21:10.752 [DEBUG] [o.a.k.c.c.internals.ConsumerCoordinator ] [T:main ] - Sending asynchronous auto-commit of offsets {} for group PPRBODCommandConsumer
2020-11-13 16:21:10.777 [DEBUG] [o.a.k.c.c.internals.ConsumerCoordinator ] [T:main ] - Completed auto-commit of offsets {} for group PPRBODCommandConsumer
2020-11-13 16:21:11.753 [DEBUG] [o.a.k.c.c.internals.ConsumerCoordinator ] [T:main ] - Sending asynchronous auto-commit of offsets {} for group PPRBODCommandConsumer
2020-11-13 16:21:11.778 [DEBUG] [o.a.k.c.c.internals.ConsumerCoordinator ] [T:main ] - Completed auto-commit of offsets {} for group PPRBODCommandConsumer
Когда я создаю в it темы — сообщения не читаются потребителем.
Но если все темы разрешены для клиента или если я использую отдельную группу потребителей для каждой темы — это отлично работает
Это известная ошибка Kafka? Или Kafka не предназначена для использования с одной потребительской группой для многих тем?