Блог

Я изо всех сил пытаюсь запустить локально наше развертывание воздушного потока. Вот журналы, которые мы в настоящее время получаем:

 webserver_1 | ____________ _____________ webserver_1 | ____ |__( )_________ __/__ /________ __ webserver_1 | ____ /| |_ /__ ___/_ /_ __ /_ __ _ | /| / / webserver_1 | ___ ___ | / _ / _ __/ _ / / /_/ /_ |/ |/ / webserver_1 | _/_/ |_/_/ /_/ /_/ /_/ ____/____/|__/ webserver_1 | [2021-10-11 17:30:43,668] {dagbag.py:496} INFO - Filling up the DagBag from /dev/null webserver_1 | [2021-10-11 17:30:44,099] {manager.py:573} WARNING - Refused to delete permission view, assoc with role exists View Menus.can_create Admin webserver_1 | [2021-10-11 17:30:44,115] {manager.py:573} WARNING - Refused to delete permission view, assoc with role exists View Menus.can_edit Admin webserver_1 | [2021-10-11 17:30:44,126] {manager.py:573} WARNING - Refused to delete permission view, assoc with role exists View Menus.can_delete Admin webserver_1 | [2021-10-11 17:30:44,461] {manager.py:573} WARNING - Refused to delete permission view, assoc with role exists DAG Runs.clear Admin webserver_1 | Traceback (most recent call last): webserver_1 | File "/usr/local/bin/airflow", line 8, in <module> webserver_1 | sys.exit(main()) webserver_1 | File "/usr/local/lib/python3.7/site-packages/airflow/__main__.py", line 40, in main webserver_1 | args.func(args) webserver_1 | File "/usr/local/lib/python3.7/site-packages/airflow/cli/cli_parser.py", line 48, in command webserver_1 | return func(*args, **kwargs) webserver_1 | File "/usr/local/lib/python3.7/site-packages/airflow/utils/cli.py", line 91, in wrapper webserver_1 | return f(*args, **kwargs) webserver_1 | File "/usr/local/lib/python3.7/site-packages/airflow/cli/commands/webserver_command.py", line 368, in webserver webserver_1 | check_if_pidfile_process_is_running(pid_file=pid_file, process_name="webserver") webserver_1 | File "/usr/local/lib/python3.7/site-packages/airflow/utils/process_utils.py", line 267, in check_if_pidfile_process_is_running webserver_1 | raise AirflowException(f"The {process_name} is already running under PID {pid}.") webserver_1 | airflow.exceptions.AirflowException: The webserver is already running under PID 330. postgres_1 | LOG: unexpected EOF on client connection with an open transaction cbb-airflow_webserver_1 exited with code 1 webserver_1 | Requirement already satisfied: google-cloud-core==1.4.1 in /usr/local/lib/python3.7/site-packages (from -r /requirements.txt (line 1)) (1.4.1) webserver_1 | Requirement already satisfied: google-cloud-datastore==1.15.0 in /usr/local/lib/python3.7/site-packages (from -r /requirements.txt (line 2)) (1.15.0) webserver_1 | Requirement already satisfied: gcsfs==0.6.1 in /usr/local/lib/python3.7/site-packages (from -r /requirements.txt (line 3)) (0.6.1) webserver_1 | Requirement already satisfied: flatten-dict in /usr/local/lib/python3.7/site-packages (from -r /requirements.txt (line 4)) (0.4.2) webserver_1 | Requirement already satisfied: bigquery_schema_generator in /usr/local/lib/python3.7/site-packages (from -r /requirements.txt (line 5)) (1.4) webserver_1 | Requirement already satisfied: backoff in /usr/local/lib/python3.7/site-packages (from -r /requirements.txt (line 6)) (1.11.1) webserver_1 | Requirement already satisfied: six==1.13.0 in /usr/local/lib/python3.7/site-packages (from -r /requirements.txt (line 7)) (1.13.0) webserver_1 | Requirement already satisfied: ndjson in /usr/local/lib/python3.7/site-packages (from -r /requirements.txt (line 8)) (0.3.1) webserver_1 | Requirement already satisfied: pymongo in /usr/local/lib/python3.7/site-packages (from -r /requirements.txt (line 10)) (3.12.0) webserver_1 | Requirement already satisfied: SQLAlchemy==1.3.15 in /usr/local/lib/python3.7/site-packages (from -r /requirements.txt (line 14)) (1.3.15) webserver_1 | Requirement already satisfied: pandas==1.3.1 in /usr/local/lib/python3.7/site-packages (from -r /requirements.txt (line 15)) (1.3.1) webserver_1 | Requirement already satisfied: numpy==1.21.1 in /usr/local/lib/python3.7/site-packages (from -r /requirements.txt (line 16)) (1.21.1) webserver_1 | Requirement already satisfied: billiard in /usr/local/lib/python3.7/site-packages (from -r /requirements.txt (line 17)) (3.6.4.0) webserver_1 | Requirement already satisfied: google-api-core<2.0.0dev,>=1.19.0 in /usr/local/lib/python3.7/site-packages (from google-cloud-core==1.4.1->-r /requirements.txt (line 1)) (1.31.2) webserver_1 | Requirement already satisfied: fsspec>=0.6.0 in /usr/local/lib/python3.7/site-packages (from gcsfs==0.6.1->-r /requirements.txt (line 3)) (2021.7.0) webserver_1 | Requirement already satisfied: google-auth-oauthlib in /usr/local/lib/python3.7/site-packages (from gcsfs==0.6.1->-r /requirements.txt (line 3)) (0.4.5) webserver_1 | Requirement already satisfied: decorator in /usr/local/lib/python3.7/site-packages (from gcsfs==0.6.1->-r /requirements.txt (line 3)) (5.0.9) webserver_1 | Requirement already satisfied: requests in /usr/local/lib/python3.7/site-packages (from gcsfs==0.6.1->-r /requirements.txt (line 3)) (2.26.0) webserver_1 | Requirement already satisfied: google-auth>=1.2 in /usr/local/lib/python3.7/site-packages (from gcsfs==0.6.1->-r /requirements.txt (line 3)) (1.35.0) webserver_1 | Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.7/site-packages (from pandas==1.3.1->-r /requirements.txt (line 15)) (2021.1) webserver_1 | Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.7/site-packages (from pandas==1.3.1->-r /requirements.txt (line 15)) (2.8.2) webserver_1 | Requirement already satisfied: importlib-metadata in /usr/local/lib/python3.7/site-packages (from flatten-dict->-r /requirements.txt (line 4)) (4.6.4) webserver_1 | Requirement already satisfied: protobuf>=3.12.0 in /usr/local/lib/python3.7/site-packages (from google-api-core<2.0.0dev,>=1.19.0->google-cloud-core==1.4.1->-r /requirements.txt (line 1)) (3.17.3) webserver_1 | Requirement already satisfied: setuptools>=40.3.0 in /usr/local/lib/python3.7/site-packages (from google-api-core<2.0.0dev,>=1.19.0->google-cloud-core==1.4.1->-r /requirements.txt (line 1)) (57.4.0) webserver_1 | Requirement already satisfied: googleapis-common-protos<2.0dev,>=1.6.0 in /usr/local/lib/python3.7/site-packages (from google-api-core<2.0.0dev,>=1.19.0->google-cloud-core==1.4.1->-r /requirements.txt (line 1)) (1.53.0) webserver_1 | Requirement already satisfied: packaging>=14.3 in /usr/local/lib/python3.7/site-packages (from google-api-core<2.0.0dev,>=1.19.0->google-cloud-core==1.4.1->-r /requirements.txt (line 1)) (21.0) webserver_1 | Requirement already satisfied: grpcio<2.0dev,>=1.29.0 in /usr/local/lib/python3.7/site-packages (from google-api-core<2.0.0dev,>=1.19.0->google-cloud-core==1.4.1->-r /requirements.txt (line 1)) (1.39.0) webserver_1 | Requirement already satisfied: rsa<5,>=3.1.4 in /usr/local/lib/python3.7/site-packages (from google-auth>=1.2->gcsfs==0.6.1->-r /requirements.txt (line 3)) (4.7.2) webserver_1 | Requirement already satisfied: cachetools<5.0,>=2.0.0 in /usr/local/lib/python3.7/site-packages (from google-auth>=1.2->gcsfs==0.6.1->-r /requirements.txt (line 3)) (4.2.2) webserver_1 | Requirement already satisfied: pyasn1-modules>=0.2.1 in /usr/local/lib/python3.7/site-packages (from google-auth>=1.2->gcsfs==0.6.1->-r /requirements.txt (line 3)) (0.2.8) webserver_1 | Requirement already satisfied: pyparsing>=2.0.2 in /usr/local/lib/python3.7/site-packages (from packaging>=14.3->google-api-core<2.0.0dev,>=1.19.0->google-cloud-core==1.4.1->-r /requirements.txt (line 1)) (2.4.7) webserver_1 | Requirement already satisfied: pyasn1<0.5.0,>=0.4.6 in /usr/local/lib/python3.7/site-packages (from pyasn1-modules>=0.2.1->google-auth>=1.2->gcsfs==0.6.1->-r /requirements.txt (line 3)) (0.4.8) webserver_1 | Requirement already satisfied: charset-normalizer~=2.0.0 in /usr/local/lib/python3.7/site-packages (from requests->gcsfs==0.6.1->-r /requirements.txt (line 3)) (2.0.4) webserver_1 | Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/site-packages (from requests->gcsfs==0.6.1->-r /requirements.txt (line 3)) (2021.5.30) webserver_1 | Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.7/site-packages (from requests->gcsfs==0.6.1->-r /requirements.txt (line 3)) (3.2) webserver_1 | Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/local/lib/python3.7/site-packages (from requests->gcsfs==0.6.1->-r /requirements.txt (line 3)) (1.26.6) webserver_1 | Requirement already satisfied: requests-oauthlib>=0.7.0 in /usr/local/lib/python3.7/site-packages (from google-auth-oauthlib->gcsfs==0.6.1->-r /requirements.txt (line 3)) (1.3.0) webserver_1 | Requirement already satisfied: oauthlib>=3.0.0 in /usr/local/lib/python3.7/site-packages (from requests-oauthlib>=0.7.0->google-auth-oauthlib->gcsfs==0.6.1->-r /requirements.txt (line 3)) (3.1.1) webserver_1 | Requirement already satisfied: typing-extensions>=3.6.4 in /usr/local/lib/python3.7/site-packages (from importlib-metadata->flatten-dict->-r /requirements.txt (line 4)) (3.10.0.0) webserver_1 | Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.7/site-packages (from importlib-metadata->flatten-dict->-r /requirements.txt (line 4)) (3.5.0) postgres_1 | LOG: incomplete startup packet webserver_1 | /usr/local/lib/python3.7/site-packages/airflow/configuration.py:345 DeprecationWarning: The hide_sensitive_variable_fields option in [admin] has been moved to the hide_sensitive_var_conn_fields option in [core] - the old setting has been used, but please update your config. webserver_1 | /usr/local/lib/python3.7/site-packages/airflow/configuration.py:345 DeprecationWarning: The default_queue option in [celery] has been moved to the default_queue option in [operators] - the old setting has been used, but please update your config. webserver_1 | /usr/local/lib/python3.7/site-packages/airflow/configuration.py:345 DeprecationWarning: The default_queue option in [celery] has been moved to the default_queue option in [operators] - the old setting has been used, but please update your config. webserver_1 | DB: postgresql psycopg2://airflow:***@postgres:5432/airflow webserver_1 | [2021-10-11 17:30:59,666] {db.py:702} INFO - Creating tables webserver_1 | INFO [alembic.runtime.migration] Context impl PostgresqlImpl. webserver_1 | INFO [alembic.runtime.migration] Will assume transactional DDL. webserver_1 | Traceback (most recent call last): webserver_1 | File "/usr/local/bin/airflow", line 8, in <module> webserver_1 | sys.exit(main()) webserver_1 | File "/usr/local/lib/python3.7/site-packages/airflow/__main__.py", line 40, in main webserver_1 | args.func(args) webserver_1 | File "/usr/local/lib/python3.7/site-packages/airflow/cli/cli_parser.py", line 48, in command webserver_1 | return func(*args, **kwargs) webserver_1 | File "/usr/local/lib/python3.7/site-packages/airflow/cli/commands/db_command.py", line 31, in initdb webserver_1 | db.initdb() webserver_1 | File "/usr/local/lib/python3.7/site-packages/airflow/utils/db.py", line 576, in initdb webserver_1 | create_default_connections() webserver_1 | File "/usr/local/lib/python3.7/site-packages/airflow/utils/session.py", line 70, in wrapper webserver_1 | return func(*args, session=session, **kwargs) webserver_1 | File "/usr/local/lib/python3.7/site-packages/airflow/utils/db.py", line 335, in create_default_connections webserver_1 | session, webserver_1 | File "/usr/local/lib/python3.7/site-packages/airflow/utils/session.py", line 67, in wrapper webserver_1 | return func(*args, **kwargs) webserver_1 | File "/usr/local/lib/python3.7/site-packages/airflow/utils/db.py", line 61, in merge_conn webserver_1 | if not session.query(Connection).filter(Connection.conn_id == conn.conn_id).first(): webserver_1 | File "/usr/local/lib/python3.7/site-packages/sqlalchemy/orm/query.py", line 3298, in first webserver_1 | ret = list(self[0:1]) webserver_1 | File "/usr/local/lib/python3.7/site-packages/sqlalchemy/orm/query.py", line 3076, in __getitem__ webserver_1 | return list(res) webserver_1 | File "/usr/local/lib/python3.7/site-packages/sqlalchemy/orm/loading.py", line 101, in instances webserver_1 | cursor.close() webserver_1 | File "/usr/local/lib/python3.7/site-packages/sqlalchemy/util/langhelpers.py", line 69, in __exit__ webserver_1 | exc_value, with_traceback=exc_tb, webserver_1 | File "/usr/local/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 178, in raise_ webserver_1 | raise exception webserver_1 | File "/usr/local/lib/python3.7/site-packages/sqlalchemy/orm/loading.py", line 81, in instances webserver_1 | rows = [proc(row) for row in fetch] webserver_1 | File "/usr/local/lib/python3.7/site-packages/sqlalchemy/orm/loading.py", line 81, in <listcomp> webserver_1 | rows = [proc(row) for row in fetch] webserver_1 | File "/usr/local/lib/python3.7/site-packages/sqlalchemy/orm/loading.py", line 599, in _instance webserver_1 | state.manager.dispatch.load(state, context) webserver_1 | File "/usr/local/lib/python3.7/site-packages/sqlalchemy/event/attr.py", line 322, in __call__ webserver_1 | fn(*args, **kw) webserver_1 | File "/usr/local/lib/python3.7/site-packages/sqlalchemy/orm/mapper.py", line 3353, in _event_on_load webserver_1 | instrumenting_mapper._reconstructor(state.obj()) webserver_1 | File "/usr/local/lib/python3.7/site-packages/airflow/models/connection.py", line 150, in on_db_load webserver_1 | if self.password: webserver_1 | File "/usr/local/lib/python3.7/site-packages/sqlalchemy/orm/attributes.py", line 356, in __get__ webserver_1 | retval = self.descriptor.__get__(instance, owner) webserver_1 | File "/usr/local/lib/python3.7/site-packages/airflow/models/connection.py", line 235, in get_password webserver_1 | return fernet.decrypt(bytes(self._password, 'utf-8')).decode() webserver_1 | File "/usr/local/lib/python3.7/site-packages/cryptography/fernet.py", line 194, in decrypt webserver_1 | raise InvalidToken webserver_1 | cryptography.fernet.InvalidToken webserver_1 | /usr/local/lib/python3.7/site-packages/airflow/configuration.py:345 DeprecationWarning: The hide_sensitive_variable_fields option in [admin] has been moved to the hide_sensitive_var_conn_fields option in [core] - the old setting has been used, but please update your config. webserver_1 | /usr/local/lib/python3.7/site-packages/airflow/configuration.py:345 DeprecationWarning: The default_queue option in [celery] has been moved to the default_queue option in [operators] - the old setting has been used, but please update your config. webserver_1 | /usr/local/lib/python3.7/site-packages/airflow/configuration.py:345 DeprecationWarning: The default_queue option in [celery] has been moved to the default_queue option in [operators] - the old setting has been used, but please update your config. webserver_1 | [2021-10-11 17:31:05,551] {manager.py:573} WARNING - Refused to delete permission view, assoc with role exists View Menus.can_create Admin webserver_1 | [2021-10-11 17:31:05,561] {manager.py:573} WARNING - Refused to delete permission view, assoc with role exists View Menus.can_edit Admin webserver_1 | [2021-10-11 17:31:05,571] {manager.py:573} WARNING - Refused to delete permission view, assoc with role exists View Menus.can_delete Admin webserver_1 | [2021-10-11 17:31:05,808] {manager.py:573} WARNING - Refused to delete permission view, assoc with role exists DAG Runs.clear Admin webserver_1 | admin already exist in the db webserver_1 | /usr/local/lib/python3.7/site-packages/airflow/configuration.py:345 DeprecationWarning: The hide_sensitive_variable_fields option in [admin] has been moved to the hide_sensitive_var_conn_fields option in [core] - the old setting has been used, but please update your config. webserver_1 | /usr/local/lib/python3.7/site-packages/airflow/configuration.py:345 DeprecationWarning: The hide_sensitive_variable_fields option in [admin] has been moved to the hide_sensitive_var_conn_fields option in [core] - the old setting has been used, but please update your config. webserver_1 | /usr/local/lib/python3.7/site-packages/airflow/configuration.py:345 DeprecationWarning: The default_queue option in [celery] has been moved to the default_queue option in [operators] - the old setting has been used, but please update your config. webserver_1 | /usr/local/lib/python3.7/site-packages/airflow/configuration.py:345 DeprecationWarning: The default_queue option in [celery] has been moved to the default_queue option in [operators] - the old setting has been used, but please update your config. webserver_1 | /usr/local/lib/python3.7/site-packages/airflow/configuration.py:345 DeprecationWarning: The default_queue option in [celery] has been moved to the default_queue option in [operators] - the old setting has been used, but please update your config.  

Я предполагаю, что проблемные линии:

• воздушный поток.исключения.Исключение AirflowException: Веб-сервер уже работает под PID 330.
• поднимите недействительный токен, криптографию.фернет.Инвалидный токен
• администратор уже существует в базе данных
• cbb-airflow_webserver_1 вышел с кодом 1
• cbb-airflow_webserver_1 вышел с кодом 137 (не показан выше, но мы получаем это также в других текущих журналах)

Я совершенно теряюсь в догадках о том, что вызвало эти проблемы. Воздушный поток работал локально просто отлично, и он перестал работать сразу же после того, как я запустил DAG для локального запуска. Когда я посещаю localhost:8082, localhost:8082/home, localhost:8082/admin, ничего не работает, все показывают стандартную страницу ошибки «Сайт недоступен».

Есть какие-нибудь мысли о том, почему это может быть так, и как мы можем это решить? Я могу поделиться нашим docker-compose.yml файлом, если это поможет. Если это поможет узнать, мы также недавно обновили Airflow с версии v1 до версии v2, хотя Airflow ранее работал над версией v2, прежде чем запуск DAG привел к тому, что все сломалось.

Скорее всего, вы используете как воздушный поток 1, так и воздушный поток 2 одновременно. Вам следует убедиться, что вы очистили свою среду и используете только одну версию airflow/docker-compose за раз.

Вы можете использовать обычные команды docker и docker-compose для уничтожения/остановки запущенных контейнеров docker. Вы можете легко найти нужные команды. Например

docker kill $(docker ps -q)

Убьет все контейнеры докера.