Блог

Я запускаю скрипт на python для подключения AWS SSM .

В моем docker-compose установлен этот том:

 - ~/.aws/:/home/airflow/.aws
 

Код Boto3:

 LOCALHOST = 1
SERVICE = 'ssm'
PROFILE = 'profile3'
#File path
CURRENT_PATH = os.path.dirname(os.path.realpath(__file__))


def get_aws_client(localhost=None):
    """
    Creates boto3 aws client for any service.
    :param localhost: Parameter that enables use of roles in localhost.
    :return: aws client object
    """
    if localhost is not None:
        globals().update(LOCALHOST=localhost)
    boto_object = Boto3AwsClient(localhost=LOCALHOST, profile=PROFILE)
    aws_client = boto_object.aws_client_connect(service=SERVICE)
    return aws_client
 

Он возвращается:

 botocore.exceptions.ProfileNotFound: The config profile (profile3) could not be found
 

Если я побегу:

 docker exec -it webserver bash 
 

И распечатать

 cat /home/airflow/.aws/credentials
cat /home/airflow/.aws/config
 

Я вижу для верительных грамот:

 [default]
aws_access_key_id = XXXXXXXXXXXXXX
aws_secret_access_key = XXXXXXXXXXXXXXXXxxxxxxxxxxxxxXXXXXXXXXXX
 

Для конфигурации:

 [default]
region=eu-west-1
output=json

[profile profile3]
region=eu-west-1
role_arn=arn:aws:iam::333333333333:role/AllowBlablahblah
source_profile=default

[profile profile2]
region=eu-west-1
role_arn=arn:aws:iam::22222222222:role/AllowBliblihblih
source_profile=default

[profile profile1]
region=eu-west-1
role_arn=arn:aws:iam::1111111111111:role/AllowBlubluhbluh
source_profile=default
 

And event I can run without problem:

 aws s3 ls
aws s3 ls --profile profile3
 

So I guess config and credentials are not really missing, and no format issue as aws cli is working.

I don’t know what’s going on here. Any idea?

Dockerfile:

 FROM apache/airflow:2.1.2-python3.8

ARG AIRFLOW_USER_HOME=/opt/airflow

ENV PYTHONPATH "${PYTHONPATH}:/"

ADD ./environtment_config/docker_src ./environtment_config/docker_src

RUN pip install -r environtment_config/docker_src/requirements.pip
 

Full docker-compose:

  version: '3'
    services:
      webserver:
        image: own-airflow2
        command: webserver
        ports:
          - 8080:8080
        healthcheck:
          test: [ "CMD", "curl", "--fail", "http://localhost:8080/health" ]
          interval: 10s
          timeout: 10s
          retries: 5
        restart: always
        build:
          context: .
          dockerfile: Dockerfile3
        env_file:
          - ./airflow.env
        container_name: webserver
        volumes:
          - ./database_utils:/database_utils
          - ./maintenance:/maintenance
          - ./utils:/utils
          - ./dags:/opt/airflow/dags
          - ./logs:/opt/airflow/logs
          - ./airflow_sqlite:/opt/airflow
          - ~/.aws/:/home/airflow/.aws
      scheduler:
        image: own-airflow2
        command: scheduler
        healthcheck:
          test: [ "CMD-SHELL", 'airflow jobs check --job-type SchedulerJob --hostname "${HOSTNAME}"' ]
          interval: 10s
          timeout: 10s
          retries: 5
        restart: always
        container_name: scheduler
        build:
          context: .
          dockerfile: Dockerfile3
        env_file:
          - ./airflow.env
        volumes:
          - ./database_utils:/database_utils
          - ./maintenance:/maintenance
          - ./utils:/utils
          - ./dags:/opt/airflow/dags
          - ./logs:/opt/airflow/logs
          - ./airflow_sqlite:/opt/airflow
          - ~/.aws/:/home/airflow/.aws
        depends_on:
          - webserver
 

Редактировать:

Я забыл сказать, что я добавил вары, такие как:

 #Boto3
AWS_CONFIG_FILE=/home/airflow/.aws/config
AWS_SHARED_CREDENTIALS_FILE=/home/airflow/.aws/credentials
 

Чтобы четко указать, какой из них является правильным путем к файлу.