6주차 - EKS Security - IRSA & Pod Identity(3)

 

EKS IRSA(IAM Roles for Service Accounts)

 

기존에는 Worker-Node 즉 EC2에 맵핑된 IAM Role를 Pod가 그대로 사용 할 수 있었다.

• EC2 Instance Profile : 사용하기 편하지만, 최소 권한 부여 원칙에 위배하며 보안상 권고하지 않음
• Pod가 탈취되면 그대로 EC2 IAM Role를 사용하여 보안적으로 취약

 

IRSA: Pod에 최소한의 IAM 권한을 주는것

          파드가 특정 IAM 역할로 Assume 할때 토큰을 AWS에 전송하고, AWS는 토큰과 EKS IdP를 통해 해당 IAM 역할을            사용할 수 있는지 검증

 

 

실습  1 ( service account Token - false)

 

1. Pod 생성 시 Service Account Token이 자동 마운트 설정 해제

# 파드1 생성
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: eks-iam-test1
spec:
  containers:
    - name: my-aws-cli
      image: amazon/aws-cli:latest
      args: ['s3', 'ls']
  restartPolicy: Never
  automountServiceAccountToken: false
  terminationGracePeriodSeconds: 0
EOF

# 확인
kubectl get pod
kubectl describe pod

# 로그 확인
kubectl logs eks-iam-test1

An error occurred (AccessDenied) when calling the ListBuckets operation: User: arn:aws:sts::890742582773:assumed-role/eksctl-myeks-nodegroup-ng1-NodeInstanceRole-CPeuWkY6c6v1/i-0eab9954006609e15 is not authorized to perform: s3:ListAllMyBuckets because no identity-based policy allows the s3:ListAllMyBuckets action


# 파드1 삭제
kubectl delete pod eks-iam-test1

 

실습  2 ( service account Token - True)

 

1. Pod 생성

# 파드2 생성
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: eks-iam-test2
spec:
  containers:
    - name: my-aws-cli
      image: amazon/aws-cli:latest
      command: ['sleep', '36000']
  restartPolicy: Never
  terminationGracePeriodSeconds: 0
EOF

# 확인
kubectl get pod
kubectl describe pod
kubectl get pod eks-iam-test2 -o yaml 

kubectl exec -it eks-iam-test2 -- ls /var/run/secrets/kubernetes.io/serviceaccount
ca.crt	namespace  token

kubectl exec -it eks-iam-test2 -- cat /var/run/secrets/kubernetes.io/serviceaccount/token ;echo
eyJhbGciOiJSUzI1NiIsImtpZCI6Ijc1M2E1YjRlYWIwMGE0NDkzYjdjMDBmMzQ2ZjEyOGJlZmVlMzQ4YzcifQ.eyJhdWQiOlsiaHR0cHM6Ly9rdWJlcm5ldGVzLmRlZmF1bHQuc3ZjIl0sImV4cCI6MTc3MzYwMjkzOCwiaWF0IjoxNzQyMDY2OTM4LCJpc3MiOiJodHRwczovL29pZGMuZWtzLmFwLW5vcnRoZWFzdC0yLmFtYXpvbmF3cy5jb20vaWQvNTI4NDFFQjMxOTMwOEY2MTE2NjQ1QUJCMUM5RURDNUUiLCJqdGkiOiJhZDI2NDA0MS1jMDdmLTRlOTYtOTVhNy0xZmQzZDQ5MzYyN2EiLCJrdWJlcm5ldGVzLmlvIjp7Im5hbWVzcGFjZSI6ImRlZmF1bHQiLCJub2RlIjp7Im5hbWUiOiJpcC0xOTItMTY4LTMtMjA0LmFwLW5vcnRoZWFzdC0yLmNvbXB1dGUuaW50ZXJuYWwiLCJ1aWQiOiJkMDUxNDBlYS02NTk0LTQ5MTEtYjE0MC0wNDg0ZTViMGE5OTcifSwicG9kIjp7Im5hbWUiOiJla3MtaWFtLXRlc3QyIiwidWlkIjoiODg2NDJhYzUtMWViMy00NGU0LWE1NGYtZmYzZjdjOTczMWNlIn0sInNlcnZpY2VhY2NvdW50Ijp7Im5hbWUiOiJkZWZhdWx0IiwidWlkIjoiZDdmMTMzN2YtYmIxZS00M2FkLWEwYTctZGRhM2Y5OWFmYjQ5In0sIndhcm5hZnRlciI6MTc0MjA3MDU0NX0sIm5iZiI6MTc0MjA2NjkzOCwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50OmRlZmF1bHQ6ZGVmYXVsdCJ9.BqOsag6X6QjN865LsLOoRXdiGqtjVR8BUJfehY-_XaeT1vY0zEp7iWNWzbAaTbOcFrhRw9FyRipWGcwsf4DUX5ydmAKCB9DviDkle42prGh9StiCjpyNcaqxRbiCeEMoIg36eDUo8uIPwgsZvPD-6ZWUgyRRJenD34G8iIGfgs7xi-IxCbqPjlZGS5CaRktokROvxf5WU3Yz57IrmfdqoAZRTdZvXZSGqLv0C11VHvtw5PpxmNLXnRzHHZkv3qL5H5cN2WCj1MDnP86GTia8q4KGh5APwhSOZ4iLfPfutHa2vVrcnzs7GeefWgq83Nhr5FYBG6sUNtQAecIL_RYzYg

kubectl exec -it eks-iam-test2 -- aws s3 ls

An error occurred (AccessDenied) when calling the ListBuckets operation: User: arn:aws:sts::890742582773:assumed-role/eksctl-myeks-nodegroup-ng1-NodeInstanceRole-CPeuWkY6c6v1/i-0eab9954006609e15 is not authorized to perform: s3:ListAllMyBuckets because no identity-based policy allows the s3:ListAllMyBuckets action
command terminated with exit code 254

 

 

2. Service Account Token 정보 확인

# 서비스 어카운트 토큰 확인
SA_TOKEN=$(kubectl exec -it eks-iam-test2 -- cat /var/run/secrets/kubernetes.io/serviceaccount/token)
echo $SA_TOKEN

# jwt 혹은 아래 JWT 웹 사이트 이용 https://jwt.io/
jwt decode $SA_TOKEN --json --iso8601



#헤더
{
  "alg": "RS256",
  "kid": "1a8fcaee12b3a8f191327b5e9b997487ae93baab"
}

# 페이로드 : OAuth2에서 쓰이는 aud, exp 속성 확인! > projectedServiceAccountToken 기능으로 토큰에 audience,exp 항목을 덧붙힘
## iss 속성 : EKS OpenID Connect Provider(EKS IdP) 주소 > 이 EKS IdP를 통해 쿠버네티스가 발급한 토큰이 유요한지 검증

 

 

실습  3 ( IRSA 설정)

초기 EKS 배포 시 iam 설정 내 withOIDC가 true로 설정되어 있어 IRSA를 사용 할 수 있음

 

1. IRSA 생성 - Service Account와 IAM Role ARN 맵핑

# Create an iamserviceaccount - AWS IAM role bound to a Kubernetes service account
eksctl create iamserviceaccount \
  --name my-sa \
  --namespace default \
  --cluster $CLUSTER_NAME \
  --approve \
  --attach-policy-arn $(aws iam list-policies --query 'Policies[?PolicyName==`AmazonS3ReadOnlyAccess`].Arn' --output text)

# 확인 >> 웹 관리 콘솔에서 CloudFormation Stack >> IAM Role 확인
# aws-load-balancer-controller IRSA는 어떤 동작을 수행할 것 인지 생각해보자!
eksctl get iamserviceaccount --cluster $CLUSTER_NAME

NAMESPACE	NAME				ROLE ARN
default		my-sa				arn:aws:iam::890742582773:role/eksctl-myeks-addon-iamserviceaccount-default--Role1-1qnld9NsJg9w
kube-system	aws-load-balancer-controller	arn:aws:iam::890742582773:role/eksctl-myeks-addon-iamserviceaccount-kube-sys-Role1-uXQZrR2aq8af

# Inspecting the newly created Kubernetes Service Account, we can see the role we want it to assume in our pod.
kubectl get sa
kubectl describe sa my-sa
Name:                my-sa
Namespace:           default
Labels:              app.kubernetes.io/managed-by=eksctl
Annotations:         eks.amazonaws.com/role-arn: arn:aws:iam::890742582773:role/eksctl-myeks-addon-iamserviceaccount-default--Role1-1qnld9NsJg9w
Image pull secrets:  <none>
Mountable secrets:   <none>
Tokens:              <none>
Events:              <none>

 

 

2. Pod 생성 ( Pod 생성 시 my-sa Service Account 지정 )

# 파드3번 생성
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: eks-iam-test3
spec:
  serviceAccountName: my-sa
  containers:
    - name: my-aws-cli
      image: amazon/aws-cli:latest
      command: ['sleep', '36000']
  restartPolicy: Never
  terminationGracePeriodSeconds: 0
EOF

# 해당 SA를 파드가 사용 시 mutatingwebhook으로 Env,Volume 추가함 : AWS IAM 역할을 Pod에 자동으로 주입
kubectl get mutatingwebhookconfigurations pod-identity-webhook -o yaml

# 파드 생성 yaml에 없던 내용이 추가됨!!!!!
# Pod Identity Webhook은 mutating webhook을 통해 아래 Env 내용과 1개의 볼륨을 추가함
kubectl get pod eks-iam-test3
kubectl get pod eks-iam-test3 -o yaml
...
    volumeMounts: 
    - mountPath: /var/run/secrets/eks.amazonaws.com/serviceaccount
      name: aws-iam-token
      readOnly: true
  ...
  volumes: 
  - name: aws-iam-token
    projected: 
      sources: 
      - serviceAccountToken: 
          audience: sts.amazonaws.com
          expirationSeconds: 86400
          path: token
...

kubectl exec -it eks-iam-test3 -- ls /var/run/secrets/eks.amazonaws.com/serviceaccount
token

kubectl exec -it eks-iam-test3 -- cat /var/run/secrets/eks.amazonaws.com/serviceaccount/token ; echo
...

kubectl describe pod eks-iam-test3
...
Environment:
      AWS_STS_REGIONAL_ENDPOINTS:   regional
      AWS_DEFAULT_REGION:           ap-northeast-2
      AWS_REGION:                   ap-northeast-2
      AWS_ROLE_ARN:                 arn:aws:iam::911283464785:role/eksctl-myeks-addon-iamserviceaccount-default-Role1-GE2DZKJYWCEN
      AWS_WEB_IDENTITY_TOKEN_FILE:  /var/run/secrets/eks.amazonaws.com/serviceaccount/token
    Mounts:
      /var/run/secrets/eks.amazonaws.com/serviceaccount from aws-iam-token (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-69rh8 (ro)
...
Volumes:
  aws-iam-token:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  86400
  kube-api-access-sn467:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
...

# 파드에서 aws cli 사용 확인
eksctl get iamserviceaccount --cluster $CLUSTER_NAME
kubectl exec -it eks-iam-test3 -- aws sts get-caller-identity --query Arn
"arn:aws:sts::911283464785:assumed-role/eksctl-myeks-addon-iamserviceaccount-default-Role1-GE2DZKJYWCEN/botocore-session-1685179271"

# 되는 것고 안되는 것은 왜그런가?
kubectl exec -it eks-iam-test3 -- aws s3 ls
kubectl exec -it eks-iam-test3 -- aws ec2 describe-instances --region ap-northeast-2
kubectl exec -it eks-iam-test3 -- aws ec2 describe-vpcs --region ap-northeast-2

 

정리: 실습1, 실습2의 경우 Service Account 사용 또는 미사용과 상관없이 Pod가 S3에 대한 IAM Role이 없기 때문에 명령어를 사용하지 못하고 있다.

실습3경우 Service Account와 S3 IAM Role를 맵핑 후 해당 Service Account로 Pod를 생성하여 정상적으로 S3를 조회하는 것을 확인 할 수 있다. (IRSA)

 

EKS Pod Identity

 

IRSA의 경우, OIDC 앤드포인트가 공개( Public )으로 되어 있고, 신뢰 정책을 엄하게 설정하지 않으면 보안에 취약 할 수있음.

 

 

 

실습  -  "eks-pod-identity-agent 설치"

 

1. eks-pod-identity-agent 설치

#
ADDON=eks-pod-identity-agent
aws eks describe-addon-versions \
    --addon-name $ADDON \
    --kubernetes-version 1.31 \
    --query "addons[].addonVersions[].[addonVersion, compatibilities[].defaultVersion]" \
    --output text
v1.2.0-eksbuild.1
True
v1.1.0-eksbuild.1
False
v1.0.0-eksbuild.1
False

# 모니터링
watch -d kubectl get pod -A

# 설치
aws eks create-addon --cluster-name $CLUSTER_NAME --addon-name eks-pod-identity-agent
혹은
eksctl create addon --cluster $CLUSTER_NAME --name eks-pod-identity-agent --version 1.3.5

# 확인
eksctl get addon --cluster $CLUSTER_NAME
kubectl -n kube-system get daemonset eks-pod-identity-agent
kubectl -n kube-system get pods -l app.kubernetes.io/name=eks-pod-identity-agent
kubectl get ds -n kube-system eks-pod-identity-agent -o yaml
...
      containers: 
      - args: 
        - --port
        - "80"
        - --cluster-name
        - myeks
        - --probe-port
        - "2703"
        command: 
        - /go-runner
        - /eks-pod-identity-agent
        - server
      ....
      ports: 
        - containerPort: 80
          name: proxy
          protocol: TCP
        - containerPort: 2703
          name: probes-port
          protocol: TCP
      ...
        securityContext: 
          capabilities: 
            add: 
            - CAP_NET_BIND_SERVICE
      ...
      hostNetwork: true
...

# 네트워크 정보 확인
## EKS Pod Identity Agent uses the hostNetwork of the node and it uses port 80 and port 2703 on a link-local address on the node. 
## This address is 169.254.170.23 for IPv4 and [fd00:ec2::23] for IPv6 clusters.
for node in $N1 $N2 $N3; do ssh ec2-user@$node sudo ss -tnlp | grep eks-pod-identit; echo "-----";done
for node in $N1 $N2 $N3; do ssh ec2-user@$node sudo ip -c route; done
for node in $N1 $N2 $N3; do ssh ec2-user@$node sudo ip -c -br -4 addr; done
for node in $N1 $N2 $N3; do ssh ec2-user@$node sudo ip -c addr; done

 

2. podidentityassociation 설정

# 
eksctl create podidentityassociation \
--cluster $CLUSTER_NAME \
--namespace default \
--service-account-name s3-sa \
--role-name s3-eks-pod-identity-role \
--permission-policy-arns arn:aws:iam::aws:policy/AmazonS3ReadOnlyAccess \
--region ap-northeast-2

# 확인
kubectl get sa
eksctl get podidentityassociation --cluster $CLUSTER_NAME
ASSOCIATION ARN											                                                      NAMESPACE	SERVICE ACCOUNT NAME	IAM ROLE ARN
arn:aws:eks:ap-northeast-2:911283464785:podidentityassociation/myeks/a-blaanudo8dc1dbddw	default		s3-sa			            arn:aws:iam::911283464785:role/s3-eks-pod-identity-role

aws eks list-pod-identity-associations --cluster-name $CLUSTER_NAME | jq
{
  "associations": [
    {
      "clusterName": "myeks",
      "namespace": "default",
      "serviceAccount": "s3-sa",
      "associationArn": "arn:aws:eks:ap-northeast-2:911283464785:podidentityassociation/myeks/a-pm07a3bg79bqa3p24",
      "associationId": "a-pm07a3bg79bqa3p24"
    }
  ]
}

# ABAC 지원을 위해 sts:Tagsession 추가
aws iam get-role --query 'Role.AssumeRolePolicyDocument' --role-name s3-eks-pod-identity-role | jq .
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "pods.eks.amazonaws.com"
      },
      "Action": [
        "sts:AssumeRole",
        "sts:TagSession"
      ]
    }
  ]
}

 

3. SA 생성 및  Pod 생성 및 확인

# 서비스어카운트, 파드 생성
kubectl create sa s3-sa

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: eks-pod-identity
spec:
  serviceAccountName: s3-sa
  containers:
    - name: my-aws-cli
      image: amazon/aws-cli:latest
      command: ['sleep', '36000']
  restartPolicy: Never
  terminationGracePeriodSeconds: 0
EOF

#
kubectl get pod eks-pod-identity -o yaml | kubectl neat
kubectl exec -it eks-pod-identity -- aws sts get-caller-identity --query Arn
"arn:aws:sts::890742582773:assumed-role/s3-eks-pod-identity-role/eks-myeks-eks-pod-id-0b9a7592-b56b-48ca-9f66-7a7c3a94ee63"

kubectl exec -it eks-pod-identity -- aws s3 ls
2025-03-15 19:07:44 yjsong
kubectl exec -it eks-pod-identity -- env | grep AWS

AWS_STS_REGIONAL_ENDPOINTS=regional
AWS_DEFAULT_REGION=ap-northeast-2
AWS_REGION=ap-northeast-2
AWS_CONTAINER_CREDENTIALS_FULL_URI=http://169.254.170.23/v1/credentials
AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE=/var/run/secrets/pods.eks.amazonaws.com/serviceaccount/eks-pod-identity-token

# 토큰 정보 확인
kubectl exec -it eks-pod-identity -- ls /var/run/secrets/pods.eks.amazonaws.com/serviceaccount/
kubectl exec -it eks-pod-identity -- cat /var/run/secrets/pods.eks.amazonaws.com/serviceaccount/eks-pod-identity-token

• 고려사항
  • SDK 최신 버전 확인
  • 워커노드에 IAM Policy 확인 : Action(eks-auth:AssumeRoleForPodIdentity)
  • 보안 솔루션으로 링크 로컬 주소 사용 가능 여부 확인, 혹은 이미 사용 중인 주소인지, iptables 로 막혀있는지 확인
  • How to migrate from IRSA to EKS Pod Identity : 기존 IRSA → PodIdentity 마이그레이션