EKS + Fargate + proxy環境でGitHub Actionsのself-hosted runnerの作成　#doda Developer Group Advent Calendar 2024

#doda Developer Group Advent Calendar 2024

doda Developer Group Advent Calendar 2024 の６日目担当の齋藤です。

EKS と Fargate を使って Actions Runner Controller (以降 ARC) を作るというものです。ただしこの検証の結果、現時点では Actions 内で Docker を起動できないことが分かりましたので、私たちの用途においては実運用までは至っておりません。

前提

GitHub Enterprise Server 3.13
Amazon EKS: Kubernetes 1.31
actions/gha-runner-scale-set-controller:0.9.3
actions/actions-runner:2.320.0

構築

AWSリソースの作成

全て CloudFormation で作ります。

Role

EKS の起動で利用します。

AWSTemplateFormatVersion: '2010-09-09'

Resources:

  ExecutionRole:
    Type: AWS::IAM::Role
    Properties:
      RoleName: runner-execution-role
      AssumeRolePolicyDocument:
        Statement:
          - Effect: Allow
            Action: sts:AssumeRole
            Principal:
              Service: eks.amazonaws.com
      ManagedPolicyArns:
        - arn:aws:iam::aws:policy/AmazonEKSClusterPolicy
        - arn:aws:iam::aws:policy/AmazonEKSServicePolicy
      Path: /
      Policies:
        - PolicyName: runner-execution-logging-policy
          PolicyDocument:
            Statement:
              - Effect: Allow
                Action:
                  - logs:CreateLogStream
                  - logs:CreateLogGroup
                  - logs:PutLogEvents
                  - logs:PutRetentionPolicy
                  - logs:DescribeLogStreams
                Resource: "*"

  FargateProfileRole:
    Type: AWS::IAM::Role
    Properties:
      RoleName: runner-fargate-profile-role
      AssumeRolePolicyDocument:
        Statement:
          - Effect: Allow
            Action: sts:AssumeRole
            Principal:
              Service: eks-fargate-pods.amazonaws.com
      ManagedPolicyArns:
        - arn:aws:iam::aws:policy/AmazonEKSFargatePodExecutionRolePolicy
      Path: /
      PermissionsBoundary: arn:aws:iam::123456789012:policy/ChildAccountRoleBoundary

ECR

ARCで利用するイメージを格納します。

AWSTemplateFormatVersion: '2010-09-09'

Resources:
  GhaRunnerScaleSetControllerECR:
    Type: AWS::ECR::Repository
    Properties:
      RepositoryName: actions/gha-runner-scale-set-controller
      ImageTagMutability: MUTABLE

  ActionsRunnerECR:
    Type: AWS::ECR::Repository
    Properties:
      RepositoryName: actions/actions-runner
      ImageTagMutability: MUTABLE

以下のイメージを pull して作成した ECR に push しておきます。 - ghcr.io/actions/gha-runner-scale-set-controller:0.9.3 - ghcr.io/actions/actions-runner:2.320.0

EFS

ARCの Kubernetes モードで利用します。

AWSTemplateFormatVersion: "2010-09-09"
Description: Amazon EFS file system for GitHub Actions Runner

Resources:
  MountTargetSecurityGroup:
    Type: AWS::EC2::SecurityGroup
    Properties:
      VpcId: vpc-12345
      GroupDescription: Security group for mount target
      SecurityGroupIngress:
        - IpProtocol: tcp
          FromPort: 2049
          ToPort: 2049
          CidrIp: 10.0.1.0/20
  FileSystem:
    Type: AWS::EFS::FileSystem
    Properties:
      PerformanceMode: generalPurpose
      FileSystemTags:
        - Key: Name
          Value: github-actions-runner-efs
  MountTarget1a:
    Type: AWS::EFS::MountTarget
    Properties:
      FileSystemId:
        Ref: FileSystem
      SubnetId: subnet-1111
      SecurityGroups:
        - Ref: MountTargetSecurityGroup
  MountTarget1c:
    Type: AWS::EFS::MountTarget
    Properties:
      FileSystemId:
        Ref: FileSystem
      SubnetId: subnet-2222
      SecurityGroups:
        - Ref: MountTargetSecurityGroup

EKS構築

EKS は eksctl で作成します。

apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

metadata:
  name: github-actions-runnner-v1-31
  region: ap-northeast-1
  version: "1.31"

iam:
  serviceRoleARN: arn:aws:iam::123456789012:role/runner-execution-role
  fargatePodExecutionRoleARN: arn:aws:iam::123456789012:role/runner-fargate-profile-role
  withOIDC: true
  serviceAccounts:
    - metadata:
        name: aws-node
        namespace: kube-system
      attachPolicyARNs:
        - arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy

iamIdentityMappings:
  - arn: arn:aws:iam::123456789012:role/runner-execution-role
    username: arn:aws:iam::123456789012:role/runner-execution-role
    groups:
      - system:masters
  - arn: arn:aws:iam::123456789012:role/runner-fargate-profile-role
    username: system:node:{{SessionName}}
    groups:
      - system:bootstrappers
      - system:nodes
      - system:node-proxier

vpc:
  id: vpc-12345
  cidr: 10.0.1.0/20
  extraCIDRs:
    - 10.0.1.0/20
  subnets:
    public:
      ap-northeast-1a:
        id: subnet-1111
      ap-northeast-1c:
        id: subnet-2222
  manageSharedNodeSecurityGroupRules: true
  nat:
    gateway: Disable
  clusterEndpoints:
    privateAccess: true
    publicAccess: true

addons:
  - name: coredns
    version: v1.11.3-eksbuild.1
    configurationValues: "{ \"replicaCount\":1 }"
    resolveConflicts: overwrite
  - name: vpc-cni
    version: v1.18.5-eksbuild.1
    resolveConflicts: overwrite
  - name: kube-proxy
    version: v1.30.3-eksbuild.5
    resolveConflicts: overwrite

fargateProfiles:
  - name: default-profile
    podExecutionRoleARN: arn:aws:iam::123456789012:role/runner-fargate-profile-role
    selectors:
      - namespace: kube-system
      - namespace: arc-systems
      - namespace: arc-runners
    subnets:
      - subnet-1111
      - subnet-2222

Kubernetesリソースの作成

基本的には GitHub の公式ドキュメント通りに適用していきます。

docs.github.com

Actions Runner Controllerのinstall

image:
  repository: 123456789012.dkr.ecr.ap-northeast-1.amazonaws.com/actions/gha-runner-scale-set-controller
  tag: 0.9.3
env:
  - name: "http_proxy"
    value: (プロキシのURL)
  - name: "https_proxy"
    value: (プロキシのURL)
  - name: "no_proxy"
    value: "172.20.0.0/16,localhost,127.0.0.1,10.0.1.0/20,169.254.169.254,.internal,api.ecr.ap-northeast-1.amazonaws.com,dkr.ecr.ap-northeast.amazonaws.com"

$ helm install arc --namespace arc-systems \
  --create-namespace oci://ghcr.io/actions/actions-runner-controller-charts/gha-runner-scale-set-controller \
  --values runner-scale-set-controller-helm-value.yaml

Secretの作成

GitHub App を使って連携するためにシークレットを保存します。

$ kubectl create namespace arc-runners
$ kubectl create secret generic pre-defined-secret \
    --namespace=arc-runners \
    --from-literal=github_app_id=123456 \
    --from-literal=github_app_installation_id=654321 \
    --from-literal=github_app_private_key='-----BEGIN RSA PRIVATE KEY-----
...
-----END RSA PRIVATE KEY-----'

Storageの作成

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: efs-sc
provisioner: efs.csi.aws.com
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: efs-pv
  labels:
    type: efs-pv
spec:
  capacity:
    storage: 5Gi
  accessModes:
    - ReadWriteMany
  storageClassName: efs-sc
  persistentVolumeReclaimPolicy: Retain
  csi:
    driver: efs.csi.aws.com
    volumeHandle: fs-1234
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: actions-pvc
  namespace: arc-runners
spec:
  selector:
    matchLabels:
      type: efs-pv
  accessModes:
    - ReadWriteMany
  volumeMode: Filesystem
  resources:
    requests:
      storage: 1Gi
  storageClassName: efs-sc

runner scale set

githubConfigUrl: https://<your_enterprise/org/repo>
githubConfigSecret: pre-defined-secret
containerMode:
  type: "kubernetes"
  kubernetesModeWorkVolumeClaim:
    accessModes: ["ReadWriteMany"]
    storageClassName: "efs-sc"
  resources:
    requests:
    storage: 1Gi

template:
  spec:
    initContainers:
      - name: kube-init
        image: 123456789012.dkr.ecr.ap-northeast-1.amazonaws.com/actions/actions-runner:2.320.0
        command: ["sudo", "chown", "-R", "1001:123", "/home/runner/_work"]
        volumeMounts:
          - name: work
        mountPath: /home/runner/_work
    containers:
      - name: runner
        image: 123456789012.dkr.ecr.ap-northeast-1.amazonaws.com/actions/actions-runner:2.320.0
        command: ["/home/runner/run.sh"]
        resources:
          requests:
            cpu: 2
            memory: 4G
          limits:
            cpu: 2
            memory: 4G
        env:
          - name: ACTIONS_RUNNER_CONTAINER_HOOKS
            value: /home/runner/k8s/index.js
          - name: ACTIONS_RUNNER_POD_NAME
            valueFrom:
              fieldRef:
                fieldPath: metadata.name
          - name: ACTIONS_RUNNER_REQUIRE_JOB_CONTAINER
            value: "false"
        volumeMounts:
          - name: work
            mountPath: /home/runner/_work
    volumes:
      - name: work
        persistentVolumeClaim:
          claimName: actions-pvc
proxy:
   http:
     url: (プロキシのURL)
   https:
     url: (プロキシのURL)
   noProxy:
     - 172.20.0.0/16
     - localhost
     - 127.0.0.1
     - 10.0.1.0/20
     - 169.254.169.254
     - .internal
     - .ap-northeast.amazonaws.com
     - .ap-northeast-1.amazonaws.com
     - api.ecr.ap-northeast-1.amazonaws.com
     - dkr.ecr.ap-northeast.amazonaws.com

$ helm install arc-runner-set \
    --namespace arc-runners \
    --create-namespace \
    --values actions-runner-helm-value.yaml \
    oci://ghcr.io/actions/actions-runner-controller-charts/gha-runner-scale-set

詰まったポイント

EFSのVolumeのマウントに失敗

Runner Pod 起動時に EFS の Volume のマウントに失敗しました。

エラー内容

  Warning  FailedAttachVolume  36s    attachdetach-controller  AttachVolume.Attach failed for volume "efs-pv" : timed out waiting for external-attacher of efs.csi.aws.com CSI driver to attach volume fs-1234

最新の EKS では StorageClass が入っているはずなのですが、なぜか起動時に作成されなかったため自分で追加しています。また、公式ドキュメントを参考にすると apiVersion が古いため、 storage.k8s.io/v1 に書き換えが必要でした。

New – AWS Fargate for Amazon EKS now supports Amazon EFS | AWS News Blogaws.amazon.com

Runenr PodでPermissionエラー

Runner 起動時に Permission エラーが発生。

エラー内容

System.UnauthorizedAccessException: Access to the path '/home/runner/_work/_tool' is denied.
 ---> System.IO.IOException: Permission denied
   --- End of inner exception stack trace ---
   at System.IO.FileSystem.CreateDirectory(String fullPath)
   at System.IO.Directory.CreateDirectory(String path)
   at GitHub.Runner.Worker.JobRunner.RunAsync(AgentJobRequestMessage message, CancellationToken jobRequestCancellationToken)
   at GitHub.Runner.Worker.JobRunner.RunAsync(AgentJobRequestMessage message, CancellationToken jobRequestCancellationToken)
   at GitHub.Runner.Worker.Worker.RunAsync(String pipeIn, String pipeOut)
   at GitHub.Runner.Worker.Program.MainAsync(IHostContext context, String[] args)

Issue で同エラーを見つけて securityContext を設定したものの解消せずでした。

github.com

しかし公式ドキュメントをよく見るとちゃんと書いてあり、initContainers で chown で所有者を明示することで解消しました。

docs.github.com

Dockerが存在しないエラー

Runner 起動時に以下の Docker が存在しないエラーが発生しました。

> Connection to the Docker daemon at '/var/run/docker.sock' failed with error "[2] No such file or directory"; ensure the Docker daemon is running and accessible

ARCでは containerMode が指定可能ですが、Fargate では特権コンテナ(privileged)はサポートされていません。そのためkubernetesモードでの実行が必要です。

docs.github.com

しかし Docker in Docker を使いたい場合には dindモードでの実行が必要でした。代替となるコンテナ管理ツールを使えば実現可能かもしれませんが私たちの用途とは異なるため試しておりません。

まとめ

私たちは採用には至りませんでしたが、Actions 内で Docker さえ使わなければ EKS + Fargate で self-hosted runner を動かすことは十分に可能です。ただし EKS + Fargate に関する記事や ARCに関する記事があまり Web 上も多くは見つからず、また社内の制約もあり記載した内容以外にも多くの詰まりポイントがありました。今後同じように構築をする方の参考になれば幸いです。