Install ExternalDNS to an EKS cluster with CDK, and register Service or Ingress host to Route53

2024-03-20 aws kubernetes

ExternalDNS is an application that synchronizes Service or Ingress host with external DNS records. Route53 records can also be created using CDK, but it is hard to handle ELBs that are created asynchronously after cluster.addManifest() and are managed by controllers.

First, let’s create a Role to edit records.

const externalDNSRole = new iam.Role(scope, 'ExternalDNSRole', {
  roleName: `ExternalDNSRole-${cluster.clusterName}`,
  assumedBy: new iam.WebIdentityPrincipal(
    cluster.openIdConnectProvider.openIdConnectProviderArn,
    {
      "StringEquals": new cdk.CfnJson(scope, 'ExternalDNSRoleStringEquals', { value: {
        [`${cluster.clusterOpenIdConnectIssuer}:aud`]: 'sts.amazonaws.com',
        [`${cluster.clusterOpenIdConnectIssuer}:sub`]: 'system:serviceaccount:external-dns:external-dns-sa'
      }}),
    },
  ),
  inlinePolicies: {
    'Route53': new iam.PolicyDocument({
      statements: [
        new iam.PolicyStatement({
          actions: [
            'route53:ChangeResourceRecordSets',
          ],
          resources: [
            'arn:aws:route53:::hostedzone/*'
          ]
        }),
        new iam.PolicyStatement({
          actions: [
            'route53:ListHostedZones',
            'route53:ListResourceRecordSets',
            'route53:ListTagsForResource'
          ],
          resources: [
            '*'
          ]
        })
      ]
    })
  }
})

I registered some records to check the behavior in the existing zone.

Specify the target zone in domainFilters, and install ExternalDNS with Helm. If multiple ExternalDNS refer to the same domain, txtOwnerId must be set to a unique value, otherwise they will try to delete records from each other because modified records are chosen by txtOwnerId.

const externalDns = cluster.addHelmChart('ExternalDNS', {
  release: 'external-dns',
  chart: 'external-dns',
  repository: 'https://kubernetes-sigs.github.io/external-dns',
  version: '1.14.3',
  namespace: 'external-dns',
  createNamespace: true,
  values: {
    serviceAccount: {
      name: "external-dns-sa",
      annotations: {
        'eks.amazonaws.com/role-arn': externalDNSRole.roleArn
      }
    },
    sources: [
      'service',
      'ingress'
    ],
    domainFilters: ['externaldns.test'],
    provider: 'aws',
    policy: 'sync', // or 'upsert-only'
    awsZoneType: 'public',
    registry: 'txt', // Specify the registry for storing ownership and labels
    txtOwnerId: 'external-dns', // Specify an identifier for this instance of ExternalDNS
  },
  wait: true,
})

if (cluster.albController) {
  externalDns.node.addDependency(cluster.albController)
}

Specifying Service’s external-dns.alpha.kubernetes.io/hostname annotations or Ingress’s rule.host,

apiVersion: v1
kind: Service
metadata:
  name: nginx
  annotations:
	  service.beta.kubernetes.io/aws-load-balancer-type: external
    external-dns.alpha.kubernetes.io/hostname: cccc.externaldns.test
spec:
  type: LoadBalancer
  ports:
  - port: 80
    name: http
    targetPort: 80
  selector:
    app: nginx

A record and TXT records like “heritage=external-dns,external-dns/owner=external-dns,external-dns/resource=service/default/nginx” are created.

Desired change: CREATE cccc.externaldns.test A [Id: /hostedzone/****]
Desired change: CREATE cccc.externaldns.test TXT [Id: /hostedzone/****]
Desired change: CREATE cname-cccc.externaldns.test TXT [Id: /hostedzone/****]
3 record(s) in zone externaldns.test. [Id: /hostedzone/****] were successfully updated
Applying provider record filter for domains: [externaldns.test. .externaldns.test.]
All records are already up to date

ALB managed by Load Balancer Controller uses Host header for routing.

Install AWS Load Balancer Controller on EKS cluster and set up ALB Ingress - sambaiz-net

If policy is “sync”, when you delete a resource, the records are deleted, but the existing records remain.

Desired change: DELETE cccc.externaldns.test A [Id: /hostedzone/****]
Desired change: DELETE cccc.externaldns.test TXT [Id: /hostedzone/****]
Desired change: DELETE cname-cccc.externaldns.test TXT [Id: /hostedzone/****]
3 record(s) in zone externaldns.test. [Id: /hostedzone/****] were successfully updated
All records are already up to date

To register a record with the same name, you need to set external-dns.alpha.kubernetes.io/set-identifier as well as external-dns.alpha.kubernetes.io/aws-weight etc.

If existing record’s routing policy is different, it occurs error.

Failure in zone externaldns.test. [Id: /hostedzone/*****] when submitting change batch: InvalidChangeBatch: [RRSet with DNS name bbbb.externaldns.test., type A, SetIdentifier nginx cannot be created as a non-weighted set exists with the same name and type.]\n\tstatus code: 400