When it doesn’t work we blame DNS. DNS is hard. Add Kubernetes to the mix and your eyes get watery, heart rate increases, palms get sweaty… and you just walk away before you risk losing anymore time in your life.

Then you came across this blog post, and had some hope. Hoping that someone, somewhere solved this problem.

Well I hope this is that blog post. The problem I solved is having DNS entries for load balanced services and ingress resources automatically be made available to my homelab network, outside of Kubernetes.

I already solved the standard homelab DNS problem. Any VM or new device that connects to my home network and advertises a hostname, will be immediately discoverable by name. Now I wanted to take it a step further. I like Kubernetes, and its easier for me to push new things to my homelab K8s cluster than creating a new VM. So it was natural that I wanted to solve this problem so I can put things in K8s and hit it by name without having to do anything else on my network.

In comes external-dns for Kubernetes… a well known optional component for Kubernetes. Its role is to update DNS records inside an external provider based on available ingress and load balanced service resources. Every minute it will scan your cluster for new or updated resources, check against an internal cache, and update your DNS provider.

Perfect, now I just need a DNS provider supported by external-dns. I decided to use PowerDNS for this. This DNS service uses MySQL for zone setup, has an API, and community supported GUIs for that API. Nothing fancy, and it can all be controlled without too much pain… as a Kubernetes resources. Yes that’s right. I’m going to run my DNS provider for external-dns, inside of K8s itself… because why not?

The first part was to cobble together all the Kubernetes manifests required to install PowerDNS. Helm to the rescue. I found a community created chart. It came close to everything I needed but still missed a few things. So I cloned it and made my own Helm chart. The chart installs PowerDNS with a single pod MariaDB and persistence storage disabled. The idea being if my PowerDNS deployment falls apart, external-dns will re-sync it within 60s of restart, so we don’t need any form of real persistence.

To make this all work, PowerDNS needs to have knowledge of the domain(s) it’s going to manage entries for. This domain should end up being a sub-domain of your main network. For this post, I’m going to assume your main network router is configured for a domain like mydomain.house and all your home devices use your router for DNS resolution. Knowing that we are going to setup PowerDNS to manage a domain called k8s.mydomain.house.

The Helm chart I created takes a list of domains you want to have PowerDNS manage and it will configure them on startup. Since there are some api keys and passwords involved in all this, you also need to set a few more details, and in the end, you have a values file for the Helm chart that looks like this:

powerdns:
  api:
    key: SOMETHING_ANYTHING
  initDomains: 
    - k8s.mydomain.house 

service:
  annotations:
    metallb.universe.tf/allow-shared-ip: powerdns  
  type: LoadBalancer
  ip: 192.168.1.200

mariadb:
  rootUser:
    password: A_PASSWORD

So a couple of things to note in that block, particularly the service section. I have an annotation specified. This is for MetalLB (which I use as a LoadBalancer) to allow the same IP to share multiple service resources on the same port, which is needed for TCP and UDP resolution on DNS.

With that yaml saved in a file you can install it all like this:

helm repo add puckpuck https://puckpuck.github.io/helm-charts
helm install powerdns puckpuck/powerdns --values my-values.yaml

PowerDNS doesn’t have a UI. I have an HTML file that sits on my hard drive, when I open it, I type in the URL for PowerDNS and my API key, and that’s my UI. I got the file from here: https://github.com/james-stevens/powerdns-webui You can find the actual html file in the htdocs folder of that repo. One day I might actual add an NGiNX pod with this file as part of my PowerDNS Helm chart, but alas, here I am writing a blog about it instead.

Now that we have PowerDNS setup, next is for external-dns. Luckily, the fine folks at Bitnami have created such a Helm chart, and it does exactly what we need. Here’s what I used as my values yaml file.

provider: pdns
domainFilters:
  - k8s.mydomain.house
txtOwnerId: k8s

pdns:
  apiUrl: http://powerdns-api
  apiPort: 8081
  apiKey: SOMETHING_ANYTHING

Then to install the chart I ran this

helm repo add bitnami https://charts.bitnami.com/bitnami
helm install external-dns bitnami/external-dns --values external-dns-values.yaml

Now we have external-dns going out on a periodic basis (every minute) finding all your services and ingress resources, checking to see if they have the external-dns annotation, then syncing that list with PowerDNS.

Almost done.

Now we need to configure your primary DNS to send any request for your new subdomain off to PowerDNS for resolution. If you read this far and don’t have an Ubiquiti router…. I’m sorry. If you have an Ubiquiti EdgeRouter you’re in luck because you only need to do one more setting 🙂

Inside the EdgeMax UI, go to the Config Tree tab, then expand service -> dns -> forwarding. From here you will click the Add button for options and set the new option to the following. Note the IP address here should match what you setup when you configured PowerDNS.

server=/k8s.mydomain.house/192.168.1.200

Click Preview on the bottom of the screen then Apply for the changes to take effect.

If you followed my instructions on how to properly setup home DNS, then all devices should get DNS resolution configured via DHCP, meaning your router is the only DNS server your home devices look for. With that being the case, when you try to DNS resolve anything under the k8s.mydomain.house domain (or whatever you configured) it will be sent to PowerDNS for resolution.

With all this in place you should now be able to network reach any Kubernetes ingress or service configured with the external-dns annotation, from anywhere in your network.

To test this out you can set the hostname property on an Ingress rule, or add this annotation to any load balanced service: external-dns.alpha.kubernetes.io/hostname

The hostname specified for the Ingress rule or the Service annotation should fall within your configured sub domain. For example: foo.k8s.mydomain.house