Tag Archives: dns

Conquering DNS in a Kubernetes homelab

1 Reply

When it doesn’t work we blame DNS. DNS is hard. Add Kubernetes to the mix and your eyes get watery, heart rate increases, palms get sweaty… and you just walk away before you risk losing anymore time in your life.

Then you came across this blog post, and had some hope. Hoping that someone, somewhere solved this problem.

Well I hope this is that blog post. The problem I solved is having DNS entries for load balanced services and ingress resources automatically be made available to my homelab network, outside of Kubernetes.

I already solved the standard homelab DNS problem. Any VM or new device that connects to my home network and advertises a hostname, will be immediately discoverable by name. Now I wanted to take it a step further. I like Kubernetes, and its easier for me to push new things to my homelab K8s cluster than creating a new VM. So it was natural that I wanted to solve this problem so I can put things in K8s and hit it by name without having to do anything else on my network.

In comes external-dns for Kubernetes… a well known optional component for Kubernetes. Its role is to update DNS records inside an external provider based on available ingress and load balanced service resources. Every minute it will scan your cluster for new or updated resources, check against an internal cache, and update your DNS provider.

Perfect, now I just need a DNS provider supported by external-dns. I decided to use PowerDNS for this. This DNS service uses MySQL for zone setup, has an API, and community supported GUIs for that API. Nothing fancy, and it can all be controlled without too much pain… as a Kubernetes resources. Yes that’s right. I’m going to run my DNS provider for external-dns, inside of K8s itself… because why not?

The first part was to cobble together all the Kubernetes manifests required to install PowerDNS. Helm to the rescue. I found a community created chart. It came close to everything I needed but still missed a few things. So I cloned it and made my own Helm chart. The chart installs PowerDNS with a single pod MariaDB and persistence storage disabled. The idea being if my PowerDNS deployment falls apart, external-dns will re-sync it within 60s of restart, so we don’t need any form of real persistence.

To make this all work, PowerDNS needs to have knowledge of the domain(s) it’s going to manage entries for. This domain should end up being a sub-domain of your main network. For this post, I’m going to assume your main network router is configured for a domain like mydomain.house and all your home devices use your router for DNS resolution. Knowing that we are going to setup PowerDNS to manage a domain called k8s.mydomain.house.

The Helm chart I created takes a list of domains you want to have PowerDNS manage and it will configure them on startup. Since there are some api keys and passwords involved in all this, you also need to set a few more details, and in the end, you have a values file for the Helm chart that looks like this:

powerdns:
  api:
    key: SOMETHING_ANYTHING
  initDomains: 
    - k8s.mydomain.house 

service:
  annotations:
    metallb.universe.tf/allow-shared-ip: powerdns  
  type: LoadBalancer
  ip: 192.168.1.200

mariadb:
  rootUser:
    password: A_PASSWORD

So a couple of things to note in that block, particularly the service section. I have an annotation specified. This is for MetalLB (which I use as a LoadBalancer) to allow the same IP to share multiple service resources on the same port, which is needed for TCP and UDP resolution on DNS.

With that yaml saved in a file you can install it all like this:

helm repo add puckpuck https://puckpuck.github.io/helm-charts
helm install powerdns puckpuck/powerdns --values my-values.yaml

PowerDNS doesn’t have a UI. I have an HTML file that sits on my hard drive, when I open it, I type in the URL for PowerDNS and my API key, and that’s my UI. I got the file from here: https://github.com/james-stevens/powerdns-webui You can find the actual html file in the htdocs folder of that repo. One day I might actual add an NGiNX pod with this file as part of my PowerDNS Helm chart, but alas, here I am writing a blog about it instead.

Now that we have PowerDNS setup, next is for external-dns. Luckily, the fine folks at Bitnami have created such a Helm chart, and it does exactly what we need. Here’s what I used as my values yaml file.

provider: pdns
domainFilters:
  - k8s.mydomain.house
txtOwnerId: k8s

pdns:
  apiUrl: http://powerdns-api
  apiPort: 8081
  apiKey: SOMETHING_ANYTHING

Then to install the chart I ran this

helm repo add bitnami https://charts.bitnami.com/bitnami
helm install external-dns bitnami/external-dns --values external-dns-values.yaml

Now we have external-dns going out on a periodic basis (every minute) finding all your services and ingress resources, checking to see if they have the external-dns annotation, then syncing that list with PowerDNS.

Almost done.

Now we need to configure your primary DNS to send any request for your new subdomain off to PowerDNS for resolution. If you read this far and don’t have an Ubiquiti router…. I’m sorry. If you have an Ubiquiti EdgeRouter you’re in luck because you only need to do one more setting 🙂

Inside the EdgeMax UI, go to the Config Tree tab, then expand service -> dns -> forwarding. From here you will click the Add button for options and set the new option to the following. Note the IP address here should match what you setup when you configured PowerDNS.

server=/k8s.mydomain.house/192.168.1.200

Click Preview on the bottom of the screen then Apply for the changes to take effect.

If you followed my instructions on how to properly setup home DNS, then all devices should get DNS resolution configured via DHCP, meaning your router is the only DNS server your home devices look for. With that being the case, when you try to DNS resolve anything under the k8s.mydomain.house domain (or whatever you configured) it will be sent to PowerDNS for resolution.

With all this in place you should now be able to network reach any Kubernetes ingress or service configured with the external-dns annotation, from anywhere in your network.

To test this out you can set the hostname property on an Ingress rule, or add this annotation to any load balanced service: external-dns.alpha.kubernetes.io/hostname

The hostname specified for the Ingress rule or the Service annotation should fall within your configured sub domain. For example: foo.k8s.mydomain.house

Easy Home DNS

Leave a reply

At home, you add a new system, maybe a Raspberry Pi, or a new VM, or a new home PC. You give it a name. Now you want to network reach that new thing by name. Not IP, by name. So you edit /etc/hosts add an entry and off to the races you go. Oops, wasn’t in sudo mode, let me try that again. There now it works.

But why does this have to be? Why can’t home networking just fucking work with hostnames?

That’s because we are doing it wrong. First you need to get yourself a good router. I have tried a few in the past, failed each time. Then I got an Ubiquiti EdgeRouter 4, and my dreams were answered. You need to get a few settings inside of your router set, then each time a new system comes online and gets an IP from the router, you will be able to hit it by hostname, without needing to configure anything special beyond standard DHCP on each host/VM/thing in your network.

Main Rule: Stop putting 1.1.1.1 or 8.8.8.8 or whatever your favorite public DNS is on every single client, and in every single alternate DNS configuration option in your network. I seen a vSphere VM customization policy wreck havoc on this because it specified 1.1.1.1 as a DNS server and that VM couldn’t ping by name internally. You might think you are helping, but all you are doing is masking something broken upstream. So fucking stop it!

Properly working DNS should delegate and forward your requests upstream where it makes sense. Like if you request something.com and your local router doesn’t have something.com as its domain then it should forward the request upstream. So the only spot you should configure the 1.1.1.1 entry is inside your local DNS server… also known as your router.

Finally your router will need to know which domain, so it can find entries. That domain should also match the default search for DHCP. Once all that is setup, DNS will magically work for you both internal, and external.

So back to making this work on an Ubiquiti EdgeRouter. Do these steps (and nothing more) and things will work. Doing more may not break things now, but in the future you may want to do some more DNS magic and get tripped up.

Set your system domain

In the EdgeOS UI, click on the System tab at the bottom of the screen, then set your System domain-name. This can be anything you really want it to be, though I recommend you spend the $15 and actually buy the name too. I set mine to a funky .house top level domain. Then I went out and bought it.

Setup DHCP domain name

To make sure everything lines up, and your router will actually service your requests when you try to ping by simple hostname we need to have DHCP communicate the domain name properly. To do this, go to the top level Services tab, then the DHCP Server sub tab. From here you should see all the DHCP servers you have configured (1 per interface). On the right side of the screen click Actions, then View Details. Set the domain name here to the same you specified above. Repeat this for each DHCP server.

Set your DNS forwarding servers

Now you likely configured these settings already when you setup your router initially, so we are just going to confirm a few things this time around. On the bottom of the screen, click the System tab. In this screen on the right side you will see a Name Server configuration option. This should only have a single entry, which is your router’s IP. That’s it. Nothing fucking else!

Next we are going to expose one of the small issues with EdgeOS. The fact that they don’t have a graphical way to give you DNS forwarding outside of the tree editor. However before we get to the tree editor, we need to make sure all our interfaces will have forwarding enabled. So go to the top level Services Tab then select the DNS sub tab. From here you should see all your connected interfaces listed. If they are not, add them now. I have 3 interfaces hooked up in my world so it looks like this for me.

Now that we have the interfaces setup for DNS forwarding, we need to tell EdgeOS where to forward the requests. Like I mentioned earlier, you may have already done this when you setup your router, but let’s double check. You need to go to the Config Tree tab, then expand service -> dns -> forwarding. Here you will see the public DNS servers configured. If not, add them as name-servers. You can add more than 1 name-server. This right here, is the only place you configure the public DNS. Don’t do it in your VMs, don’t do it in your vSphere networking policies, don’t do it on your local systems, don’t do it on any other fucking device you have that connects to the internet via this router. Nowhere else!

Now you are setup. You will be able to ping everything by simple hostname, or with the domain name suffix you specified. Any new system or VM that comes online using DHCP to connect will get proper DNS rules and just work. If you need to configure something with a static IP in the client (should not do this), then make sure the only DNS entry is for the router/gateway itself.

Happy home networking by name.