Tag Archives: ubiquiti

Conquering DNS in a Kubernetes homelab

When it doesn’t work we blame DNS. DNS is hard. Add Kubernetes to the mix and your eyes get watery, heart rate increases, palms get sweaty… and you just walk away before you risk losing anymore time in your life.

Then you came across this blog post, and had some hope. Hoping that someone, somewhere solved this problem.

Well I hope this is that blog post. The problem I solved is having DNS entries for load balanced services and ingress resources automatically be made available to my homelab network, outside of Kubernetes.

I already solved the standard homelab DNS problem. Any VM or new device that connects to my home network and advertises a hostname, will be immediately discoverable by name. Now I wanted to take it a step further. I like Kubernetes, and its easier for me to push new things to my homelab K8s cluster than creating a new VM. So it was natural that I wanted to solve this problem so I can put things in K8s and hit it by name without having to do anything else on my network.

In comes external-dns for Kubernetes… a well known optional component for Kubernetes. Its role is to update DNS records inside an external provider based on available ingress and load balanced service resources. Every minute it will scan your cluster for new or updated resources, check against an internal cache, and update your DNS provider.

Perfect, now I just need a DNS provider supported by external-dns. I decided to use PowerDNS for this. This DNS service uses MySQL for zone setup, has an API, and community supported GUIs for that API. Nothing fancy, and it can all be controlled without too much pain… as a Kubernetes resources. Yes that’s right. I’m going to run my DNS provider for external-dns, inside of K8s itself… because why not?

The first part was to cobble together all the Kubernetes manifests required to install PowerDNS. Helm to the rescue. I found a community created chart. It came close to everything I needed but still missed a few things. So I cloned it and made my own Helm chart. The chart installs PowerDNS with a single pod MariaDB and persistence storage disabled. The idea being if my PowerDNS deployment falls apart, external-dns will re-sync it within 60s of restart, so we don’t need any form of real persistence.

To make this all work, PowerDNS needs to have knowledge of the domain(s) it’s going to manage entries for. This domain should end up being a sub-domain of your main network. For this post, I’m going to assume your main network router is configured for a domain like mydomain.house and all your home devices use your router for DNS resolution. Knowing that we are going to setup PowerDNS to manage a domain called k8s.mydomain.house.

The Helm chart I created takes a list of domains you want to have PowerDNS manage and it will configure them on startup. Since there are some api keys and passwords involved in all this, you also need to set a few more details, and in the end, you have a values file for the Helm chart that looks like this:

powerdns:
  api:
    key: SOMETHING_ANYTHING
  initDomains: 
    - k8s.mydomain.house 

service:
  annotations:
    metallb.universe.tf/allow-shared-ip: powerdns  
  type: LoadBalancer
  ip: 192.168.1.200

mariadb:
  rootUser:
    password: A_PASSWORD

So a couple of things to note in that block, particularly the service section. I have an annotation specified. This is for MetalLB (which I use as a LoadBalancer) to allow the same IP to share multiple service resources on the same port, which is needed for TCP and UDP resolution on DNS.

With that yaml saved in a file you can install it all like this:

helm repo add puckpuck https://puckpuck.github.io/helm-charts
helm install powerdns puckpuck/powerdns --values my-values.yaml

PowerDNS doesn’t have a UI. I have an HTML file that sits on my hard drive, when I open it, I type in the URL for PowerDNS and my API key, and that’s my UI. I got the file from here: https://github.com/james-stevens/powerdns-webui You can find the actual html file in the htdocs folder of that repo. One day I might actual add an NGiNX pod with this file as part of my PowerDNS Helm chart, but alas, here I am writing a blog about it instead.

Now that we have PowerDNS setup, next is for external-dns. Luckily, the fine folks at Bitnami have created such a Helm chart, and it does exactly what we need. Here’s what I used as my values yaml file.

provider: pdns
domainFilters:
  - k8s.mydomain.house
txtOwnerId: k8s

pdns:
  apiUrl: http://powerdns-api
  apiPort: 8081
  apiKey: SOMETHING_ANYTHING

Then to install the chart I ran this

helm repo add bitnami https://charts.bitnami.com/bitnami
helm install external-dns bitnami/external-dns --values external-dns-values.yaml

Now we have external-dns going out on a periodic basis (every minute) finding all your services and ingress resources, checking to see if they have the external-dns annotation, then syncing that list with PowerDNS.

Almost done.

Now we need to configure your primary DNS to send any request for your new subdomain off to PowerDNS for resolution. If you read this far and don’t have an Ubiquiti router…. I’m sorry. If you have an Ubiquiti EdgeRouter you’re in luck because you only need to do one more setting 🙂

Inside the EdgeMax UI, go to the Config Tree tab, then expand service -> dns -> forwarding. From here you will click the Add button for options and set the new option to the following. Note the IP address here should match what you setup when you configured PowerDNS.

server=/k8s.mydomain.house/192.168.1.200

Click Preview on the bottom of the screen then Apply for the changes to take effect.

If you followed my instructions on how to properly setup home DNS, then all devices should get DNS resolution configured via DHCP, meaning your router is the only DNS server your home devices look for. With that being the case, when you try to DNS resolve anything under the k8s.mydomain.house domain (or whatever you configured) it will be sent to PowerDNS for resolution.

With all this in place you should now be able to network reach any Kubernetes ingress or service configured with the external-dns annotation, from anywhere in your network.

To test this out you can set the hostname property on an Ingress rule, or add this annotation to any load balanced service: external-dns.alpha.kubernetes.io/hostname

The hostname specified for the Ingress rule or the Service annotation should fall within your configured sub domain. For example: foo.k8s.mydomain.house

Easy Home DNS

Set your system domain

In the EdgeOS UI, click on the System tab at the bottom of the screen, then set your System domain-name. This can be anything you really want it to be, though I recommend you spend the $15 and actually buy the name too. I set mine to a funky .house top level domain. Then I went out and bought it.

Setup DHCP domain name

To make sure everything lines up, and your router will actually service your requests when you try to ping by simple hostname we need to have DHCP communicate the domain name properly. To do this, go to the top level Services tab, then the DHCP Server sub tab. From here you should see all the DHCP servers you have configured (1 per interface). On the right side of the screen click Actions, then View Details. Set the domain name here to the same you specified above. Repeat this for each DHCP server.

Set your DNS forwarding servers

Now you likely configured these settings already when you setup your router initially, so we are just going to confirm a few things this time around. On the bottom of the screen, click the System tab. In this screen on the right side you will see a Name Server configuration option. This should only have a single entry, which is your router’s IP. That’s it. Nothing fucking else!

Next we are going to expose one of the small issues with EdgeOS. The fact that they don’t have a graphical way to give you DNS forwarding outside of the tree editor. However before we get to the tree editor, we need to make sure all our interfaces will have forwarding enabled. So go to the top level Services Tab then select the DNS sub tab. From here you should see all your connected interfaces listed. If they are not, add them now. I have 3 interfaces hooked up in my world so it looks like this for me.

Now that we have the interfaces setup for DNS forwarding, we need to tell EdgeOS where to forward the requests. Like I mentioned earlier, you may have already done this when you setup your router, but let’s double check. You need to go to the Config Tree tab, then expand service -> dns -> forwarding. Here you will see the public DNS servers configured. If not, add them as name-servers. You can add more than 1 name-server. This right here, is the only place you configure the public DNS. Don’t do it in your VMs, don’t do it in your vSphere networking policies, don’t do it on your local systems, don’t do it on any other fucking device you have that connects to the internet via this router. Nowhere else!

Now you are setup. You will be able to ping everything by simple hostname, or with the domain name suffix you specified. Any new system or VM that comes online using DHCP to connect will get proper DNS rules and just work. If you need to configure something with a static IP in the client (should not do this), then make sure the only DNS entry is for the router/gateway itself.

Happy home networking by name.

Replace Bell HomeHub with Ubiquiti EdgeRouter

4 Replies

I have to say the Bell HomeHub 3000 is a fucking piece of shit, and I pity anyone that is forced to use it. I replaced mine, and it was easier than I thought, and got much better speeds out of my internal network and internet speeds.

Doing it was quite easy, and because, I struggled to find the exact steps to do this online, I decided to write and blog them out here. Warning I only cover Fibe Internet. I know you can also do Fibe TV this way, but I’m not clear on the exact steps to enable Fibe TV as well. If you also have phone, unplug it and optional get a VoIP service.

Physically, to connect the fiber line, I used the SFP module provided by Bell inside the HomeHub 3000. I took this and plugged it into the SFP port on my EdgeRouter. I did not have any compatibility issues with this.

Step 1

You need to find your Bell Internet credentials. This is an account name that starts with a “b” and a password. You may have written them down when you setup you Bell Fibe service. You can also get it by login into your Bell account and looking at your Internet service. Here you will find your account name, and you’ll get an option to change your password.

Note the Account name and password we will need them in a later step.

Step 2

Assuming your router is configured and working on your network at least locally, get into the EdgeRouter’s UI main Dashboard. Here you get a list of all your ethernet devices and how they are connected. On an EdgeRouter 4 (my model) the eth3 port is the SFP port, your model may be slightly different here, but the steps should all be the same.

I already configured mine with names, and you can see I also use my other ports for reasons.

Step 3

To make internet work with Bell we need to connect on vlan 35 using PPPoE. So first we add a vlan interface. Click on the Add Interface button, then Add VLAN. From here setup your vlan with an ID of 35 on your SFP port (eth3). I gave mine a name of Internet (PPPoE) because that’s what this is going to be used for. Leave MTU to 1500 and No Address.

Step 4

Next we need to add the PPPoE interface. Click on the Add Interface button, then Add PPPoE. Set PPPoE ID to 0. The Interface is the vlan interface we created in Step 3 (eth3.35). Fill in your Bell Internet credentials (starts with b) that you found in Step 1. Finally set MTU to 1492. This is required for large file transfers with Bell.

Step 5

Verify it all works. Your setup should look something like this and have Connected on the PPPoE interface.

Once this is all setup and done you should have full speed at your router and anything connected to it. Take that HomeHub and make it a paper weight. I recommend against throwing it out. I’m sure Bell would want it back.

I have been running this setup for several months now without any issues. I have gone through multiple power outages and device restarts, as well as prolonged periods without any power interruption. We never had issues with internet connectivity.

Ubiquiti UniFi controller across subnets

Homelab network with Ubiquiti

Leave a reply

A couple months ago when the need came to fix some home networking issues, I turned to a company that I often heard a co-worker talk abut: Ubiquiti. I started humble with just an EdgeRouter 4 as the entry point and main routing for my home. Then when the urge to build out a full homelab + network setup came, I turned to Ubiquiti again.

I’m doing 2 things. 1) Setting up a more reliable and wider range wifi network for my home. 2) Setting up a more robust network for homelab connectivity.

Ubiquiti offers the UniFi line that can do all of this. The UI for the UniFi controller is very good. However I also found the UniFi line to be more expensive / performance than the EdgeMax for anything you put in a rack (switch, router, etc). For WiFi mesh, UniFi does this very well, and offers multiple different types of access points based on what you need it for. The user interface is also much better for UniFi, using a single controller for all your devices. You can kinda do something similar with UNMS (EdgeMax in the cloud), but it’s not the same as you get with UniFi (which also has cloud management).

For price reasons, and because network should not be something you need to configure much after initial setup, I decided to use the EdgeMax line for anything in my rack, and the UniFi line for WiFi. This is probably the only issue with my setup. Half of my network is EdgeMax managed and the other half is UniFi.

My house is already well connected, and I have cat5 running to quite a few rooms already, so going with powered over ethernet (PoE) access points made this easier. I try to hide the access points but they still aren’t awful to look at when mounted to the wall or ceiling. We need 3 access points to cover the house and backyard properly. So noted I need a switch with at least 3 PoE (maybe a 4th).

I also have a few other “devices” if you will that need to connect into the home network. An NVR for security cameras, and a couple of audio sinks that we can use as input sources for the whole home audio system. So that’s an EdgeRouter 10XP that we will use for all devices.

I have a 24-port Gigabit switch which has been my tried and true for several years now. That will remain to connect most of all the things that in the house. TVs, set top boxes, my office, things like that.

Finally the future home lab, and anything we call part of it (ie: storage). For this I will have 1 server with dual ethernet ports. In the future I may add a second server. Servers also have management ports, for Dell this is iDRAC. So that’s 3 ports per compute server. Looking at storage I have just a single ethernet port NAS today, but potential upgrade to one with dual. With all of this I also need a switch that can do link aggregation to take advantage of dual networks. I can solve this with an EdgeSwitch 10X.

Note: I can solve both devices and compute with a single 16-port switch, but the cost was more than $100 difference, and I have plenty of rack room.

Each switch gets it’s own port from my EdgeRouter 4, each on their own subnet. This means I get 750+ addresses in my house divided into 3 groups. All of this handled without having to do anything special. It’s all just default plug in everything together and turn it on.

Setting up the UniFi access points to create a WiFi mesh is easy as well. Plug in your devices into a network switch with PoE, then configure it using the UniFi Controller software. The software will discover your endpoints, provision them, and configure to any wifi / network settings you specified in the controller setup wizard. You can tweak your wifi settings anytime, including many different radio optimizations.

Note: If everything is on the same subnet this is a piece of cake, and that’s how I did it on the initial setup. Later in life I moved my controller software to its own VM running on a different subnet. Doing this requires a few extra steps to be completed that I mention here.

PuckPuck

more of the same… but different