Author Archives: puckpuck

Making backups less boring

Backing up data in your home is important. What if your computer crashes, what happens to all those photos on it? Here’s a harder one. What if your home catches fire. What happens to the photos now? Mix in a homelab and this is a big ball of nasty.

If you pay for home fire insurance, you should also pay for a backup solution. If you have a homelab, you need a storage and backup solution cause well, shit will hit the fan with a homelab. Backups don’t need to be boring or annoying.

Doing a home based backup makes a lot of sense. You get the speed and reliability with frequent updates at a low operational cost. Doing a cloud based backup makes a lot of sense in addition to this so you can have the security if things really go bad. Doing both is the best of all worlds.

For home based I run a Synology setup. For the features and functions you get with Synology’s Disk Station Manager UI and plugins, nothing else beats the platform.

I have 2 volumes setup. One large volume formatted using btrfs file format. This format gives lots of great reliability and backup features including snapshots of files. You can also create shared folders in btrfs with size quotas, which MacOS will property detect with Time Machine. So we have multiple systems keep their Time Machine backups on the same volume.

The other and smaller volume is formatted using ext4. This has slightly better write throughput, which I use for persistent storage in my homelab. On this volume I have multiple Kubernetes persistent volumes using iSCSI, and a mounted shared to my vSphere homelab using NFS.

All systems in the home backup regularly to the NAS. As mentioned earlier, when you use Synology btrfs you can create file shares with size quotas. Meaning we can use a single volume, with multiple shares each for a Time Machine. Setup the shares on the “smallish” side at first, and expand as we need independently. Before this would require a dedicated volume, which does not offer the flexibility to grow/shrink on demand.

My homelab setup also gets backed up nightly to the main volume using a plugin for Synology called Active Backup. This is the only not so great moment in my world with Synology. To do this, I had to create a connection to my server (easy), but then I had to create a separate backup task for each VM. Ideally this would just backup all on the server, and/or query my server for available VMs and prompt me for which ones to backup in a quick operation. Active Backup runs nightly at 2am on my NAS.

So all your data is on your NAS. You have a backup of it, so if anything crashes you can recover, and do so quickly. That’s important. We are all humans and the majority of backup requests is not because something failed, but because you as a human fucked up. That’s fine, the backed up copy is right there. And if you are using advanced backup features (ie: Time Machine) you can even go back to a specific point in time.

Then a freak accident happened, and the whole fucking thing burned to the ground.

This is why we do cloud based backups as well. The entire contents of your local NAS should get backed up offsite on a regular (daily) basis. Ideally and because it exists, the offsite backup is the cloud. Now I’m not talking about the consumer cloud backup products. You got this far in a blog, you don’t need to understand that. I mean using an enterprise class cloud based backup setup like Google Cloud Storage or Amazon Web Services S3. Going this route will save you plenty of money and might even be easier to setup.

I’m going to make it easy for you: use Google Cloud Storage. I used to use AWS S3 glacier. It worked. My cost was about $12 / month. But if I need a file… fuck it and wait. Want to clean up something cause you know homelabs and humans and shit. Yeah right. Cleanup is running some script to get your “archive list”, then another arcane script to delete each entry in the list… one at a time. Fuck that noise. Use Google Cloud Storage and be happy.

Today my bill is about $6 / month for GCS, and that includes storage and nightly backups from my NAS. What’s great about GCS is you can also move to different storage classes at will. I use coldline storage classes in GCS for all the backups. Because backups aren’t strictly additive (they also delete/update) this had the best cost ratio of storage classes.

Doing a backup to GCS on Synology is done using the Cloud Sync plugin. Nothing fancy here, just setup a task for each shared folder you want to backup. However I will add an additional caveat to this. Because of Time Machine, and it doing backups once per hour. You would be better served to schedule a narrow time slot in which the Cloud Sync plugin will operate. I do a 4am-7am window. Typically the sync is done within 30 minutes, but just in case I moved lots of large files into my NAS, the extra time will be handy.

I also configured Cloud Sync to sync upto 10 files at a time. This helps alot with bandwidth, especially if you have lots of small files. On initial sync I configured 20 files at a time, and it would saturate my entire 500 MiB/s fiber connection, when moving larger files.

All in all, I continue to recommend a Synology platform for NAS. It may cost a little more than others, but the time saved in setting up the NAS, and adding additional features is well worth it, and paid back dividends many times over.

Making home(lab) storage easy

Building a homelab, you need to store data. Own a home with kids, you need to store data.

I have a homelab, so my storage needs are a little bigger, but no matter what, if you need to store data, you need a solution. You have 3 choices. Build your own, buy a QNAP, or buy a Synology.

  1. Build your own is cheap, can be done with FreeNAS, will likely be a pain in the ass in some way. I don’t recommend this for anyone unless you know what the fuck you are doing.
  2. Buy QNAP. This is a “good” solution. It will work. You can do whatever you want to fucking do with it, but you need to click a bunch of buttons, and install some shit, and configure that thing. If your phone is Android, this is the NAS you want.
  3. Buy Synology. This will just work for 95% of the things you need with 2-3 mouse clicks. If your phone is an iPhone, this is the NAS you want.

So knowing the above, I went Synology. Did I pay a Synology Tax? Yes. But time is also money, Synology made my family and homelab needs easy.

Picked a platform, now to figure out the size, and growth. This is where I failed. I bought a DS218+ instead of a DS718. Maybe at first I didn’t understand or realize why, but most certainly I should of plucked down the few extra bucks and got the 718. I say this because I will be upgrading soon.

Silver lining: my homelab aspirations have grown since I bought the NAS, and I would much prefer an RS819 instead… something I wouldn’t have considered 6 months ago. So all is not a wash, and soon I will move up to a rack mount NAS chassis.

So why don’t I care about the upgrade? More of the Synology magic. Some people call this the Synology tax, but it’s a reason why you drop a few extra dollars on your setup. In my case, my setup is done using Synology SHR across the drives. This allows me to add additional capacity without needing to match drive size. I can grow my RAID cluster without having to keep my drives the same, as my budget allows. With this, I can slowly eliminate smaller drives while growing the entire footprint, without having to take a shock to my budget so I can do a new RAID setup.

With Synology, I get a bunch of easy to use out of the box features, but more importantly, I can go from small, to whatever the fuck size I want, without the need to redo my setup. Just keep adding drives, of any size I want, and like the rest of the Synology experience… it will just work.

Easy Home DNS

At home, you add a new system, maybe a Raspberry Pi, or a new VM, or a new home PC. You give it a name. Now you want to network reach that new thing by name. Not IP, by name. So you edit /etc/hosts add an entry and off to the races you go. Oops, wasn’t in sudo mode, let me try that again. There now it works.

But why does this have to be? Why can’t home networking just fucking work with hostnames?

That’s because we are doing it wrong. First you need to get yourself a good router. I have tried a few in the past, failed each time. Then I got an Ubiquiti EdgeRouter 4, and my dreams were answered. You need to get a few settings inside of your router set, then each time a new system comes online and gets an IP from the router, you will be able to hit it by hostname, without needing to configure anything special beyond standard DHCP on each host/VM/thing in your network.

Main Rule: Stop putting 1.1.1.1 or 8.8.8.8 or whatever your favorite public DNS is on every single client, and in every single alternate DNS configuration option in your network. I seen a vSphere VM customization policy wreck havoc on this because it specified 1.1.1.1 as a DNS server and that VM couldn’t ping by name internally. You might think you are helping, but all you are doing is masking something broken upstream. So fucking stop it!

Properly working DNS should delegate and forward your requests upstream where it makes sense. Like if you request something.com and your local router doesn’t have something.com as its domain then it should forward the request upstream. So the only spot you should configure the 1.1.1.1 entry is inside your local DNS server… also known as your router.

Finally your router will need to know which domain, so it can find entries. That domain should also match the default search for DHCP. Once all that is setup, DNS will magically work for you both internal, and external.

So back to making this work on an Ubiquiti EdgeRouter. Do these steps (and nothing more) and things will work. Doing more may not break things now, but in the future you may want to do some more DNS magic and get tripped up.

Set your system domain

In the EdgeOS UI, click on the System tab at the bottom of the screen, then set your System domain-name. This can be anything you really want it to be, though I recommend you spend the $15 and actually buy the name too. I set mine to a funky .house top level domain. Then I went out and bought it.

Setup DHCP domain name

To make sure everything lines up, and your router will actually service your requests when you try to ping by simple hostname we need to have DHCP communicate the domain name properly. To do this, go to the top level Services tab, then the DHCP Server sub tab. From here you should see all the DHCP servers you have configured (1 per interface). On the right side of the screen click Actions, then View Details. Set the domain name here to the same you specified above. Repeat this for each DHCP server.

Set your DNS forwarding servers

Now you likely configured these settings already when you setup your router initially, so we are just going to confirm a few things this time around. On the bottom of the screen, click the System tab. In this screen on the right side you will see a Name Server configuration option. This should only have a single entry, which is your router’s IP. That’s it. Nothing fucking else!

Next we are going to expose one of the small issues with EdgeOS. The fact that they don’t have a graphical way to give you DNS forwarding outside of the tree editor. However before we get to the tree editor, we need to make sure all our interfaces will have forwarding enabled. So go to the top level Services Tab then select the DNS sub tab. From here you should see all your connected interfaces listed. If they are not, add them now. I have 3 interfaces hooked up in my world so it looks like this for me.

Now that we have the interfaces setup for DNS forwarding, we need to tell EdgeOS where to forward the requests. Like I mentioned earlier, you may have already done this when you setup your router, but let’s double check. You need to go to the Config Tree tab, then expand service -> dns -> forwarding. Here you will see the public DNS servers configured. If not, add them as name-servers. You can add more than 1 name-server. This right here, is the only place you configure the public DNS. Don’t do it in your VMs, don’t do it in your vSphere networking policies, don’t do it on your local systems, don’t do it on any other fucking device you have that connects to the internet via this router. Nowhere else!

Now you are setup. You will be able to ping everything by simple hostname, or with the domain name suffix you specified. Any new system or VM that comes online using DHCP to connect will get proper DNS rules and just work. If you need to configure something with a static IP in the client (should not do this), then make sure the only DNS entry is for the router/gateway itself.

Happy home networking by name.

Auto add IDs and links to headings in WordPress

Seems simple, but I just want to add a small little working link icon like this next to each heading in my blog. I thought it was going to be a super easy thing, like install a plugin and be done. I thought wrong.

So this is multiple things in one. I want all headings to automatically get an HTML anchor, and I want each one to get a link icon that only shows up on hover to link to that HTML anchor.

Luckily Jeroen Sormani wrote a blog post that got me going in the right direction. But his blog fell short in a few places. It used older bootstrap icons, which no longer ship with WordPress. It also only worked with headings that you didn’t already specify an id on, existing headings that had an id wouldn’t get the fix. Finally, and this is more just a me thing, the hover showed up to the left of the heading, instead of to the right.

But all these things are easy fixes.

Get Font Awesome

First get yourself the Font Awesome WordPress plugin so you can get the link icon (and a bunch of other handy ones). It’s free, and only adds a few static resources to your pages from the official Font Awesome CDN, so your users likely already have it cached.

Add auto-id function

You will need to add a php function to your site theme to find all headings, and modify their output to include the HTML anchor as well as a clickable link icon. To do this in wp-admin, go to Appearance, then Theme Editor. Next you will select Theme Functions and add this to the php file.

/**
 * Automatically add IDs to headings such as <h2></h2>
 */
function auto_id_headings( $content ) {
    
    $content = preg_replace_callback( '/(\<h[1-6](.*?))\>(.*)(<\/h[1-6]>)/i', function( $matches ) {
        if ( ! stripos( $matches[0], 'id=' ) ) {
            $heading_link = '<a href="#' . sanitize_title( $matches[3] ) . '" class="heading-link"><i class="fas fa-link"></i></a>';
            $matches[0] = $matches[1] . $matches[2] . ' id="' . sanitize_title( $matches[3] ) . '">' . $matches[3] . $heading_link . $matches[4];
        } else {
            $startpos = stripos( $matches[2], 'id="') + 4;
            $endpos = stripos( $matches[2], '"', $startpos);
            $length = $endpos - $startpos;
            $title = substr( $matches[2], $startpos, $length);
            
            $heading_link = '<a href="#' . $title . '" class="heading-link"><i class="fas fa-link"></i></a>';
            $matches[0] = $matches[1] . '>' . $matches[3] . $heading_link . $matches[4];
        }
        
        return $matches[0];
        
    }, $content );
    
    return $content;
    
}
add_filter( 'the_content', 'auto_id_headings' );

The function above adds onto the original function by Jeroen. This one will also add the icon and link to headings which already had an ID as well.

Add CSS Styling

Finally we need to add some CSS styling to all this to get it to work and look proper. You may want to play with the margin settings here to your specific requirements. To add this, inside of wp-admin, go to Themes then Edit CSS. Add the following to your CSS styles.

/* hover links on headings */
h1 a.heading-link,h2 a.heading-link,h3 a.heading-link,h4 a.heading-link,h5 a.heading-link,h6 a.heading-link {
	opacity: 0;
	position: absolute;
	margin-left: 0.25rem;
}

h1:hover a.heading-link,h2:hover a.heading-link,h3:hover a.heading-link,h4:hover a.heading-link,h5:hover a.heading-link,h6:hover a.heading-link {
	opacity: 1
}

That’s it. Now all your existing and new headings will get auto-ids with little linkable icons next to them.

Replace Bell HomeHub with Ubiquiti EdgeRouter

I have to say the Bell HomeHub 3000 is a fucking piece of shit, and I pity anyone that is forced to use it. I replaced mine, and it was easier than I thought, and got much better speeds out of my internal network and internet speeds.

Doing it was quite easy, and because, I struggled to find the exact steps to do this online, I decided to write and blog them out here. Warning I only cover Fibe Internet. I know you can also do Fibe TV this way, but I’m not clear on the exact steps to enable Fibe TV as well. If you also have phone, unplug it and optional get a VoIP service.

Physically, to connect the fiber line, I used the SFP module provided by Bell inside the HomeHub 3000. I took this and plugged it into the SFP port on my EdgeRouter. I did not have any compatibility issues with this.

Step 1

You need to find your Bell Internet credentials. This is an account name that starts with a “b” and a password. You may have written them down when you setup you Bell Fibe service. You can also get it by login into your Bell account and looking at your Internet service. Here you will find your account name, and you’ll get an option to change your password.

Note the Account name and password we will need them in a later step.

Step 2

Assuming your router is configured and working on your network at least locally, get into the EdgeRouter’s UI main Dashboard. Here you get a list of all your ethernet devices and how they are connected. On an EdgeRouter 4 (my model) the eth3 port is the SFP port, your model may be slightly different here, but the steps should all be the same.

I already configured mine with names, and you can see I also use my other ports for reasons.

Step 3

To make internet work with Bell we need to connect on vlan 35 using PPPoE. So first we add a vlan interface. Click on the Add Interface button, then Add VLAN. From here setup your vlan with an ID of 35 on your SFP port (eth3). I gave mine a name of Internet (PPPoE) because that’s what this is going to be used for. Leave MTU to 1500 and No Address.

Step 4

Next we need to add the PPPoE interface. Click on the Add Interface button, then Add PPPoE. Set PPPoE ID to 0. The Interface is the vlan interface we created in Step 3 (eth3.35). Fill in your Bell Internet credentials (starts with b) that you found in Step 1. Finally set MTU to 1492. This is required for large file transfers with Bell.

Step 5

Verify it all works. Your setup should look something like this and have Connected on the PPPoE interface.

Once this is all setup and done you should have full speed at your router and anything connected to it. Take that HomeHub and make it a paper weight. I recommend against throwing it out. I’m sure Bell would want it back.

I have been running this setup for several months now without any issues. I have gone through multiple power outages and device restarts, as well as prolonged periods without any power interruption. We never had issues with internet connectivity.

Ubiquiti UniFi controller across subnets

If you are like me you have you some UniFi devices on a different subnet than your UniFi controller. Trying to provision an access point that doesn’t automatically discover is still possible using the Ubiquiti Discovery Tool but there is another and much easier option.

To make this happen all you need to do is enable L2 network discovery option in your UniFi controller. Log into your UniFi controller and go to the Controller Settings section. Depending on if you are using the classic (old) or new UI this will be in a slightly different spot.

In the classic (old) UI you will find this right in the middle of the screen for Controller Settings Section.

classic (old) UI

In the new UI you will find this in Controller Settings, then go to the Advanced Configuration section.

new UI

Once enabled and saved, you can return to your device discovery section and give it a few minutes for your devices in different subnets to appear. That’s it!

Homelab network with Ubiquiti

A couple months ago when the need came to fix some home networking issues, I turned to a company that I often heard a co-worker talk abut: Ubiquiti. I started humble with just an EdgeRouter 4 as the entry point and main routing for my home. Then when the urge to build out a full homelab + network setup came, I turned to Ubiquiti again.

I’m doing 2 things. 1) Setting up a more reliable and wider range wifi network for my home. 2) Setting up a more robust network for homelab connectivity.

Ubiquiti offers the UniFi line that can do all of this. The UI for the UniFi controller is very good. However I also found the UniFi line to be more expensive / performance than the EdgeMax for anything you put in a rack (switch, router, etc). For WiFi mesh, UniFi does this very well, and offers multiple different types of access points based on what you need it for. The user interface is also much better for UniFi, using a single controller for all your devices. You can kinda do something similar with UNMS (EdgeMax in the cloud), but it’s not the same as you get with UniFi (which also has cloud management).

For price reasons, and because network should not be something you need to configure much after initial setup, I decided to use the EdgeMax line for anything in my rack, and the UniFi line for WiFi. This is probably the only issue with my setup. Half of my network is EdgeMax managed and the other half is UniFi.

My house is already well connected, and I have cat5 running to quite a few rooms already, so going with powered over ethernet (PoE) access points made this easier. I try to hide the access points but they still aren’t awful to look at when mounted to the wall or ceiling. We need 3 access points to cover the house and backyard properly. So noted I need a switch with at least 3 PoE (maybe a 4th).

I also have a few other “devices” if you will that need to connect into the home network. An NVR for security cameras, and a couple of audio sinks that we can use as input sources for the whole home audio system. So that’s an EdgeRouter 10XP that we will use for all devices.

I have a 24-port Gigabit switch which has been my tried and true for several years now. That will remain to connect most of all the things that in the house. TVs, set top boxes, my office, things like that.

Finally the future home lab, and anything we call part of it (ie: storage). For this I will have 1 server with dual ethernet ports. In the future I may add a second server. Servers also have management ports, for Dell this is iDRAC. So that’s 3 ports per compute server. Looking at storage I have just a single ethernet port NAS today, but potential upgrade to one with dual. With all of this I also need a switch that can do link aggregation to take advantage of dual networks. I can solve this with an EdgeSwitch 10X.

Note: I can solve both devices and compute with a single 16-port switch, but the cost was more than $100 difference, and I have plenty of rack room.

Each switch gets it’s own port from my EdgeRouter 4, each on their own subnet. This means I get 750+ addresses in my house divided into 3 groups. All of this handled without having to do anything special. It’s all just default plug in everything together and turn it on.

Setting up the UniFi access points to create a WiFi mesh is easy as well. Plug in your devices into a network switch with PoE, then configure it using the UniFi Controller software. The software will discover your endpoints, provision them, and configure to any wifi / network settings you specified in the controller setup wizard. You can tweak your wifi settings anytime, including many different radio optimizations.

Note: If everything is on the same subnet this is a piece of cake, and that’s how I did it on the initial setup. Later in life I moved my controller software to its own VM running on a different subnet. Doing this requires a few extra steps to be completed that I mention here.

Homelab – start to finish

So I did it. Writing this post is actually me saying it’s finished. That’s my homelab, the one with the end goal to have my puckpuck.com domain pointed to a WordPress site running inside Kubernetes, setup in a vSphere environment, served up from the closet in my home over SSL for the world to consume.

Just 2 months ago I had a simple-ish home network setup. ISP modem/router -> My WiFi router -> Switch for wired devices. Also included was a NAS device plugged into the switch that we used for home backups and photos.

When we had this home built 10 years ago we dreamed of having it well connected. During construction, with some friends, I pulled over a mile of cabling through the walls. Mostly ethernet, but it also included coax, speaker wires, and even a few hdmi runs. Most of the cabling terminates in a single room I call the “LAN closet”, though the rest of the family calls “the brain”.

So with all the cable pulled through we ended up with a fairly extensive whole home audio setup, a bunch of splitters in daisy chain for antenna TV, and a 2 post rack to kinda hold it all together. Everything terminated directly into components, no real organization, but it worked and it was reliable. That stayed in place for almost 10 years with just a few minor swap-in updates done.

it wasn’t pretty but it worked

In its last incarnation before the “redo” you can see we have a Bell branded router for fiber termination in our home. This router was the initial reasoning behind doing this. I can’t say enough bad things about this router. Port forwarding didn’t work, Wifi was awful at near range, devices constantly dropped. I even had Bell replace the router, and was still plagued with the same issues.

So finally out of frustration I set out eliminate the Bell supplied piece-of-shit router and terminate the fiber connection on my own device. This was my fist use of an Ubiquiti device. I got an EdgeRouter to terminate the fiber, and plugged my wifi router into it. Just doing this alone was a significant boost in internet speeds, especially for wired devices. Setting the router up was easy, and though the EdgeMax UI isn’t elegant, I didn’t have an issue finding what I needed.

Then it bit me, the homelab bug. Maybe it’s because I started following /r/homelab, or maybe I just got sick and tired of looking at my tired rack, and really wanted to finish it.

So I set out to plan out what I wanted this homelab of mine to do, and how I was going to achieve it. Took an inventory of what I needed, laid out what the rack would look like from top to bottom, figured out each cable, where it would terminate on a patch panel, and ultimately onto the switch or rack device afterwards. I had multiple spreadsheets and diagrams to help me figure this all out.

Stickers are fun!

Nothing was spared in planning, and I can’t stress how important this is. Equally important is to only create a plan that you can actually commit to. This stuff isn’t cheap. Sure you want to expand to your dream setup, but don’t short yourself today for something you will do in 2+ years. Technology evolves quickly, and 2 years is a long time in this space.

I approached the setup in 3 phases: network, storage, and compute. The phases were repeated for both the home and my Kubernetes setup.

The home network is especially important. Though I want to run all the new techy stuff and configure it ad-nauseam to have fun. Ultimately this needs to just work, especially for my wife and kids. Since wifi was sketchy before, especially in our back yard and deck area, I was able to convince my wife on the notion that I was gonna build a better network. She obliged and I set my plan in action buying all the network gear needed.

My network is all redone now, and except for a 24-port switch I carried over from before, it’s all Ubiquiti now too. That experience with the EdgeRouter convinced me that Ubiquiti is enterprise grade networking that you can use in the home. Perfect for the prosumer.

On the storage front I have my trusted 2-bay Synology NAS that I carried over. Though I have dreams of needing more storage and to dynamic flex that storage, this all costs money, and the existing storage setup I had would be at about 50% capacity (though it would need a re-configure). Later in life, I’ll get a bigger (and rack mountable) Synology NAS.

Finally for home compute I found a Dell PowerEdge R420 on ebay with 600GB of SAS 15k storage. There might be some Dell/VMware bias since I used to work for a VMware, but generally speaking this is a well supported and popular homelab setup for compute. I was getting a recent enough system including dual socket compute without breaking the bank.

After putting it all together I’m very pleased with the physical results of my lan closet.

#labporn

I go over in detail why I did what for each phase (network, storage, compute) for both the home/homelab as well as Kubernetes inside so I could run WordPress and reach my final goal for this setup… well at least my current final goal 😜

Inventory List:

  • Network
    • Ubiquiti EdgeRouter 4 (main router)
    • Ubiquiti EdgeSwitch 10XP (devices switch)
    • Ubiquiti EdgeSwitch 10X (services switch)
    • TrendNet TEG-S240g (home switch)
    • Ubiquiti AP-AC-LR (x3 for Wifi mesh)
  • Storage
    • Synology DS218+
  • Compute
    • Dell PowerEdge R420
  • Security
    • Amcrest NV4108E-HS (NVR for 5 cameras)
  • Power
    • APC Back-UPS 425
    • Rockville 9-port Power Strip
  • Audio
    • HTD MA-1235 (amplifier – main)
    • HTD MC2-86 (controller for main amp)
    • HTD MC-66 (controller for amp in back yard)
    • Denon AVR-X1000 (a/v receiver for tv and speakers in adjacent room)
    • Apple AirPort Express (AirPlay audio sink used as input source)

ROOOAAAADDD TRIP!!!!

I moved this past July.  Earlier in the month, I had the worst plane travel experience ever.  Here we talk about the wild ride I embarked on with a friend to cross the country from the beautiful bay area of California to the super friendly area of Windsor, Ontario, Canada.  Not sure what I did? Stop reading now, go to maps.google.com and figure it out yourself.

The trip started the faithful morning of Saturday July 27th.  Now in natural PuckPuck style, there’s no way we are doing this in a direct straight line route… that would be too easy.  I had to criss-cross the sites, see everything, and make my truck pay dearly for the miles I was about to put it through.  We were going from the San Francisco Bay Area to Windsor.  A quick look of any map will tell you I-80 to the Chicago area, get on I-94 and cross the border in Detroit and boom you are there.  Yeah fucking right!  That way is for pansies.

Joining me for the ride was the delightful Leigh Anne.  Yes she’s a girl, in fact she’s my wife’s best friend, and yes she sat next to me for 5 entire days, even slept in the same room with me.  But alas she wasn’t interested in what I had to offer, so no “whoopie” was performed.

The trip started by waking up with a nasty hang-over, about an hour late.  Good way to start if you ask me.  So next let’s get driving, by going west (see: the wrong way) on I-580, across the bay bridge, through San Francisco, and north over the most pictured bridge in the world, the Golden Gate.  So at this point, I have made negative progress.  My destination was east, and here I am about 60 miles west of my departure.

Straight to Napa we go.  Not because we wanted wine.  I have a friend whom lives there, and agreed to let me park my U-Haul trailer there while I continued on the next leg of my journey.  Huh?!?! leave the trailer behind?!?!

The next leg of the journey was the redwood forest.  I wanted to drive through a tree, and on my last day possible I was going to fucking do it.  Willits, California.  The Gateway to the redwood forest.  The land of rednecks, and cops whom pull me over.  Cue the music getting stopped at a party with a scratch.  Yes I got pulled over just a few hours into our trip.

Back to the trip in a paragraph.  About 6 weeks prior the stickers on my vehicle plates were due, but in the wonderfully expensive state of California, the DMV wouldn’t allow me to renew for just a month.  Since registration fees for our vehicles were about $1000 together I said: “fuck it, I’ll take the risk”.  Fast forward to about 2 weeks before the trip, where I’m in Los Angeles at a company event, losing my mind, money and driver’s license in a bar.

Back to the trip, where this cop was following me with lights blasting for at least 5 minutes, unbeknownst to me.  You see the SUV is full of stuff, so the rear-view mirror is essentially blocked and useless.  If it weren’t for Leigh Anne spilling pop all over her lap, and noticing the cop in the side mirror as she was scrambling from keeping the cool liquid out of her crotch, I wouldn’t of pulled over.

He walks over, and says he’s pulling me over because my front windows are tinted.  Wait a second here.  I lived in California for 2.5 years, never had a problem, and on the day I’m leaving you tell me my car is illegal?  WTF?!?!  Okay, now comes the fun stuff (think 2 paragraphs back).  I hand over my registration, insurance, and an expired California driver’s license.  He leaves returns, makes a few comments about my vehicle, and the amazing odor it was emitting.  Leaves, comes back, reminds me I don’t have a license, my plates are expired, my vehicle window tint is illegal, the odor emitting from my car is also sketchy, then he proceeds to hand me back my stuff, slaps the roof my the SUV and says: “Have a nice day!”

“Have a nice day!”

Umm excuse me while I attempt to un-pucker my ass cheeks.   Did he just really tell me to have a nice day, instead of throwing the fucking book and pedestal at me?  Did that really happen?

An hour later I drove through that tree, and an hour later back through Willits, where my butt cheeks puckered again.  I didn’t think I could hold my breath that long.  It felt like hours.

We get back to Sonoma county, on our way to Napa where I stop for gas.  That day, was some kind of racing event at Infineon raceway.  Lots of cool looking cars at the gas station.  My SUV, has the fuel tank on the right side of the vehicle.  I fill it up, checking out the cars, wishing I could enjoy some of the weed that the driver just finished (or maybe still was) smoking.  Check the oil, go in the gas station for a restroom break, back to the truck, start it up, put it in drive and leave.  About 3 feet later I hear a big crash and bang.  Stop real quick, and look back to a hole in my window, noticing the gas pump was still in the SUV.  YES!! I was that guy. You know the one that they have pictures of at gas stations for driving away with the gas pump still in the vehicle.

I’m not even 12 hours into my trip, and I managed to get pulled over, and destroy the right rear window on my SUV.  Fucking Lovely!  This trip is going to be a blast.  Luckily I didn’t break the actual gas pump.  Unluckily, my window is smashed, and I don’t have an actual house or time to fix it.

MacGyver time. I’m moving, so I have boxes, scissors and packing tape.  I also have a really cool looking brown non-see through window.

Get to Napa, get my trailer, and finally I can start heading east. About 12 hours after it started, I finally started to make positive progress towards on my trip.  Stopped at a Michael’s along the way to buy black cardboard so my non-see through window could at least be the same color as my SUV.  Just beyond Sacramento, still in California, we take a hotel and call it a day.  Drink some Jameson, get a little drunk go to bed, and get ready for another day.

Woo hoo new day, new tank full of gas (this time leaving gas pump at the station), and off we go.  First stop, Lake Tahoe.  Now on this trip, I had a requirement for the passenger.  You have to tea-bag Lake Tahoe.  Only problem is Leigh Anne isn’t exactly equipped for tea-bagging.  So I grabbed a couple of tea bags (the kind you drink with hot water), gave her a belt and some tape, had her strap it around her waist, and get going.  As she walked into the ice cold lake, she proudly dipped her bags, for a fine photo opp.

Climb the mountain go back down the other side, stop midway down because I can smell burning brakes, and we decide being able to stop is important when hauling a trailer.  Get down the Sierras, and welcome to the wasteland, also known as Nevada.  Straight roads, boring sights, red dirt, boring sights, bright sun, boring sights, and like 5 cars on a major highway.  Did I mention this part was boring?

A few hours and firework purchase later we get to Utah. Next stop the Bonneville Speedway at the Salt Flats. All I can say is no picture does this place justice. All you can see is white, then it’s like the earth just falls off and there’s nothing. We get on the main access road, and all I can see is well… nothing but white.  Stop, get golf clubs out.. FORE!!!!  Now in a few years if you hear about a car at the salt flats that rolled and crashed because it hit a golf ball… I didn’t do it. Seriously this place was very very cool, and I would love to return with more time to just drive around and be in a salty place.

Back on I-80 we go, towards the great Salt Lake City.  As we approach the city, we can see the great salt lake out in the distance to our left.  This is where Utah just went from “meh” to “the most eerie place on earth”.  Rising from the lake we can see a cloud.  Nothing too abnormal, this is what happens in the world.  Sun heats water, water evaporates, forms a cloud, and clouds rain water back to earth.  But this cloud was a cloud from the salt lake… the EVIL salt lake.

As we continued to drive on I-80, the cloud started to float over the highway.  And this is when lightning started to flash.  Then a rainbow formed, but this rainbow was in front of lots of lightning, and a cloud getting darker by the second.  As we drove under the rainbow, like it was a gateway, the cloud came down on us.  It was raining from above, left, right, heck even the road was bubbling up water from the cracks.  There was water coming from everywhere.  I can only explain it like this: “I was driving in the fucking salt lake on I-80″.  Sun is shining, lighting glistening, thunder cracking, rainbows glowing, even the fucking birds were flying… in the middle of a storm.  It was like an episode from the twilight zone, and we were the main characters.

We finally break through this storm, and approach the last mountain range before Salt Lake City.  Stop for gas (again doing good for not leaving with the pump in the truck), and go around this last mountain range.  This is where I finally understood what rainbows are for.

We have been lied to.  All this time, rainbows are synonymous with happiness, butterflies, pretty.  Well fuck that shit.  I have pictures to prove, that rainbows are actually the gateway to hell.  There, forming end to end, I seen the biggest rainbow I have ever seen.  And just beyond it, the darkest, scariest, lighting filled cloud I never wanted to see.  The rainbow was the gateway to death and I wasn’t going to be a part of it… so we turned left and drove north on I-15.

Enter Idaho.  About 2 miles in, darkness fell, look up and stars.  Stars everywhere.  Stop get out (next to a corn field of course), and just amaze in a sight I have never seen before.  The night sky is something that you only see in pictures from the best photographers.  It was gorgeous.  The milky way was there in full glory.  Simply amazing.  But this stop has to be short, cause it’s late and we are still an hour (or more) away from Idaho Falls where our hotel was.

As we are driving, yet another storm is brewing.  You hear about blue, red and green lighting.  You think people are hallucinating.  Maybe we were, maybe we shouldn’t of eaten those mushrooms for dinner, but man, there was some pretty cool, all colors in the (death) rainbow, lighting going off.

Get to our hotel, which felt like was in the middle of the ghetto.  Easily the worst hotel I have stayed in (besides that place you could rent by the hour).  The next morning when I awoke, I seen a man wearing his underwear standing on his front porch, with a woman popping a zit off his back.  Yup!  Made me want bacon and eggs.

Back on the road.  Yellowknife park here we come.  Now I planned this road trip.  Somehow I missed the fact that we were going to drive through Montana (even if it was only for like 10 miles).  Anyhow seen a really quaint town that belonged in the movies at the gateway to the park.

Get in the park, drive straight towards old faithful.  We got lucky.  We pulled up, walked over to the geyser, where a crowd was gathered,  Overhead someone saying “its time”, so we quickly found the best view angle, pulled out our cameras, watched the eruption, and 5 minutes after arriving we were on our way out of there.

Yellowstone park is beautiful.  Gorgeous scenery everywhere.  One point we seen this pond with orange water in it (likely some sulfur reaction from all the seismic activity).  Hey lets check it out.  Pull over, get out and got for a trail walk.  Take some pictures, relax for a few minutes, head back and… well… I lost my key to the SUV.  FUCK!!!!

We’re in the middle of nowhere, no cell reception, windows are rolled down, sunroof open, and no fucking keys!!! I keep my key chain to a minimal.  Meaning all that I lost was the actual key and key fob.  Nothing else.  Nothing really shinny that would reflect light, just a black key, and black key fob… lost in the middle of nowhere, in Yellowstone park.  DOUBLE FUCK!!!  After having what I can only assume is 3 consecutive heart attacks, Leigh Anne finds it and we can breath a sigh of relief.

As soon as we left the park, heart still beating a thousand beats a minute, we see a local watering hole.  Time to stop, breath, and do what anyone would do in this situation.  Whiskey on the rocks please!  A little bit of food to wash that down, nerves back to normal, lets continue to drive.

At this point we “thought” we were done mountain driving, and La Bamba (I named my SUV) would be fine.  I also thought my truck was legal in California 2 days prior.  You see the road I planned to drive on, also happened to be the Big Horn Pass.

Big Horn pass, was an awesome super duper narrow, twisted, falling rocks everywhere road.  Eventually we get to 9000 feet, and start the eventual descent back to reality.  Lots of cool wildlife, finally got to see a moose, as well as other creatures that I would rather keep my distance from.

Clear the pass, get on I-90.  One would think I-90 in the middle of summer would be a well travelled, major interstate in the United States.  One would also be wrong.  I have never seen a road darker, and more desolate (remember we were in Idaho the night before).  We could drive for miles over miles without seeing a single car or light.  We pulled over at this “pull off parking” area.  It’s like an off-ramp, short parking area, on-ramp.  No building, just a spot to stop and park, and maybe take a nap.  And this is where we notice the VW micro bus with a peace sign.  The same one we seen the day before just outside the salt flats in Utah on I-80.  Just now we are far more east, and a few hours north.

Get to Cedar Rapids Iowa for the night.  Got to bed (after a few drinks of course), and wake up to go see a bunch of faces carved into the side of a mountain.  On our way there, that VW micro bus was also coming along as well.  Took some pictures, and back to the road we go.

Ok, now we are exhausted, and if you made it this far in reading, you should be as well.  At this point, excitement is a store called Wall Drug, and this weird artsy thing we seen in the middle of a corn field.  I had visions of really bad horror movies.

Did you know that after 180 something you’ll likely just lose count of what you were doing (counting corn fields).  After a long boring corn field filled day of driving, we arrive at Des Moines to spend the night.

Finally the home stretch.  I’ve done most of this part of the drive many times before.  Across the top of the Mississippi, and 4 hours into the day of driving I was back in familiar territory of the Chicago area.  We drive through the windy city, around lake Michigan, and set our sights on Detroit.

Around 9:30pm at night, we arrived in Canuck territory, without even having to declare that trailer full of goods I was hauling (yes Canada customs is super strict).  Maybe the border agent was too busy thinking that her igloo was melting, and how she would care for her pet beaver. or maybe she just looked at me and thought I already had enough excitement for a while.  Either case, she couldn’t be bothered to ask what I was doing, or why I had a dual axle U-haul trailer behind me.Side Note: Unfortunately for me, since I didn’t get pulled into secondary inspection at customs, I never got to properly import my vehicle, and had to deal with incompetent customs officers whom screwed up my paperwork 2X before I could legally import my Canadian vehicle, back into Canada.  When we returned the second time to get the paperwork straightened out, the customs officer insisted she did it right, and even brought her supervisor over to explain it to us.  Supervisor looked at what we were trying to do, looked at the customs officer and said, you need to do it the way they (my wife and I) are describing, not the way you previously did.

We arrived, celebrated that we just did an awesome road trip, and made it to our destination with the truck (mostly) in one piece, and our bodies intact… although I’m pretty confident my mind is still somewhere in Wyoming.

You think your plane ride was rough?

Recently moved from California to Canada.  No easy task.  This one is about the plane ride that will make your worse plane ride feel like a dream cruise.

Early morning of Saturday July 6th.  Wake the kids, feed the cats some drugs to make them sleepy, order a car cause we have too many passengers and luggage to fit in our SUV, and lets go to the San Francisco airport.

We are 6+2 in all.  That’s 3 kids, 3 adults, and 2 cats.  8 checked bags, a stroller, 6 carry-ons, and 2 cat carriers.  That’s alot of luggage when standing in line to check-in.  Delta was awesome and got us a porter to help us out with our Mt. Everest of luggage.

Security time.  I (not sure why) decide that I don’t want to take my big fat 20+ pound cat out of his carrier, and therefore am subject to a private check.  Fine.  Only issue is the cat fur fucks up the explosive testing machine, and they think I’m laced in C4.  10 minutes or so later in a private room, they determine I’m not a terrorist looking to blow up a plane (and my 2 cats) and thus let me go.

Get to the gate in time, my wife goes to the coffee/breakfast kiosk and gets everyone something to munch on… but me of course.  Even though I’m the one that handed her money to buy everyone something.  I’m sure she still loves me… I think.

Board the plane.  Since we have small kids (and cats) we get to board first.  Yippee.  Get settled in, get the kids setup with whatever they need to be entertained, sit down, buckle up and wait.  This is where the fun really starts.

You see the time is about 11:25am, on Saturday July 6th, at San Francisco airport.  We just pushed off the gate, everything seems fine.  My son looking out the window,  amazed by all the planes and equipment says: “Look dad a firetruck!  A green firetruck!”.  Now this doesn’t really faze me.  I travel often.  Airports are my second office, and seeing a firetruck on the tarmac isn’t anything out of the ordinary… or so I thought.  This firetruck was doing something out of the ordinary.  This firetruck was on its way to end of runway 28L.

Here’s a picture I took from my phone of what that firetruck was doing.

Are you fucking kidding me?!?! There’s like 10000 flights a day in and out of the US, there hasn’t been a major airline incident in the US in years, and this happens – on the day I’m moving on what is potentially going to be the hardest flight of my life (and I fly alot – like stupid lots).

Delta Airlines, were good in this case. We sat on the tarmac for maybe 20 minutes, before the pilot pulled us back in to let us off our about to be cancelled flight.  Because I fly so much, I’m also a member of Delta Sky Clubs.  So off to the Sky Club we go to relax, figure out what’s going on and try to keep our tempers down.  Of course the Sky Club has CNN on full blare on every TV.  My son, looks at a TV and says: “That plane is broken!”.  Cute.  Seeing all the kids in there, my wife pulls out my laptop, puts on Cars 2, and the next thing you know, my laptop is being used as a movie theatre screen.  After about 2 hours, I look out on the tarmac, and as you would know it, there sits 2 United Airlines planes. 2 Hours later… still fucking there.  Really?!?!  Seriously United, THIS IS WHY PEOPLE HATE YOUR AIRLINE!  Put the customer first!  You maxed out your 3 hour tarmac period for what exactly?  To piss off the passengers whom were helplessly held, likely against their will, so you can “hope” that the airport will re-open after a fucking plane just crash landed at it?  Seriously, United Airlines get the fuck out… you’re drunk!

After a while I got the awesome Delta agent to rebook us all on a flight leaving Monday morning (getting us all to sit together was a challenge).  So next we are off to get our luggage, call up a car and go back to my empty house.  Small problem.  Since the airport was closed, they weren’t letting anyone in.  So we put the kids and cats in my SUV, packed it with as much luggage as we could, and I drove them home, while my wife took public transit back home with her friend.

So we get back home, only now it’s empty, as in the fridge is empty.  You see for the week prior, we only bought food that would of been consumed by that Saturday.  Well now we return and there is no food, and well it’s not exactly easy to buy food that isn’t called junk for 1.5 more days of living.

Fast forward to Monday the 8th.  We pack our bags, call that car again, rinse and repeat to the airport.  All is well, we get to security.  This time I decide not to get raped in a private room, and decide to take my big fat pata-poof of a cat out of his carrier to go through security.  The way you can silence a noisy security area with a cat that is larger than most dogs is quite entertaining.  We had a collective jaw dropping of everyone in presence.

Get to the gate, and this time I’m buying food.  I, of course make sure everyone gets something to eat and drink.  Board the plane, get seated, and off we go, without any hitch…. yeah right.  This wouldn’t be the fucking plane trip from hell if it was that easy.

We’re about 20 minutes from landing, when the pilot comes on the PA system to let us know there is a ground halt at Detroit Metro airport due to weather, and we will be circling for about 20 minutes.  Sure enough 20 minutes later the pilot gets back on the PA system, and says traffic control still has the ground halt, and it will be another 20 minutes.  Wait another 20 minute, and now the pilot says: “They still haven’t lifted the ground halt, we’re about to run out of fuel, so we’re going to Chicago”.  Wait, What?!?!  FUCK!!!

Ok, so we land at Chicago O’Hare, and taxi to some holding area on the tarmac, so we can get fuel.  We were flying in a very common plane.  A Boeing 767.  You would think the fucking ground crew at one of the world’s busiest airports, would know what kind of fucking fuel a Boeing 767 takes.  You would also be wrong in thinking that.  Eventually they bring the correct fuel truck, fuel us up, and about 1.5 hours later we are off.

Get to Detroit.  Get to our luggage, and get going to our home in Canada.  All that is left is crossing a border…. easy right?

I came prepared.  I read the website, got all my import paperwork in order, and we actually breeze through the import process quite well.  Then my wife comes running in and says: “We need to start the citizenship process for Sawyer” (our youngest son – whom happens to be a yankee).  So we do that, only thing is, the computer program that does this, is running on a toaster oven somewhere in bum fucking no where, powered by a hamster, with the same processing capacity of a withered tulip.  Seriously, if the government wanted to save money, you can pay less overtime in hourly wages to border agents, and take that money saved and buy a real fucking computer to run your immigration software with.

Eventually we arrive, something like 13 hours later.  That glass of Scotch on the rocks at my lightly-famous bar was well deserved.

I got to sleep for 4 hours on a simple air mattress, to wake at 5:30am so I can catch a flight back to California (original schedule had me returning Tuesday morning).

The next time you say you had a bad flight or travel experience just remember.  You weren’t the Tessier family flying with 3 kids and 2 cats, on that weekend in July 2013.

PS: Thank You Leigh Anne!  You’re the best.