Windows Server 2022 upgrade notes

Upgraded Windows Server 2019 to Server 2022 today. I RDP’ed in and mounted iso and setup from within windows. It downloads some updates and performs some installation steps within windows. Took 10-20 minutes to get to 100% then you have to click a button for it to restart. At that point the RDP session terminates obviously. Plug in monitor and keyboard. The restart takes quite a while with nothing on the screen. Eventually windows logon screen is showed and you set up the country etc.

Evaluation Version doesn’t support in place upgrades
The option to keep files/settings is greyed out when upgrading from Windows Server 2019 Eval to Windows Server 2022 Eval.
The only option available is to keep nothing.
To do in place upgrades while retaining files/settings, you must have a fully licensed Windows Server.

Set Computer Name and DNS suffix

Set timezone

Set IP address
Install Hyper V Manager
Setup vswitch VLAN ID
Set static IP and default gateway on vswitch nic
Default gateway was prefilled/greyed out but going in a second time shows a blank default gateway.

Allow Core Network Diagonistics in Firewall to enable ping echos
Edit rule to specify which subnets to respond to, by default it’s only local subnet

Enable RDP
Allow Remote Connections to this computer

Install BOINC
Exit BOINC
Copy boinc data folder contents to c:\programdata\boinc
Restart BOINC
Do sychronization
Confirm tasks and host list

Upgrading HDDs/SSDs using CloneZilla

Just upgraded the boot drive on my HTPC from 128GB SSD to 480GB SSD using CloneZilla. Back in the day, my weapon of choice was Norton Ghost but it was discontinued. CloneZilla works even better/faster/easier than Norton Ghost.

I wasn’t able to get the UEFI .zip files to boot, but downloading the .iso file and using Rufus to “burn” the iso to the same USB key I was struggling with, worked like a charm. It’s weird cuz the computer is UEFI and in the end using the iso method it was also using UEFI.

Regardless, the clone took about 10 minutes for about 93GB of data. Restarted the computer and Windows boot right up none the wiser. Only thing I had to do was extend the partition to take advantage of the full drive, else it was still showing a max of 115GB.

Going to do the same on my server now, going from 128GB SSD to 1TB SSD.

Finally Extended My Pool with a 2nd vdev Tonight

I started with 4x 4TB HDDs in a RAIDZ2 about 2 yrs ago on an old PC and I quickly ran out of space a few months down the road. The snapshots of my backups were taking up most of the space.

So I bought another 4x 4TB, a PC IT mode HBA, some 3.5″ adapter brackets, and molex splitter-adapters for SATA power, and finally installed every tonight. Physical install took the longest… having to remove the power connector for the motherboard, all four DIMMs to install the drives in various remaining bays that in retrospect probably weren’t meant for hard drives.

After everything installed… I have a total of 15.17TiB from 8x 4TB drives.

Mixed feelings really. Relieved I’m no longer living at 95%+ full. Disappointed I’m nowhere near 32TB which admittedly is a false expectation due to marketing math 1000 vs 1024 and RAIDz2.

I guess by my math I could have a theoretical max of around 20TiB usable if i were able to run a single RAIDz2 vdev instead of two. Since I have two RAIDz2 vdevs I lose 4 drives to parity.

You’d think there would be a way to grow the existing RAIDz2 vdev but nope. I hope dRAID support comes soon as that seems to do what I think I want.

Makes a lot more sense to have 6 data drives and 2 parity, instead of 4 and 4.

Sorry for the rant/speech. I’m sure ppl who know this already knew this years ago. I didn’t find this out until after I had built my freenas and loaded it with data. And ppl that didn’t know this probably wouldn’t ever care to know.

Ending on a positive note… the drives and available space seemingly came online immediately with no delay whatsoever!

Still running Freenas 11. I think I’ll upgrade after a few days of this being stable.

SSH Keys

Stolen from Digital Ocean:
https://www.digitalocean.com/community/tutorials/how-to-set-up-ssh-keys-on-ubuntu-1804

Create RSA key pair

Copy publickey to server

ssh-copy-id username@remote_host

cat ~/.ssh/id_rsa.pub | ssh username@remote_host “mkdir -p ~/.ssh && touch ~/.ssh/authorized_keys && chmod -R go= ~/.ssh && cat >> ~/.ssh/authorized_keys”

echo public_key_string >> ~/.ssh/authorized_keys

chmod -R go= ~/.ssh

chown -R sammy:sammy ~/.ssh

Make sure the authorized_keys content is on one line

Disable Password Authentication

sudo nano /etc/ssh/sshd_config

PasswordAuthentication no

sudo systemctl restart ssh

VLAN tagging

So in the process of moving my IOT devices to its own VLAN, I ended up moving all of my endpoint devices onto its own VLAN as well, on a separate network than my servers. storage and networking infrastructure.

Originally this started off with just the wifi devices on the new VLAN but after I was done doing that, it just felt like some wired devices had to be moved too, the ones that weren’t a server per se.

You could argue this was a useless endeavour since all the devices have access to the servers and vice versa. It just feels cleaner. There’s probably a better technical reason I can’t think of right now.

So after I setup the devices VLAN and pointed the SSID at it, the wired devices became my next target. There were a couple of wired PCs that was obviously going on the new VLAN, but they were connected to a Fortinet switch in another room instead of the Meraki stack. So I had to log into that switch, create the VLANs/descriptions, and apply the corresponding VLANs to the physical switchports. Those went easy enough surprisingly.

It was when I got to my HTPC that I ran into some learning opportunities. In reality, the HTPC also functioned as a Hyper V server where one of my DNS servers resided. The server had to be on the server VLAN (cuz of its IP address) while the HTPC technically should be on the devices VLAN.

Long story short, there are a couple ways of doing this. In the end I put the switchport in trunk mode, with no native VLAN declared. Then in Hyper V Manager, set the virtual switch to the devices VLAN. Then in the VM settings, set the VLAN ID to the server VLAN.

The other way works as well I think. If you set the native VLAN for the trunk port, then you don’t need to declare a VLAN ID on the virtual switch. And you just set the VLAN ID on the VM.

Isolating IOT devices to its own VLAN

I’ve put this off long enough and finally today I decided to move my IOT devices (speaker, camera, lock) to their own VLAN.

What made me procrastinate wasn’t the creation/moving them to a VLAN, that’s easy enough with my Meraki network stack. All it takes is the creation of a new VLAN, a new SSID and VLAN tag it, and firewall rules to block it from accessing the rest of the network. What made me procrastinate was the thought that quite often the phone needs to be on the same network, can access the IOT device and I wasn’t about to switch my phone to using the IOT SSID everytime I needed to use IOT.

Turns out that’s not entirely true at all… the camera was happy being in its own VLAN with no access to my phone… the app just treated it like a remote device as if I was connecting to it when I wasn’t at home.

The lock was a little more finicky. I recall there was a setting for the Auto Unlock to recognize you were home based on your phone connecting to the home wifi. Turns out it now allows you to pick a different wifi than the SSID the lock is connected to.

Finally the speaker was what made me hesitate the most. And rightly so. As I moved all the IOT devices to their own subnet/VLAN, Spotify wasn’t able to detect/connect to the speaker unless I switched to the IOT SSID, or allowed IOT VLAN access to the rest of my LAN.

First I tried to write firewall rules to allow/deny traffic in the SSID. This had limited success. Then I took a step backwards by trying to write allow/deny rules in the ACL of the switch cuz that’s what the internet said. But this made no sense as I finally came to my senses and wrote the firewall rules in the… firewall.

I had three goals so I wrote three rules:

  1. I wanted every device regardless of their IOT to use my DNS servers (for now anyways, this may change when I add guest network).
  2. I wanted devices outside of the IOT network to be able to ping IOT devices (like Spotify on my phone to detect the speaker).
  3. I wanted IOT devices to not be able to ping anything outside of its VLAN.

I may need to tighten this down a bit more as I add more VLANs but for now this works as I intended. IOT is banished in its own VLAN. I can still access IOT devices. But they can’t initiate any traffic outside of its VLAN.

What happens when Windows Evaluation period runs out?

Windows Server powers itself off.

slmgr /rearm to add another 180 days

I was surprised my VMs and Windows Server wasn’t responding to pings or ssh. Went over to check the HP host and found it was turned off.

When I powered it back on everything was looking fine so I went digging through event viewer. Had to scroll through a bunch of stuff but one obscure “information” entry in Application Logs that said:

The license period for this installation of Windows has expired. The operating system is shutting down.

One hour as well as a half hour before Windows Server shut itself down, there are a couple of information logs that said:

The license period for this installation of Windows has expired. The operating system will shutdown every hour.

It logged an identical entry when I turned the machine back on as well.

Nextcloud server running out of space

The 50GB I provisioned to the Nextcloud HyperV VM is filling up fast. There’s less than 9% left available. I grew this from 10GB to 20GB to 50GB. Today we’re growing it to 60GB only. I could grow it even bigger but I don’t expect it to grow much larger much faster as the bulk of the storage is photos and videos from my mobile devices. Plus coming back to expand the hard drive once a few months is good practice and reminder of how to do this in Linux.

Since this is a Ubuntu server, I followed the instructions I found here: https://vander.host/knowledgebase/operating-systems/how-to-resize-an-ubuntu-18-04-lvm-disk/. It says 18.04 but it works on my 20.04 LTS as well.

Before getting into Ubuntu, I started by expanding the VHDX in Hyper V. Actually before this step, I merged the old checkpoints first. (In retrospect I probably should’ve created a new checkpoint at this stage but I have multiple backups to restore from… so YOLO!)

After this, Ubuntu needed a reboot to recognize the new disk size for some reason. I thought I remember the last time I did this I was able to do everything online without unmounting anything or rebooting anything. Need to look this one up.

Edit 5/21/2021:
To rescan disks:
echo ‘1’ > /sys/class/scsi_disk/0\:0\:0\:0/device/rescan

Once fdisk -l showed the new disk size I ran through the rest of the following commands to resize the partition/physical volume/logical volume/filesystem to recognize the new disk space.

Ubuntu Commands

  • echo ‘1’ > /sys/class/scsi_disk/0\:0\:0\:0/device/rescan
  • fdisk -l (note it’s partition 3 by looking at the current Size)
  • parted
    • resizepart, Fix, 3, 100% (type this instead), quit
  • pvresize /dev/sda3
  • lvextend -l +100%FREE /dev/mapper/ubuntu–vg-ubuntu–lv
  • resize2fs /dev/mapper/ubuntu–vg-ubuntu–lv
  • df -h

By default the system will prompt you for size in GB but you can actually type 100% and 100%FREE to use up all of the space without having to calculate down to the decimals or leaving unused space to specify a GB amount.

So you can’t rename your Veeam server

Just logged in and discovered my Veeam server is not working. Both the Veeam service and the SQL service refuse to start. Looking around the internet it looks like renaming a Veeam/SQL server is not good for your Veeam/SQL installation and I might have done exactly that. Posts date back to 2013.

https://forums.veeam.com/veeam-backup-replication-f2/how-to-rename-veeam-backup-server-host-t7019.html

Since I haven’t actually done anything with Veeam since the original installation, I’m just going to repair install and go from there. So far so good.

FreeNAS limitations

It’s been about a month since I re-setup my FreeNAS box. The first iteration was really just a test deployment since the chassis it was originally installed in, only supported two HDDs.

While this particular chassis of mine supports up to 8 drives, the motherboard only has built in support for 4 and I’ll likely need to find a SAS HBA and perhaps some power connectors to hook up additional drives. A SATA card will do normally for SATA drives but I actually have an enterprise SAS SSD that could be used to help with the performance.

I’ve discovered some limitations with FreeNAS. While it is a good platform for home use, it is making a foray into the enterprise space. However without support for the following features I don’t know how well it will do.

Erasure Coding

Instead of striping the RAID parity across physical volumes, erasure coding divides the PVs into chunks and stripes the parity across chunks. The main benefit to this is RAID rebuild times are cut down significantly, esp with the large capacity SATA drives these days, as the empty spaces are skipped in the RAID rebuild. Without this, the re-silver process for FreeNAS can take hours/days depending on the size of the physical volumes. It is because of this, the internet forums are littered with RAIDZ2 (RAID6) or die comments.

This hasn’t been a concern for major vendors like HPE/Lenovo/Dell for close to, if not over, a decade.

Online RAID expansion & Dynamic RAID Migration

Some small NAS deployments may have started with 2 large capacity drives, setup in a RAID 1 mirror for redundancy. As storage requirements grow, it makes sense to add drives to the array and maybe even change to a RAID 5 or RAID 6 for better efficiency. (You only lose 1 drive for RAID5 for parity, and 2 for RAID6, but RAID1/10 you lose half of the capacity of your drives). Without the ability to dynamically change the RAID type after the pool has been set up, the only way around this is to backup the data, rebuild the pool, and restore the data. That, or migrate/replicate the data to a new pool with the new RAID already setup. While inconvenient, this is not likely to be an everyday challenge as you’re not likely to change RAID types often.

As mentioned above, storage requirements are growing constantly. Without any support to add drives to an existing RAID array, FreeNAS requires you to increase your storage a vdev at a time. Meaning if you had 4x 4TB drives in a RAID6, you’re going to need 4x 4TB in another RAID 6, and put them into a RAID 60 to grow the storage pool. You’re not going to be able to expand it into a 5x or 6x or even 8x 4TB RAID 6.

Both of these are features I’ve seen in the enterprise space literally 2 decades ago.

Workaround?

The only real workaround is to plan ahead so you don’t get yourself into a situation where you need to grow your pool or change the RAID type. Some people recommend throwing as many drives as you can into a RAIDZ2 and pray you won’t run out. Others suggest building 4x drive vdevs as a time and adding them to the pool in a stripe and hope you don’t have two drive failures in one vdev.

The good news is that FreeNAS is actually based on ZFS and I’ve seen documentation pointing towards support for dRAID but reviews have been spotty? Still need to do some more research and reading into this area.

Have to admit though lack of these two features are enough to make me keep looking for another NAS OS/FS.