[Blog] New Server Build (Daydreaming)

Current Setup:

Dell PowerEdge T110 II (Hiigara)

• Xeon E3-1230v2
• 32GB of DDR3
• 750GB SATA SSD, 1TB HDD
• 10Gbit SFP+
• ESXi 7.0

Why am I upgrading?

TLDR; Wife is tired of having another mini tower in the living room. Wants it in the server rack. Instead of jerry-rigging a mount, better to upgrade to a 2U/3U/4U form factor. 

I currently have this server running in the living room. This is a limitation of the configuration of the house and where the internet comes in. My wife, understanding as she is, has finally had enough of the pile of computers in the living room, and has decreed that this server MUST go into the server rack to help declutter. 

Having done a ton of research, they don't make an easy to use rack-kit for this model of Dell. I could rig something up using a shelf but that seemed annoying and not that cheap ( $80-100). I could also transplant the components into a 3/4U case, but that would cost about as much, and be no gurantee that the motherboard and PSU would fit. 

My esxi server has been running great for the past 10 years. Performance-wise, I haven't been disappointed in it. It certainly hasn't "felt" like I needed an upgrade. I'm guessing this was helped by the fact that every other non-work computer I have access to are slower than this computer, and I haven't got a new computer since this one. 

That all changed when I got into my head to build a gaming computer. It's relatively modest with a circa 2019 Ryzen processor, and 2nd hand graphics card ( featured in another blog post). Because of the research I did for that machine, it showed me just how old and slow this server is compared to modern systems. 

With all that, if I'm putting money into this server, maybe its more worth it to add more budget, and purposely create an esxi machine for the rack. 

Needs and Wants

• Relatively modern performance. Let's say at-or-better than a mid-teir Ryzen processor.
• 6C/12T minimum, 12C/24T preferably. Likely Single Processor, but Dual Processor would be slick
• 64+GB of RAM, maybe up to 128GB
• 3U preferably, 4U if absolutely needed, 2U if possible
• "budget" not looking to break the bank, aiming for $600 range, but could go higher if I find I need to
• transcoding support - Either via Intel Quicksync in the processor, or via cheap-ish GP

[Blog] Upgrading the home network for >1Gbps internet service part 2

 TLDR; I chose the UCG Fiber https://store.ui.com/us/en/category/cloud-gateways-compact/collections/cloud-gateway-fiber/products/ucg-fiber It has the right combination of 2.5GBase-T, 10GBase-T and SFP+ ports that can support 10Gbit Fiber, with a price that's hard to beat. 

Early on in the process it became pretty clear that using the provided XB8 modem/router from Xfinity was not going to cut it. If used as intended, it was serviceable, but once I started adding simple things like port forwarding, or trying to deactivate the Wifi, it basically gave up. This put me on course to get a new, 2+GBit capable router. 

Requirements

1. Capable of routing 2.5+GBit/s. Preferably even higher as I dont want to have to upgrade again if the internet gets faster
2. A few 2.5Gbit ports, this goes hand-in-hand with being able to route 2.5Base-T speeds
3. A few SFP+ ports, for 10Gbit connectivity to the CR326-24G
4. Nice to have: 10GBase-T port, since the NAS has multiple 10GBase-T ports already ( save on a SFP+ NIC)
5. Cheap-ish

Candidates

• DIY Router VM inside my ESXI server
• DIY Router using a mini-PC from China
• Turn-key solution such as Ubiquity or Miktotik

DIY Router VM

• Pros
  
• Cheapest by far. Would simply need to get a 2.5GBit NIC
• Potentially very flexible with powerful PFSense or similar software
• Cons
• Point of failure, ESXI server goes down, no internet
• ESXi server will become critical to basic network operation
• Quite a bit of configuration and time to maintain settings
• Still need a 2.5GBit switch that has SFP+ to connect to rest of network

DIY Router via MiniPC

• Pros
• Isolate network
• Cons
• Can get expensive. A miniPC that has 10GBit, 2.5GBit and SFP+ can get pricey, close to $300
• Support is nonexistant
• Same amount of configuration and time requirements as VM Solution
• Still need a 2.5GBit switch that has SFP+ to connect to rest of network

Turnkey Solution

• Pros:
• Simplest solution
• Ubiquity would integrate well with existing APs
• Cons:
• Can be expensive
• Potentially limited functionality based on who created the software

In the end the Ubiquity Cloud Gateway Fiber fit the best into my needs AND was cheaper than a DIY solution. Especially considering that it had enough ports to act as the router AND switch. With 4 2.5GBit ports, it has enough ports for some "medium speed" devices. The 10GBase-T port could plug directly into the NAS. Finally, the 2x SFP+ 10GBit ports is enough to connect the ESXi server and backbone switch to the router at full 10GBit. All for the low low price of $274. I couldn't beat that when trying to roll-my-own. 

[Blog] Upgrading the home network for >1Gbps internet service part 1

 Recently Xfinity upgraded my service to 2100 Mb/s down, 300 Mb/s up. This is likely in no small part because of competition from ISPs such as Sail/Twist and Sonic starting to encroach on their monopoly. 

300Mbit up is a welcome change. Its still not the promise land of symmetric up/down of old Google Fiber, but its a step in the right direction. 

Moving to >1Gbps internet presents a challenge for wired networks as Gigabit ethernet has been the standard for more than a decade now. I adopted gigabit infrastructure relatively early on, but with a hodge-podge of consumer grade 4/5/8 port unmanaged gigabit switches.

Past Upgrades

Last year I finally upgraded to a CRS326-24G+2S-RM managed switch as my backbone, with a Dell Connect 2816 managed switch in my upstairs office. This allowed me better control of the network ( and to get rid of a few of my tiny switches, though not all). 

With this recent (admittedly low-cost) investment, I'm hesitant to whole-shot upgrade everything. But upgrades will be necessary as even with the new internet speed, none of my computers can directly utilize the new speed. 

Necessary Upgrade Goals

I had to identify the critical parts that MUST be upgraded to best utilize this new speed. This came down to:

1. Modem
• Potential speed increase is worthless if the modem can't run fast enough
• This was taken care of my Xfinity upgrading me to the XB8 Router+Modem. It can utilitze the speed and contains 3 Gigabit ports, 1 2.5 Gbit port(key), and Wifi 7 support
2. Router:
• I run my own router, sicne I dont like the idea of Xfinity getting access to my internal network.
• Plus the XB8 is good for simple configs, but once you step away from that, it gets cumbersome very quickly
3. Backbone network (CRS) 
• Though this is primarily a gigabit switch, it DOES contain 2 10Gb/s SFP+ ports
4. NAS:
• One of the perks of using a very overbuilt motherboard for my NAS build is that it contiains 2x10GBase-T (RJ-45) ports
5. ESXi server
• My Dell PowerEdge T110 II based esxi server is getting long in the tooth, but still does everything I've asked of it. Unfortunately only has gigabit ports.
• BUT, it does have PCI-e slots that are open and can accept additional NICs, whether it be 2.5Gb or 10Gb SFP+ based.

Not upgrading ( for now )

• Unifi U6 LR
• Unifi AP AC Lite
• Both of these APs are still working well, and I'm hesitant to upgrade them. But they are both limited to a 1Gb WAN port
• Misc laptops/desktops
• Not every machine really NEEDs 2.5 or 10Gb speed. I'd love to have all 10Gb network, but money is a thing.
• IOT devices
• None of these are even saturating 1Gbs, and they're wireless. Upgrading these would be a waste. 

2.5Gbe vs 10Gbe?

That brings up another question. Should I upgrade to 2.5Gb for now or go all the way to 10Gb? 2.5Gb would allow me to utilize the new speed, its cheaper, lower power and if i'm being honest, probably sufficient for my needs.

10Gb is more future proof. For certain machines/connections, the difference can be felt and utilized. The 2 most obvious machines are the ones highlighted, the NAS and esxi servers. Also, certain work machines do have 10Gbase-T ports that can be utilized. 

I eventually decided on a hybrid network. 10Gbit between the router and the backbone switch, NAS and Esxi machine. 2.5Gb between the modem and router, but with a router that's flexible enough to handle 10Gbit service (if that ever happens). 

Next entry will detail my meandering decision making in choosing my router. 

[howto] Get CEC working with Kodi on a Vizio TV ( P55 )

 Original solution from her: https://forum.kodi.tv/showthread.php?tid=352993

1. SSH into kodi, usually root and libreelec ( should change this asap)
2. Stop Kodi via the GUI or terminal with "systemctl stop kodi". 
3. Edit the CEC Adapter config file in "/storage/.kodi/userdata/peripheral_data/ via the terminal". This configuration file name varies based on the adapter. For example, for RPI, it seems to be "rpi_2708_1001.xml", and for a Pulse Eight CEC-Adapter, it seems to be "usb_2548_1002_CEC_Adapter.xml". 
4. Change line "device_type = 1" to "device_type = 4". This will force the adapter to register as a Playback device rather than a Recording device.
5. Restart Kodi by restarting the machine or running "systemctl start kodi" in the terminal.
6. May need to restart TV or go into CEC settings and Discover 

Edit: The most important part got left out. 

[Recipe] Josh's Sweet Potatoes

Washed, dried, well oiled. Fairly liberal kosher salt, ground cinnamon, ground cardamom. 

Baked at 400F for 80 minutes

[Recipe] Josh's Mac & Cheese

 Mac’n’Cheese

Ingredients:

◦2 pieces sandwich bread

◦3/4 lb corn/rice elbow pasta

◦4 tbsp brown rice flour

◦3/4 whole yellow onion

◦16 oz sharp cheddar cheese

◦4 cups whole milk

◦7 tbsp butter

◦1 egg

◦Salt, black pepper, hot paprika, bay leaves, dry mustard

Execution:

◦Lightly toast 2 pieces of sandwich bread. Deconstruct with food processor. Bake crumbs at 300F for 10 minutes in flat baking sheet. Set aside to cool.

◦Cook 3/4 lb of elbow pasta corn/rice pasta al dente, salted normally while boiling. Wash to cool and prevent further cooking.

◦Dice 3/4 of whole onion.

◦Set oven to 350F.

◦Melt 4 tbsp butter in bottom of large pot, allow bubbles to subside. Add diced onion and 1 tsp salt. Sweat until translucent.

◦Add 4 tbsp brown rice flour to butter/onion mix, stir for 10 minutes on low to medium heat to form a blonde roux. 

◦Add 1.5 tbsp dry mustard, 1.5 bay leaves, 1 tsp hot paprika, heat on low for 1 minute, stirring constantly to combine.

◦Wisk in 4 cups whole milk, simmer for 10 minutes, stirring regularly to prevent caking on bottom of pot.

◦Remove bay leaves.

◦Beat 1 egg separately. Temper with hot milk mixture, then fully combine with milk mixture.

◦Pour 12 oz of sharp cheddar cheese in milk mixture, stirring to combine.

◦Pour pre-cooked pasta noodles in milk mixture, stirring to combine.

◦Season with 2 tsp black pepper

◦Place all in baking dish. Sprinkle remaining 4 oz of sharp cheddar cheese evenly across top.

◦Melt 3 oz butter in pan, allowing bubbles to subside. Add all breadcrumbs and stir to fully coat with butter and remove clumps.

◦Spread breadcrumbs evenly across top of mac’n’cheese.

◦Bake uncovered in 350F oven for 30 minutes.

[howTo] List directories that contain a file, according to a pattern

BASH text manipulation can do it, but this is more elegant if you're already using find in linux...

find (pattern here) -printf '%h\n' | sort | uniq

Example:
find . -type -f -name '*txt' -printf '%h\n' | sort | uniq

[Info] Another 2-drive failure on RAID5

Currently trying to fix another 2-drive failure on my mdadm raid5. The regular solution of --assemble --force is not working. But first some context.

The rig is a 7-drive mdadm RAID5 consisting of mis-matched branded 2TB drives. 5 of those drives are attached to SATA ports on the motherboard, while another 2 are in a 5-disk Rosewill SATA enclosure. This enclosure is attached via a Sil 3132 PCIe eSata card that supports port-multiplication.

03:00.0 RAID bus controller: Silicon Image, Inc. SiI 3132 Serial ATA Raid II Controller (rev 01)

This and the enclosure came as part of Rosewil's RSV-5 system.

I had previous been usin gthe RSV-4 system, which worked fairly well. That system eventually gave me errors withing dmesg, that I originally attributed to a bad eSATA cable. Eventually that enclosure died because of an errant power surge.

Replacing it with the RSV-5 yielded hte same dmesg errors. No amount of replacement would alleviate the errors. Eventually I figured out, that the errors when away after I upgraded from Ubuntu 14.04 to 16.04. It has performed well since.

Recently I ran into similar ( but not exact) errors as before.

I have now tried my usual trick ( --assemble --force, and the drives in the correct order). But now the error comes back as:

sudo mdadm --verbose --assemble --force /dev/md127 /dev/sdf1 /dev/sdc1 /dev/sdb1 /dev/sdd1 /dev/sdg1 /dev/sde1 /dev/sdh1
mdadm: looking for devices for /dev/md127
mdadm: /dev/sdf1 is identified as a member of /dev/md127, slot 0.
mdadm: /dev/sdc1 is identified as a member of /dev/md127, slot 1.
mdadm: /dev/sdb1 is identified as a member of /dev/md127, slot 2.
mdadm: /dev/sdd1 is identified as a member of /dev/md127, slot 3.
mdadm: /dev/sdg1 is identified as a member of /dev/md127, slot 4.
mdadm: /dev/sde1 is identified as a member of /dev/md127, slot 5.
mdadm: /dev/sdh1 is identified as a member of /dev/md127, slot 6.
mdadm: added /dev/sdc1 to /dev/md127 as 1
mdadm: added /dev/sdb1 to /dev/md127 as 2
mdadm: added /dev/sdd1 to /dev/md127 as 3
mdadm: added /dev/sdg1 to /dev/md127 as 4 (possibly out of date)
mdadm: added /dev/sde1 to /dev/md127 as 5
mdadm: added /dev/sdh1 to /dev/md127 as 6 (possibly out of date)
mdadm: added /dev/sdf1 to /dev/md127 as 0
mdadm: /dev/md127 assembled from 5 drives - not enough to start the array.

I thought it was because the Event counter was too far off:

         Events : 120796
         Events : 120796
         Events : 120796
         Events : 120796
         Events : 120796
         Events : 120788
         Events : 120788

For posterity here is a good example of a drive:

/dev/sdb1:
          Magic : a92b4efc
        Version : 0.90.00
           UUID : f21f5306:c8a07e60:fad3a920:52a40d5b
  Creation Time : Tue Dec 21 20:21:48 2010
     Raid Level : raid5
  Used Dev Size : 1953511936 (1863.01 GiB 2000.40 GB)
     Array Size : 11721071616 (11178.09 GiB 12002.38 GB)
   Raid Devices : 7
  Total Devices : 5
Preferred Minor : 127
    Update Time : Thu Aug  9 23:39:20 2018
          State : clean
 Active Devices : 5
Working Devices : 5
 Failed Devices : 2
  Spare Devices : 0
       Checksum : ce567320 - correct
         Events : 120796
         Layout : left-symmetric
     Chunk Size : 64K
      Number   Major   Minor   RaidDevice State
this     2       8       17        2      active sync   /dev/sdb1
   0     0       8       81        0      active sync   /dev/sdf1
   1     1       8       33        1      active sync   /dev/sdc1
   2     2       8       17        2      active sync   /dev/sdb1
   3     3       8       49        3      active sync   /dev/sdd1
   4     4       0        0        4      faulty removed
   5     5       8       65        5      active sync   /dev/sde1
   6     6       0        0        6      faulty removed

A drive that was kicked:

/dev/sdh1:
          Magic : a92b4efc
        Version : 0.90.00
           UUID : f21f5306:c8a07e60:fad3a920:52a40d5b
  Creation Time : Tue Dec 21 20:21:48 2010
     Raid Level : raid5
  Used Dev Size : 1953511936 (1863.01 GiB 2000.40 GB)
     Array Size : 11721071616 (11178.09 GiB 12002.38 GB)
   Raid Devices : 7
  Total Devices : 7
Preferred Minor : 127
    Update Time : Thu Aug  9 23:31:29 2018
          State : clean
 Active Devices : 7
Working Devices : 7
 Failed Devices : 0
  Spare Devices : 0
       Checksum : ce567281 - correct
         Events : 120788
         Layout : left-symmetric
     Chunk Size : 64K
      Number   Major   Minor   RaidDevice State
this     6       8      113        6      active sync   /dev/sdh1
   0     0       8       81        0      active sync   /dev/sdf1
   1     1       8       33        1      active sync   /dev/sdc1
   2     2       8       17        2      active sync   /dev/sdb1
   3     3       8       49        3      active sync   /dev/sdd1
   4     4       8       97        4      active sync   /dev/sdg1
   5     5       8       65        5      active sync   /dev/sde1
   6     6       8      113        6      active sync   /dev/sdh1

I opened up this thread on ubuntu forums:

https://ubuntuforums.org/showthread.php?t=2399971&p=13796868#post13796868

They suggested I try --run.

This yielded:

sudo mdadm --verbose --assemble --force --run /dev/md127 /dev/sdf1 /dev/sdc1 /dev/sdb1 /dev/sdd1 /dev/sdg1 /dev/sde1 /dev/sdh1
mdadm: looking for devices for /dev/md127
mdadm: /dev/sdf1 is identified as a member of /dev/md127, slot 0.
mdadm: /dev/sdc1 is identified as a member of /dev/md127, slot 1.
mdadm: /dev/sdb1 is identified as a member of /dev/md127, slot 2.
mdadm: /dev/sdd1 is identified as a member of /dev/md127, slot 3.
mdadm: /dev/sdg1 is identified as a member of /dev/md127, slot 4.
mdadm: /dev/sde1 is identified as a member of /dev/md127, slot 5.
mdadm: /dev/sdh1 is identified as a member of /dev/md127, slot 6.
mdadm: added /dev/sdc1 to /dev/md127 as 1
mdadm: added /dev/sdb1 to /dev/md127 as 2
mdadm: added /dev/sdd1 to /dev/md127 as 3
mdadm: added /dev/sdg1 to /dev/md127 as 4 (possibly out of date)
mdadm: added /dev/sde1 to /dev/md127 as 5
mdadm: added /dev/sdh1 to /dev/md127 as 6 (possibly out of date)
mdadm: added /dev/sdf1 to /dev/md127 as 0
mdadm: failed to RUN_ARRAY /dev/md127: Input/output error
mdadm: Not enough devices to start the array.

It was suggested that maybe the Input/Output error was because of one of the drives instead of the md127 not being able to be created.

Googling has not resulted in any conclusive direction.


I did find this possible sokution:

https://ubuntuforums.org/showthread.php?t=2276699&page=2&highlight=mdadm+event+counter

sudo mdadm --stop /dev/md2
sudo mdadm --zero-superblock /dev/sd[abcdhijk]
sudo mdadm --create --assume-clean /dev/md2 /dev/sd[abcdhijk]

Though it is cautioned as the "nuclear option" and thus I'm saiving it for when all other alternatives have been exhausted .

 I will keep this page updated as I try more things.