[Blog] Upgrading the home network for >1Gbps internet service part 1

 Recently Xfinity upgraded my service to 2100 Mb/s down, 300 Mb/s up. This is likely in no small part because of competition from ISPs such as Sail/Twist and Sonic starting to encroach on their monopoly. 

300Mbit up is a welcome change. Its still not the promise land of symmetric up/down of old Google Fiber, but its a step in the right direction. 

Moving to >1Gbps internet presents a challenge for wired networks as Gigabit ethernet has been the standard for more than a decade now. I adopted gigabit infrastructure relatively early on, but with a hodge-podge of consumer grade 4/5/8 port unmanaged gigabit switches.

Past Upgrades

Last year I finally upgraded to a CRS326-24G+2S-RM managed switch as my backbone, with a Dell Connect 2816 managed switch in my upstairs office. This allowed me better control of the network ( and to get rid of a few of my tiny switches, though not all). 

With this recent (admittedly low-cost) investment, I'm hesitant to whole-shot upgrade everything. But upgrades will be necessary as even with the new internet speed, none of my computers can directly utilize the new speed. 

Necessary Upgrade Goals

I had to identify the critical parts that MUST be upgraded to best utilize this new speed. This came down to:

1. Modem
• Potential speed increase is worthless if the modem can't run fast enough
• This was taken care of my Xfinity upgrading me to the XB8 Router+Modem. It can utilitze the speed and contains 3 Gigabit ports, 1 2.5 Gbit port(key), and Wifi 7 support
2. Router:
• I run my own router, sicne I dont like the idea of Xfinity getting access to my internal network.
• Plus the XB8 is good for simple configs, but once you step away from that, it gets cumbersome very quickly
3. Backbone network (CRS) 
• Though this is primarily a gigabit switch, it DOES contain 2 10Gb/s SFP+ ports
4. NAS:
• One of the perks of using a very overbuilt motherboard for my NAS build is that it contiains 2x10GBase-T (RJ-45) ports
5. ESXi server
• My Dell PowerEdge T110 II based esxi server is getting long in the tooth, but still does everything I've asked of it. Unfortunately only has gigabit ports.
• BUT, it does have PCI-e slots that are open and can accept additional NICs, whether it be 2.5Gb or 10Gb SFP+ based.

Not upgrading ( for now )

• Unifi U6 LR
• Unifi AP AC Lite
• Both of these APs are still working well, and I'm hesitant to upgrade them. But they are both limited to a 1Gb WAN port
• Misc laptops/desktops
• Not every machine really NEEDs 2.5 or 10Gb speed. I'd love to have all 10Gb network, but money is a thing.
• IOT devices
• None of these are even saturating 1Gbs, and they're wireless. Upgrading these would be a waste. 

2.5Gbe vs 10Gbe?

That brings up another question. Should I upgrade to 2.5Gb for now or go all the way to 10Gb? 2.5Gb would allow me to utilize the new speed, its cheaper, lower power and if i'm being honest, probably sufficient for my needs.

10Gb is more future proof. For certain machines/connections, the difference can be felt and utilized. The 2 most obvious machines are the ones highlighted, the NAS and esxi servers. Also, certain work machines do have 10Gbase-T ports that can be utilized. 

I eventually decided on a hybrid network. 10Gbit between the router and the backbone switch, NAS and Esxi machine. 2.5Gb between the modem and router, but with a router that's flexible enough to handle 10Gbit service (if that ever happens). 

Next entry will detail my meandering decision making in choosing my router. 

[howto] Get CEC working with Kodi on a Vizio TV ( P55 )

 Original solution from her: https://forum.kodi.tv/showthread.php?tid=352993

1. SSH into kodi, usually root and libreelec ( should change this asap)
2. Stop Kodi via the GUI or terminal with "systemctl stop kodi". 
3. Edit the CEC Adapter config file in "/storage/.kodi/userdata/peripheral_data/ via the terminal". This configuration file name varies based on the adapter. For example, for RPI, it seems to be "rpi_2708_1001.xml", and for a Pulse Eight CEC-Adapter, it seems to be "usb_2548_1002_CEC_Adapter.xml". 
4. Change line "device_type = 1" to "device_type = 4". This will force the adapter to register as a Playback device rather than a Recording device.
5. Restart Kodi by restarting the machine or running "systemctl start kodi" in the terminal.
6. May need to restart TV or go into CEC settings and Discover 

Edit: The most important part got left out. 

[Recipe] Josh's Sweet Potatoes

Washed, dried, well oiled. Fairly liberal kosher salt, ground cinnamon, ground cardamom. 

Baked at 400F for 80 minutes

[Recipe] Josh's Mac & Cheese

 Mac’n’Cheese

Ingredients:

◦2 pieces sandwich bread

◦3/4 lb corn/rice elbow pasta

◦4 tbsp brown rice flour

◦3/4 whole yellow onion

◦16 oz sharp cheddar cheese

◦4 cups whole milk

◦7 tbsp butter

◦1 egg

◦Salt, black pepper, hot paprika, bay leaves, dry mustard

Execution:

◦Lightly toast 2 pieces of sandwich bread. Deconstruct with food processor. Bake crumbs at 300F for 10 minutes in flat baking sheet. Set aside to cool.

◦Cook 3/4 lb of elbow pasta corn/rice pasta al dente, salted normally while boiling. Wash to cool and prevent further cooking.

◦Dice 3/4 of whole onion.

◦Set oven to 350F.

◦Melt 4 tbsp butter in bottom of large pot, allow bubbles to subside. Add diced onion and 1 tsp salt. Sweat until translucent.

◦Add 4 tbsp brown rice flour to butter/onion mix, stir for 10 minutes on low to medium heat to form a blonde roux. 

◦Add 1.5 tbsp dry mustard, 1.5 bay leaves, 1 tsp hot paprika, heat on low for 1 minute, stirring constantly to combine.

◦Wisk in 4 cups whole milk, simmer for 10 minutes, stirring regularly to prevent caking on bottom of pot.

◦Remove bay leaves.

◦Beat 1 egg separately. Temper with hot milk mixture, then fully combine with milk mixture.

◦Pour 12 oz of sharp cheddar cheese in milk mixture, stirring to combine.

◦Pour pre-cooked pasta noodles in milk mixture, stirring to combine.

◦Season with 2 tsp black pepper

◦Place all in baking dish. Sprinkle remaining 4 oz of sharp cheddar cheese evenly across top.

◦Melt 3 oz butter in pan, allowing bubbles to subside. Add all breadcrumbs and stir to fully coat with butter and remove clumps.

◦Spread breadcrumbs evenly across top of mac’n’cheese.

◦Bake uncovered in 350F oven for 30 minutes.

[howTo] List directories that contain a file, according to a pattern

BASH text manipulation can do it, but this is more elegant if you're already using find in linux...

find (pattern here) -printf '%h\n' | sort | uniq

Example:
find . -type -f -name '*txt' -printf '%h\n' | sort | uniq

[Info] Another 2-drive failure on RAID5

Currently trying to fix another 2-drive failure on my mdadm raid5. The regular solution of --assemble --force is not working. But first some context.

The rig is a 7-drive mdadm RAID5 consisting of mis-matched branded 2TB drives. 5 of those drives are attached to SATA ports on the motherboard, while another 2 are in a 5-disk Rosewill SATA enclosure. This enclosure is attached via a Sil 3132 PCIe eSata card that supports port-multiplication.

03:00.0 RAID bus controller: Silicon Image, Inc. SiI 3132 Serial ATA Raid II Controller (rev 01)

This and the enclosure came as part of Rosewil's RSV-5 system.

I had previous been usin gthe RSV-4 system, which worked fairly well. That system eventually gave me errors withing dmesg, that I originally attributed to a bad eSATA cable. Eventually that enclosure died because of an errant power surge.

Replacing it with the RSV-5 yielded hte same dmesg errors. No amount of replacement would alleviate the errors. Eventually I figured out, that the errors when away after I upgraded from Ubuntu 14.04 to 16.04. It has performed well since.

Recently I ran into similar ( but not exact) errors as before.

I have now tried my usual trick ( --assemble --force, and the drives in the correct order). But now the error comes back as:

sudo mdadm --verbose --assemble --force /dev/md127 /dev/sdf1 /dev/sdc1 /dev/sdb1 /dev/sdd1 /dev/sdg1 /dev/sde1 /dev/sdh1
mdadm: looking for devices for /dev/md127
mdadm: /dev/sdf1 is identified as a member of /dev/md127, slot 0.
mdadm: /dev/sdc1 is identified as a member of /dev/md127, slot 1.
mdadm: /dev/sdb1 is identified as a member of /dev/md127, slot 2.
mdadm: /dev/sdd1 is identified as a member of /dev/md127, slot 3.
mdadm: /dev/sdg1 is identified as a member of /dev/md127, slot 4.
mdadm: /dev/sde1 is identified as a member of /dev/md127, slot 5.
mdadm: /dev/sdh1 is identified as a member of /dev/md127, slot 6.
mdadm: added /dev/sdc1 to /dev/md127 as 1
mdadm: added /dev/sdb1 to /dev/md127 as 2
mdadm: added /dev/sdd1 to /dev/md127 as 3
mdadm: added /dev/sdg1 to /dev/md127 as 4 (possibly out of date)
mdadm: added /dev/sde1 to /dev/md127 as 5
mdadm: added /dev/sdh1 to /dev/md127 as 6 (possibly out of date)
mdadm: added /dev/sdf1 to /dev/md127 as 0
mdadm: /dev/md127 assembled from 5 drives - not enough to start the array.

I thought it was because the Event counter was too far off:

         Events : 120796
         Events : 120796
         Events : 120796
         Events : 120796
         Events : 120796
         Events : 120788
         Events : 120788

For posterity here is a good example of a drive:

/dev/sdb1:
          Magic : a92b4efc
        Version : 0.90.00
           UUID : f21f5306:c8a07e60:fad3a920:52a40d5b
  Creation Time : Tue Dec 21 20:21:48 2010
     Raid Level : raid5
  Used Dev Size : 1953511936 (1863.01 GiB 2000.40 GB)
     Array Size : 11721071616 (11178.09 GiB 12002.38 GB)
   Raid Devices : 7
  Total Devices : 5
Preferred Minor : 127
    Update Time : Thu Aug  9 23:39:20 2018
          State : clean
 Active Devices : 5
Working Devices : 5
 Failed Devices : 2
  Spare Devices : 0
       Checksum : ce567320 - correct
         Events : 120796
         Layout : left-symmetric
     Chunk Size : 64K
      Number   Major   Minor   RaidDevice State
this     2       8       17        2      active sync   /dev/sdb1
   0     0       8       81        0      active sync   /dev/sdf1
   1     1       8       33        1      active sync   /dev/sdc1
   2     2       8       17        2      active sync   /dev/sdb1
   3     3       8       49        3      active sync   /dev/sdd1
   4     4       0        0        4      faulty removed
   5     5       8       65        5      active sync   /dev/sde1
   6     6       0        0        6      faulty removed

A drive that was kicked:

/dev/sdh1:
          Magic : a92b4efc
        Version : 0.90.00
           UUID : f21f5306:c8a07e60:fad3a920:52a40d5b
  Creation Time : Tue Dec 21 20:21:48 2010
     Raid Level : raid5
  Used Dev Size : 1953511936 (1863.01 GiB 2000.40 GB)
     Array Size : 11721071616 (11178.09 GiB 12002.38 GB)
   Raid Devices : 7
  Total Devices : 7
Preferred Minor : 127
    Update Time : Thu Aug  9 23:31:29 2018
          State : clean
 Active Devices : 7
Working Devices : 7
 Failed Devices : 0
  Spare Devices : 0
       Checksum : ce567281 - correct
         Events : 120788
         Layout : left-symmetric
     Chunk Size : 64K
      Number   Major   Minor   RaidDevice State
this     6       8      113        6      active sync   /dev/sdh1
   0     0       8       81        0      active sync   /dev/sdf1
   1     1       8       33        1      active sync   /dev/sdc1
   2     2       8       17        2      active sync   /dev/sdb1
   3     3       8       49        3      active sync   /dev/sdd1
   4     4       8       97        4      active sync   /dev/sdg1
   5     5       8       65        5      active sync   /dev/sde1
   6     6       8      113        6      active sync   /dev/sdh1

I opened up this thread on ubuntu forums:

https://ubuntuforums.org/showthread.php?t=2399971&p=13796868#post13796868

They suggested I try --run.

This yielded:

sudo mdadm --verbose --assemble --force --run /dev/md127 /dev/sdf1 /dev/sdc1 /dev/sdb1 /dev/sdd1 /dev/sdg1 /dev/sde1 /dev/sdh1
mdadm: looking for devices for /dev/md127
mdadm: /dev/sdf1 is identified as a member of /dev/md127, slot 0.
mdadm: /dev/sdc1 is identified as a member of /dev/md127, slot 1.
mdadm: /dev/sdb1 is identified as a member of /dev/md127, slot 2.
mdadm: /dev/sdd1 is identified as a member of /dev/md127, slot 3.
mdadm: /dev/sdg1 is identified as a member of /dev/md127, slot 4.
mdadm: /dev/sde1 is identified as a member of /dev/md127, slot 5.
mdadm: /dev/sdh1 is identified as a member of /dev/md127, slot 6.
mdadm: added /dev/sdc1 to /dev/md127 as 1
mdadm: added /dev/sdb1 to /dev/md127 as 2
mdadm: added /dev/sdd1 to /dev/md127 as 3
mdadm: added /dev/sdg1 to /dev/md127 as 4 (possibly out of date)
mdadm: added /dev/sde1 to /dev/md127 as 5
mdadm: added /dev/sdh1 to /dev/md127 as 6 (possibly out of date)
mdadm: added /dev/sdf1 to /dev/md127 as 0
mdadm: failed to RUN_ARRAY /dev/md127: Input/output error
mdadm: Not enough devices to start the array.

It was suggested that maybe the Input/Output error was because of one of the drives instead of the md127 not being able to be created.

Googling has not resulted in any conclusive direction.


I did find this possible sokution:

https://ubuntuforums.org/showthread.php?t=2276699&page=2&highlight=mdadm+event+counter

sudo mdadm --stop /dev/md2
sudo mdadm --zero-superblock /dev/sd[abcdhijk]
sudo mdadm --create --assume-clean /dev/md2 /dev/sd[abcdhijk]

Though it is cautioned as the "nuclear option" and thus I'm saiving it for when all other alternatives have been exhausted .

 I will keep this page updated as I try more things.

[howTo] Improve piaware results

I noticed I've got drastically better results with piaware ( Raspberry Pi + Flight Aware ADS-B ) by enabling SSH, and upgrading the underlying raspbian packages.

Enable ssh on raspbian:

• Mount the micro-sd on another computer. 
• Create an empty file called "ssh" 
• Boot the Pi
• sudo apt-get update; sudo apt-get upgrade

Alternatively you can go through flightware and issue an "Upgrade all installed Debian Packages" command. 

My results went from: 18k reports to 50k reports (Fri vs Fri comparison). With no change to antenna placement or hardware. 

Dramatic difference. Pretty obvious when the change was made. 

[howTo] Fix DNS on OpenVPN headless install + (Private Internet Access)

Ran into an issue when getting a headless OpenVPN client working with Private Internet Access. After the connection was up, system could not resolve DNS.

Quick and dirty solution is to add a public nameserver to /etc/resolvconf/resolv.conf.d/head

cat /etc/resolvconf/resolv.conf.d/head

nameserver 8.8.8.8

There's probably a better way to have OpenVPN get a DNS from PIA, but I'll have work that out later.