GoJimmyPi: 2021

Monday, July 5, 2021

Find gojimmypi at gojimmypi.github.io

I'm currently working on my new blog home at gojimmypi.github.io

After implementing a variety of features such as dark mode, syntax highlighting, infinite scrolling, and more in my fork of the Gridster Jekyll Theme, my new blog is finally live at gojimmypi.github.io

All of my blog posts from here have been migrated to the new site. I have some migration notes.

Development occurs in a subdirectory repo called gridster-jekyll-theme.

Sunday, March 21, 2021

Migration from BlogSpot to GitHub Pages

As convenient as blogger.com marginally is... I've decided enough is enough with the wonky editing, complete lack of integrated source control, and always that little bit of nagging wonder if one day google will just pull the plug and decommission it.

It was several years ago I tried to migrate to my GitHub Pages site at gojimmypi.github.io. Although I consider myself rather persistent, I ended up giving up at the time due to unreconcilable issues with Jekyll. (which admittedly were not supposed to be supported in Windows, and WSL was still somewhat new).

Fast forward a few years and WOW! My Jekyll install experience was nearly trouble free.

So ok, once Jekyll is working, there's still the tiny little issue of templates. I found one I liked from jekyllthemes.io, specifically the gridster-jekyll-theme. Alas nothing is ever as easy as it is supposed to be. Of course, if it is too easy, it is no fun, eh? So the issue with this particular theme, is not only that it is old, but there are a few quirky things that need to be fixed. There are literally hundreds of forks. Which one might be of interest?

How about looking at which fork has the most likes? Unfortunately, there's no such feature on GutHub, but stackoverflow to the rescue once again. There was only one answer to the question "How to find the best changes from downstream Github forks?" and it didn't sound promising with the comment "It's 3 years old, I can't seem to get it working, but someone could fork it" - but find_forks does work!

cd /mnt/c/workspace/
git clone https://github.com/elimohl/find_forks.git                             
cd find_forks
pip3 install tqdm 
pip3 install pygithub
./find_forks.py DigitalMindCH/gridster-jekyll-theme

It should output something list this:

# usage: find_forks.py [-h] [-u USERNAME] [-p [PASSWORD]] [-s SLEEP_INTERVAL] repo_name
0 $  ./find_forks.py DigitalMindCH/gridster-jekyll-theme
 95%|| 197/208 [02:23 00:08,  1.37it/s]Forks   Stars
https://github.com/pcesar65/renameme    1
https://github.com/tomer-ben-david/techblog     1
https://github.com/cristinafsanz/paintings      1
https://github.com/zirkc/appworms       0
https://github.com/bullwinkle-org/gridster-jekyll-theme 0
https://github.com/jfcorugedo/jfcorugedo.github.io      0
https://github.com/tanaphum/gridster-jekyll-theme       0
https://github.com/librarywon/librarywon.github.io      0
...

There's also brute-force method to call the GitHub API:

curl https://api.github.com/repos/DigitalMindCH/gridster-jekyll-theme/forks?sort=watchers

But that's only so useful.

So what about the most changes in a fork? I found forkizard, but nothing happened after the scan. I noticed an open issue and fortunately a PR to fix it! So the one to use is at the trufanov-nok/forkizard fork. I also created my own fork, just in case it ever gets deleted, as this could be really handy again in the future. It works really quite well:

# The source code is in Go. After this operation, 324 MB of additional disk space will be used:
sudo apt-get install golang-go

git clone https://github.com/trufanov-nok/forkizard.git
go build

Run it like this:

0 $  ./forkizard "DigitalMindCH/gridster-jekyll-theme"
2021/03/21 13:14:55 208 forks
 208 / 208 [============================================================] 100.00% 1m26s
done
/AwarenessOverload/AwarenessOverload              +388 -0
/vikasksrivastava/gridster-jekyll-theme           +293 -0
/BrunnaRosa/BrunnaRosa                            +227 -0
/ickc/gridster-jekyll-theme                       +138 -0
/NNUP/nnup.github.io                              +92 -0
/cett/cett.github.io                              +68 -0
/samyak45/Gridster                                +66 -0
/UntrustableRus/gridster-jekyll-theme             +53 -0
/guriosam/ajudamaceio                             +43 -0
/thiagohersan-jekyll/gridster-jekyll-theme-simple +43 -0

I ended up choosing the ickc/gridster-jekyll-theme with a demo at ickc.github.io/gridster-jekyll-theme (my fork is located here) I have it installed to a subdirectory in my GitHub.io page like this:

cd /mnt/c/workspace/gojimmypi.github.io

# the repo is over 200MB in size!
git clone https://github.com/gojimmypi/gridster-jekyll-theme.git --depth 1

cd gridster-jekyll-theme
bundle install
bundle update github-pages
bundle exec jekyll serve

The author was also kind in digging up the documentation from the wayback machine, and included it as part of the theme.

Now to migrate pages...

Saturday, February 27, 2021

Dual WAN OpenVPN with EdgeRouter X or RT-AX86U

This a blog about NordVPN (and networking setup in general) for the Ubiquiti EdgeRouter X. If you are interested in this topic, then you may also be interested in my notes on the Pi-Hole ad blocker.

WARNING: Before configuring your router, be sure to read until the end. TL;DR - config does not survive a reboot. This blog is a work in progress.

I have poor internet bandwidth even after load balancing two different providers. One problem I see is that once a connection is established with basic load balancing the connection is "stuck" on one ISP. So when available bandwidth drops to almost nothing, the same connection continues to be used.

I was hoping to perhaps setup a virtual tunnel using the dual WAN connections to an OpenVPN provider. That way the VPN server would aggregate the two channels and only appear to have a single public-facing IP address. Fortunately I had this idea while NordVPN was having a birthday party sale!

Download Filezilla, the EdgeOS OS User Guide, the latest EdgeRouter X Firmware, and of course putty. If things get dire, the Jounin Tftpd64 server works well. (See EdgeRouter - TFTP Recovery)

It is probably best to start with the most recent firmware image. The version used here is v2.0.9-hotfix.1. Other downloads can be found at the EdgeRouter-X download site.

Note that for a freshly factory-reset EdgeRouter, no DHCP is available so a static IP address is needed; the web GUI can be found at http://192.168.1.1/ on Eth0.

Thanks to Willie Howe's Ubiquiti EdgeRouter X Hidden Console Port video, I was able to easily find and connect a serial port to the internal UART connector of the EdgeRouter-X:

Connect via putty to the serial port. 57600, 8N1:

For a brief time, the UART TTY will allow the bootloader update from something other than flash. This is vastly simpler than the convoluted power / hold-the-reset button method:

Please choose the operation:
   1: Load system code to SDRAM via TFTP.                                    
   2: Load system code then write to Flash via TFTP.
   3: Boot system code via Flash (default).
   4: Entr boot command line interface.
   7: Load Boot Loader code then write to Flash via Serial.
   9: Load Boot Loader code then write to Flash via TFTP.
   r: Start TFTP recovery.
default: 3

This command will "soft" factory reset (actually just configuration reset) the router:

sudo cp /opt/vyatta/etc/config.boot.default /config/config.boot      
reboot

If you find yourself so desperate that a TFTP recovery image restore is needed, here are the settings I used:

(my PC, the client sending the TFTP image file, was manually set to 192.168.1.23)

Once the router has been factory reset, the web GUI can be used for a basic config by visiting your local https://192.168.1.1:

Reminder that if a new network was setup other than the default 192.168.1.0 with the wizard, you'll need to move the network cable from Eth0 to Eth2 and change the fixed IP address setting to DHCP.

The ip route command after a fresh basic config looks like this:

default via 192.168.37.254 dev eth0 proto zebra
192.168.37.0/24 dev eth0 proto kernel scope link src 192.168.37.81
192.168.170.0/24 dev switch0 proto kernel scope link src 192.168.170.1

See the detailed EdgeRouter and Ubiquiti setup with NordVPN.

The nano editor can also be installed:.

configure
set system package repository stretch components 'main contrib non-free'       
set system package repository stretch distribution stretch
set system package repository stretch url http://http.us.debian.org/debian
commit 
save 
exit

# DO NOT RUN UPGRADE, (unless you want to break the EdgeRouter dependencies)
sudo apt-get update

sudo apt-get install nano

#
sudo rm -R /var/lib/apt/lists
sudo rm -R /var/cache/apt/archives

# optional: put back the empty directies as we should have found them
sudo mkdir /var/lib/apt/lists/partial
sudo echo "" | sudo tee /var/cache/apt/archives/lock
sudo mkdir /var/cache/apt/archives/partial

To configure the OpenVPN service, open an ssh connection or use the TTY described above:

sudo -i
cd /config/                                                                
mkdir -p openvpn
chmod 770 openvpn

Copy your nordvpn ovpn (the ca1098 example used here is this file) and nordvpnauth.txt (service username on line 1, password on line 2) files to the /config/openvpn with FileZilla. Note that resetting your password on the ndaccount page will NOT close VPN connections. It should log you out of web sites, but I saw my VPN connection stay open after a password change.

If you are debugging and want to see VPN logs, add the this line to your ovpn file:

  log /config/openvpn/vpn.log

You will now be generating connection logs which can be found in that directory, open them using this command through SSH:

  cat /config/openvpn/vpn.log

(That's one of the many tips I received from NordVPN support; they are awesome)

After copying the two files onto the EdgeRouter, apply proper permissions:

sudo chmod 600  /config/openvpn/vpn.log
sudo chmod 600  /config/openvpn/nordvpnauth.txt

# edit for your specific file name:
sudo chmod 600  /config/openvpn/ca1098.nordvpn.com.udp.ovpn

Note that when doing a "soft" factory reset, step above such as the nano install, vpn directories, and copied files are retained.

On a freshly factory reset EdgeRouter OS, establish connection via SFTP or UART, (be sure to edit the line with ca1098.nordvpn.com.udp.ovpn for your own server for best performance, as well as your local desired network addresses) and configure as shown below:

configure

# Download Server file recommended by NordVPN from:
#  
#     https://nordvpn.com/servers/tools/
#
# Edit the next line and for your file name:
set interfaces openvpn vtun0 config-file /config/openvpn/ca1098.nordvpn.com.udp.ovpn

set interfaces openvpn vtun0 description 'OpenVPN VPN tunnel'
commit

# rule 5000 and 5002 are defaults for WAN1/WAN2
# so we will usee 5100 instead of the NoprnVPN tutorial 5000
set service nat rule 5100 description 'OpenVPN Clients'
set service nat rule 5100 log disable
set service nat rule 5100 outbound-interface vtun0
set service nat rule 5100 source address 192.168.170.0/24
set service nat rule 5100 type masquerade
commit

# we'll route all protocols for all addresses on vtun0
set protocols static table 1 interface-route 0.0.0.0/0 next-hop-interface vtun0

# each network segment (e.g. eth2, eth3, eth4, switch0) will need rules
set firewall modify SOURCE_ROUTE rule 10 description 'traffic from 192.168.170.0/24 to vtun0'
set firewall modify SOURCE_ROUTE rule 10 source address 192.168.170.0/24
set firewall modify SOURCE_ROUTE rule 10 modify table 1

# here we assume Eth0 and Eth1 are WAN ports connected to ISP 
# physical ports connected to ISP are what the tunnel travels on
# the remainder of the ports, eth2, eth3, eth4 and switch zero are all routed to vtun0
set interfaces switch switch0 firewall in modify SOURCE_ROUTE
set interfaces ethernet eth2 firewall in modify SOURCE_ROUTE
set interfaces ethernet eth3 firewall in modify SOURCE_ROUTE
set interfaces ethernet eth4 firewall in modify SOURCE_ROUTE

# reminder that if there's a local DNS such as a Pi-Hole on a different network segment, 
# it will NOT be reachable. 

commit
save

That's it! Right? Well sorta. Um, actually no. Visit https://nord-help.com/ and confirm the top of the page shows the text "Protected":

To see your actual ISP address on the EdgeRouter from commandline (thanks stackoverflow for sed tip):

curl -v --silent https://www.ipchicken.com 2>&1 | grep -A 1 "Address:" \
                   | sed -e 's/<[^>]*>//g'

Another helpful tip is converting the config file to commands, like this:

cli-shell-api --show-commands --show-cfg1 /dev/null --show-cfg2 /config/config.boot showConfig

If it was only that easy; I would not have this blog. Now things get interesting. Reboot the EdgeRouter. You'll see that it ends up in a completely unhappy state: It appears there's no network connectivity at all! The first couple of times I did that, I had to factory reset the hard way.

Actually, with the TTY UART connection, we can see there is network connectivity but the VPN connection is not working. Worse, the problem after reboot seems to be with the routing/switching on the Eth1 .. Eth4 ports.

It appears there's nothing immediately obviously wrong from the logs. The TTY connection is really quite insightful at boot time. Plus you don't have to see their wondering if it crashed, is still booting, or has some other problem.

There's a suggestion to restart the OpenVPN connection like this:

#!/bin/vbash
# Restart OPENVPN Tunnel
VTUN="vtun0"

if [ -f "/var/run/openvpn-$VTUN.pid" ]; then
  sudo kill $(cat "/var/run/openvpn-$VTUN.pid")
  echo $(cat "/var/run/openvpn-$VTUN.pid")
fi
sudo /usr/sbin/openvpn --daemon --verb 3 --writepid /var/run/openvpn-$VTUN.pid \
                       --status /var/run/openvpn/status/$VTUN.status 30 \
                       --config  /config/user-data/openvpn/nordvpn.ovpn \
                       --dev-type tun --dev $VTUN
                       
ls -l /var/run/openvpn*.pid

Use systemctl not /etc/init.d/openvpn

However I found this easier:

# To reset the VPN connection                                          
sudo systemctl stop openvpn.service
sudo systemctl start openvpn.service
sudo systemctl status openvpn.service

Simply resetting the VPN connection after reboot does not resolve the problem. It seems to still be a routing problem.

The ip address command can be used to see what IP addresses are assigned, and which ports are active on the EdgeRouter. Using the ip route command, you should see something like this. Here 192.168.2.x is the WAN network; 192.168.170.x is the EdgeRouter:

0.0.0.0/24 dev vtun0 proto kernel scope link
default via 192.168.2.254 dev eth0 proto zebra
10.8.3.0/24 dev vtun0 proto kernel scope link src 10.8.3.7
192.168.2.0/24 dev eth0 proto kernel scope link src 192.168.2.81
192.168.170.0/24 dev switch0 proto kernel scope link src 192.168.170.1

Note that while it appeared the OpenVPN was working properly ("Protected" at NordVPN), I still saw Active: inactive (dead) here:

ubnt@EdgeRouter-X-5-Port:~$ sudo systemctl status openvpn.service
* openvpn.service - OpenVPN service
   Loaded: loaded (/lib/systemd/system/openvpn.service; disabled; vendor preset:
   Active: inactive (dead)

It seems that after a period of inactivity, perhaps the connection is closed, but once more internet activity is seen, the conection is re=established:

ubnt@EdgeRouter-X-5-Port:~$ sudo systemctl status openvpn.service
* openvpn.service - OpenVPN service
   Loaded: loaded (/lib/systemd/system/openvpn.service; disabled; vendor preset:
   Active: active (exited) since Sun 2021-02-28 20:06:13 UTC; 32s ago
  Process: 4709 ExecStart=/bin/true (code=exited, status=0/SUCCESS)
 Main PID: 4709 (code=exited, status=0/SUCCESS)

One method I found to re-establish a TCP/IP link to the EdgeRouter after the VPN config and reboot is to delete and recreate the interface as described here:

configure
delete interfaces ethernet eth3
set interfaces ethernet eth3 address 192.168.173.1/24
commit ; save

The above configuration commands will setup a routed segment on Eth3 (the second port from the right, next to the PoE one). Assign a manual IP address such as 192.168.173.23 to a PC with manual DNS, and now we are back to having an operational port! This also survives a reboot! Visit your local https://192.168.170.1 to see the EdgeOS web GUI.

There are some notes on setting up DHCP like this:

configure
set service dhcp-server disabled false
set service dhcp-server shared-network-name LAN1 authoritative enable
set service dhcp-server shared-network-name LAN1 subnet 192.168.173.0/24 default-router 192.168.173.1
set service dhcp-server shared-network-name LAN1 subnet 192.168.173.0/24 dns-server 192.168.173.1
set service dhcp-server shared-network-name LAN1 subnet 192.168.173.0/24 lease 86400
set service dhcp-server shared-network-name LAN1 subnet 192.168.173.0/24 start 192.168.173.38 stop 192.168.173.243
commit ; save

If you are no longer debugging and added the this line to your ovpn file, you may wish to remove it:

  log /config/openvpn/vpn.log

Note that if you are using a Raspberry Pi for ad-blocking DNS with the Pi-Hole, (see also my prior blog) then that RPi needs to be separately setup to use the VPN. See http://dnsleaktest.com/

Where I'm currently at:

I'm thinking if I can configure Eth2 .. Eth4 like that, but as a switched segment on the 192.168.170.0 network, perhaps that will survive a reboot as well?

The RT-AX86U:

With some of the problems I saw with the EdgeRouter, I thought I'd try a different router. There's an online Web GUI simulator to test drive some of the Asus features in the firmware UI without actually having your own router.

I tried ASUS AX5700 WiFi 6 Gaming Router (RT-AX86U) as it supports dual WAN, however neither of my ISP devices is LACP-IEEE 802.3ad capable, which is required for ASUS WAN bonded aggregation. Yes, I tried it anyhow, and the performance was unusably poor.

Additionally, despite having a UI to setup OpenVPN and appearance of it being successfully connected to VPN server, it was not actually being used:

I tried contacting Asus support. My first email was bounced back "mailbox full".

Asus support via a web form was not very helpful. In fact, their response claimed to be "confidential and not for publication". I did not agree to a non-disclosure agreement with them, nor does their web page indicate the support response is confidential:

The folks at NordVPN are considerably more helpful and cooperative. They did confirm that sometimes traffic is not routed though OpenVPN connection, even though it is up! This is completely unacceptable. I bought the RT-AX86U specifically as it was on the recommended list of NordVPN routers. I plan to return the router. Not worth $250 to me.

Unlike Asus, NordVPN has no such "confidentiality" in their tech support. So I will share this informative message from their helpful support staff:

If you want a router that is an out-of-the-box solution for NordVPN provided security and anonymity, you should head over to our partners at https://nordvpn.com/flashrouters/ . They will provide you with a preconfigured router that will deliver top-notch security and performance.

You can also buy a router and configure it yourself. We usually recommend these routers for simple home usage and if you do not require much performance (up to 10-40Mbps respectively): Asus RT-N18U, Asus RT-AC66U, Asus RT-AC68U, Asus RT-AC87U, Asus RT-AC3200.

Please note, that while you can get many different routers from other manufacturers, we strongly recommend getting an Asus router, as it has a built-in OpenVPN client on it default firmware and requires minimal effort to set up. Most other routers require firmware flashing, which might be a troublesome procedure and is not supported on all platforms.

You can also acquire any router that supports custom firmware:

DD-WRT: https://www.dd-wrt.com/wiki/index.php/Supported_Devices
Tomato: http://tomato.groov.pl/?page_id=69
Padavan: https://wikidevi.com/wiki/List_of_Padavan_firmware_supported_devices
OpenWRT: https://wiki.openwrt.org/toh/start
Merlin: https://github.com/RMerl/asuswrt-merlin/wiki/Supported-Devices

If you want a router that provides the best performance possible (in best case scenario up to 60Mbps), please look at these ones:

Asus: RT-AC66U B1, RT-AC68U, RT-AC86U, RT-AX3000, RT-AX82U, RT-AX86U, RT-AX88U, RT-AC5300
D-Link (Would need to be flashed with DD-WRT): DIR-885L, DIR-895L
Linksys (Would need to be flashed with DD-WRT): EA8500, WRT1900ACS, WRT3200ACM
Netgear (Would need to be flashed with DD-WRT): R7500, R7800, R8500, R9000