(2021-03-01) Massive Server Upgrade

I've now slid into choosing now to be the time for making lots of changes to my Linode

Going to try to do all this on a new linode, then transfer the IP#.

Mar25 finally facing it

  • Ubuntu 20.04 LTS, dedicated $30/mo plus $5 backup
  • root password xxx
  • ssh/glish in
  • follow some bits here
  • sudo apt install nginx
  • go back to do security
  • adduser xxx password yyy, adduser xxx sudo
  • not bothering with key pairs etc
  • nano /etc/ssh/sshd_configPermitRootLogin no
  • sudo systemctl restart sshd
  • apt-get install fail2ban for Fail2Ban
  • sudo apt-get install ufw for ufw
  • sudo ufw allow ssh
  • sudo ufw enable
sudo ufw default allow outgoing
sudo ufw default deny incoming
sudo ufw allow http
sudo ufw allow https
sudo ufw allow 25
cd /srv
mkdir www
cd www
mkdir wikiflux.net
cd wikiflux.net
mkdir app
mkdir logs
cd ..
mkdir wikigraph.net
cd wikigraph.net
mkdir app
mkdir logs
(no doing other domains for now)
  • it already responds to python3.... 3.8.5
  • sudo apt install python3-pipThe following NEW packages will be installed: binutils binutils-common binutils-x86-64-linux-gnu build-essential cpp cpp-9 dpkg-dev fakeroot g++ g++-9 gcc gcc-9 gcc-9-base libalgorithm-diff-perl libalgorithm-diff-xs-perl libalgorithm-merge-perl libasan5 libatomic1 libbinutils libc-dev-bin libc6-dev libcc1-0 libcrypt-dev libctf-nobfd0 libctf0 libdpkg-perl libexpat1-dev libfakeroot libfile-fcntllock-perl libgcc-9-dev libgomp1 libisl22 libitm1 liblsan0 libmpc3 libpython3-dev libpython3.8-dev libquadmath0 libstdc++-9-dev libtsan0 libubsan1 linux-libc-dev make manpages-dev python-pip-whl python3-dev python3-pip python3-wheel python3.8-dev zlib1g-dev
  • runs through bunch of mirrors, ends with
Get:50 http://mirrors.linode.com/ubuntu focal-updates/universe amd64 python3-pip all 20.0.2-5ubuntu1.1 [230 kB]
Err:8 http://mirrors.linode.com/ubuntu focal-updates/main amd64 linux-libc-dev amd64 5.4.0-66.74
  404  Not Found [IP: 2600:3c00:1::68c8:17a2 80]
Fetched 45.6 MB in 0s (113 MB/s)
E: Failed to fetch http://mirrors.linode.com/ubuntu/pool/main/l/linux/linux-libc-dev_5.4.0-66.74_amd64.deb  404  Not Found [IP: 2600:3c00:1::68c8:17a2 80]
E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?
  • ah, did sudo apt-get update then tried again and it worked
  • not bothering with virtualenv
  • pip3 install flask
  • python3 -m flask --version
Python 3.8.5
Flask 1.1.2
Werkzeug 1.0.1
  • cd /srv/www/wikiflux.net/app
  • nano app.py paste/save from here
  • sudo ufw allow 5000
  • python3 app.py → says Running on http://127.0.0.1:5000/
  • but if I try to hit it from home, get no response
  • but if I ssh into box, do curl localhost:5000, I get the HTML response, so that's something....

Mar27

  • sticking with uwsgi
  • currently I launch with start_uwsgi_app.sh in my server's /app/ directory, containing:
#!/bin/bash -e
uwsgi --socket :9004 --file /srv/www/wikiflux.net/app/wikiweb.py --callable app --processes 2 >/srv/www/wikiflux.net/logs/uwsgi_app.log 2>&1 &
  • make test variant: start_uwsgi_app_test.sh
#!/bin/bash -e
uwsgi --socket :9004 --file /srv/www/wikiflux.net/app/app.py --callable app --processes 2 >/srv/www/wikiflux.net/logs/uwsgi_app_test.log 2>&1 &
  • trying to scp a copy of my working/live nginx.conf file up - getting denied. Finally manage to lock myself out.
  • don't see myself listed in iptables
  • sudo fail2ban-client stop
  • sudo ufw disable
  • now I can scp
  • derp that was the wrong conf - was /etc/nginx/nginx.conf but the real one was at /opt/nginx/conf/nginx.conf - now copied over
  • I think my plan is to create a minimal nginx.conf from my existing file and the flask-intro example... oh wait there's also the default nginx.conf on the new linode....

Mar28

  • So I think my plan is to take the new-linode default file, and merge in the minimal bits necessary to talk to uwsgi+flask.
  • nginx -s reloadnginx: [emerg] "location" directive is not allowed here in /etc/nginx/nginx.conf:47... I guess it needs to be within a server block
  • added server block, server_name .flux.garden, 45.33.14.189;.... reload → no complaint, but hit ip, get 502 Bad Gateway
  • check uwsgi_app_test.log, find start_uwsgi_app_test.sh: line 2: uwsgi: command not found well derp
  • command-line uwsgiCommand 'uwsgi' not found, but can be installed with:.... hmm which one?
  • this looks like a good reference, but still gives some choices... I think I'll take the pip approach, esp since I don't want to auto-daemonize it, since I'll have separate instances for separate domains....
  • sudo apt-get install python-dev hrm I see lots of python-2 stuff floating by...
  • so add sudo apt-get install python3-dev
  • sudo pip3 install uwsgiSuccessfully installed uwsgi-2.0.19.1
  • nginx reload, hit URL, same 502
  • derp rerun bash start_uwsgi_app_test.sh - now hit URL and get success! http://45.33.14.189/
  • hrm what's next? get it responding to https://flux.garden/ and get SSL working
  • so change DNS for just this domain to point to new linode IP
  • hit http://flux.garden and get generic nginx home page, http://webseitz.flux.garden same, but hit IP and get the flask page.
  • not seeing the nginx hits in access.log or uwsgi_app_test.log
  • there are entries in /var/log/nginx/access.log
  • double-check the server_name docs, the wildcard is right, oops need list to be space-separated not comma-separated. Fix, reload → success!
  • using CertBot/LetsEncrypt doc
  • add-apt-repository ppa:certbot/certbotThe PPA has been DEPRECATED. To get up to date instructions on how to get certbot for your systems, please see https://certbot.eff.org/docs/install.html. More info: https://launchpad.net/~certbot/+archive/ubuntu/certbot
  • apt-get install certbot
  • apt-get install python3-certbot-nginx
  • certbot --nginxSaving debug log to /var/log/letsencrypt/letsencrypt.log
  • hmm if I want SSL for all the hostnames in a domain, do I need wildcards? Some CAs (such as Let’s Encrypt) require that domain validation for wildcard domains must be done through modifications to DNS records which means that the dns-01 challenge type must be used. I guess that means yes. Check linode plugin info which leads to here
  • snap should already be installed
  • snap install core; sudo snap refresh core
  • apt-get remove certbot
  • snap install --classic certbot
  • `ln -s /snap/bin/certbot /usr/bin/certbot*
  • snap set certbot trust-plugin-with-root=ok
  • snap install certbot-dns-linode
  • looks like I need to generate a "Personal Access Token", but unclear on the details.... just "Domains", just "Read Only"?

Mar29

  • I guess I'll give it read-only for all types
  • store it in /.secrets/certbot/linode.ini
  • chmod 600 linode.ini
  • run
certbot certonly \
  --dns-linode \
  --dns-linode-credentials /.secrets/certbot/linode.ini \
  -d *.flux.garden
  • Encountered exception during recovery: ValueError: invalid literal for int() with base 10: '[<blank>|3|4]'
  • I guess I was being too literal. I'll change the file to just dns_linode_version = 3
  • try again → Unexpected error determining zone identifier for flux.garden: Linode api error: [{'ERRORMESSAGE': 'Authentication failed', 'ERRORCODE': 4}]
  • changed the domain arg to -d "*.flux.garden" → same fail

Mar30

  • support has responded: it looks like you'll want to be sure that the API token has read/write permissions for Domains. The reason for this is the plugin adds and then removes a TXT record from your DNS in order to process the DNS challenge created by Let's Encrypt. Creating a token with just that permission should do the trick.
  • revoke the old token, create new one, edit the linode.ini, run same command, get exactly same outcome.
  • derp, I realize this whole system is about the plugin mucking with my DNS nameserver. But I still have my registrar, Hover, as my nameserver! So need to follow these instructions.
  • change the Hover nameservers to point to ns1.linode.com etc
  • create flux.garden domain at linode, let it "insert default records"
  • re-run the certbot command, get exact same result. Maybe need to give it some time....
  • restating more verbosely, calling
certbot certonly \
  --dns-linode \
  --dns-linode-credentials /.secrets/certbot/linode.ini \
  -d "*.flux.garden"
  • results in
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator dns-linode, Installer None
Requesting a certificate for *.flux.garden
Performing the following challenges:
dns-01 challenge for flux.garden
Cleaning up challenges
Unexpected error determining zone identifier for flux.garden: Linode api error: [{'ERRORMESSAGE': 'Authentication failed', 'ERRORCODE': 4}]

Mar31

  • no support responses
  • try https://letsdebug.net/ get StatusNotOperational - The current status as reported by the Let's Encrypt status page is Service Disruption well that's certainly clear.
    • hit the Verbose button, get back 4 blue blocks - what order do they matter?
    • First one is: An internal error occurred while checking the domain... Failed to query certwatch database to check rate limits
    • 2nd is: Challenge update failures for webseitz.flux.garden in order https://acme-staging-v02.api.letsencrypt.org/acme/order/5751349/22595578 acme: error code 400 "urn:ietf:params:acme:error:dns": DNS problem: NXDOMAIN looking up TXT for _acme-challenge.webseitz.flux.garden - check that a DNS record exists for this domain
  • check https://check-your-website.server-daten.de/?q=flux.garden items that seem important
    • IP lookup fails for *.flux.garden for A, AAAA, CNAME
    • connect failure for https://flux.garden/, which my browser agrees with - hrm is that an nginx.conf issue, or uwsgi, or what? And is that even relevant to the certbot issue?
  • going to get that SSL working without the cert, so edit block of nginx.conf
	server {
		listen   80;
		listen 443 ssl;
		server_name .flux.garden 45.33.14.189;
		if ($scheme = http) {
        return 301 https://$server_name$request_uri;
    }
  • nope not buying that:
nginx -s reload
nginx: [emerg] no "ssl_certificate" is defined for the "listen ... ssl" directive in /etc/nginx/nginx.conf:37
  • ok let's set up for the cert/key even if we don't have it
  • mkdir -p /root/certs/flux.garden/
  • edit nginx.conf as though I had something in there, put the ssl_certificate/key bits in the same server block, pointing to files which don't actually exist.
  • restart → nginx: [emerg] "ssl_certificate" directive is not allowed here in /etc/nginx/nginx.conf:43 though I see it in a server block here
  • moved the cert lines up a few lines within the block so they come before server_name (though I see example that put them after)
  • restart → fail, ok at least that makes sense
nginx: [emerg] cannot load certificate "/root/certs/flux.garden/flux.garden.crt": BIO_new_file() failed (SSL: error:02001002:system library:fopen:No such file or directory:fopen('/root/certs/flux.garden/flux.garden.crt','r') error:2006D080:BIO routines:BIO_new_file:no such file)
  • but I feel like I'm in a chicken/egg situation here.... maybe all the HTTP stuff is irrelevant to the certbot issues, and once I settle that, then I turn on SSL support using that cert for the server....
  • ah, support responded...
  • ah, here's someone with the same error-msg I had: This looks to be due to a known issue that the Linode DNS plugin is not available because it was introduced after the PPA was created. Which then suggests this.

Meta: would this have been any easier with AWS?


Edited: |