(2021-03-01) Massive Server Upgrade

I've now slid into choosing now to be the time for making lots of changes to my Linode

update Ubuntu
get Python-3 in place for my Flask FluxGarden server, while still having Python-2 for my legacy web.py servers - (2021-02-07) Migrating to Python3
get SSL working, at least for FluxGarden, and maybe for fluxent.com.
get my updated Flask code running - new structure, etc.

Going to try to do all this on a new linode, then transfer the IP#.

Mar25 finally facing it

Ubuntu 20.04 LTS, dedicated $30/mo plus $5 backup
root password xxx
ssh/glish in
follow some nginx bits here
sudo apt install nginx
go back to do security
adduser xxx password yyy, adduser xxx sudo
not bothering with key pairs etc
nano /etc/ssh/sshd_config → PermitRootLogin no
sudo systemctl restart sshd
apt-get install fail2ban for Fail2Ban
sudo apt-get install ufw for ufw
sudo ufw allow ssh
sudo ufw enable

sudo ufw default allow outgoing
sudo ufw default deny incoming
sudo ufw allow http
sudo ufw allow https
sudo ufw allow 25

cd /srv
mkdir www
cd www
mkdir wikiflux.net
cd wikiflux.net
mkdir app
mkdir logs
cd ..
mkdir wikigraph.net
cd wikigraph.net
mkdir app
mkdir logs
(no doing other domains for now)

it already responds to python3.... 3.8.5
sudo apt install python3-pip → The following NEW packages will be installed: binutils binutils-common binutils-x86-64-linux-gnu build-essential cpp cpp-9 dpkg-dev fakeroot g++ g++-9 gcc gcc-9 gcc-9-base libalgorithm-diff-perl libalgorithm-diff-xs-perl libalgorithm-merge-perl libasan5 libatomic1 libbinutils libc-dev-bin libc6-dev libcc1-0 libcrypt-dev libctf-nobfd0 libctf0 libdpkg-perl libexpat1-dev libfakeroot libfile-fcntllock-perl libgcc-9-dev libgomp1 libisl22 libitm1 liblsan0 libmpc3 libpython3-dev libpython3.8-dev libquadmath0 libstdc++-9-dev libtsan0 libubsan1 linux-libc-dev make manpages-dev python-pip-whl python3-dev python3-pip python3-wheel python3.8-dev zlib1g-dev
runs through bunch of mirrors, ends with

Get:50 http://mirrors.linode.com/ubuntu focal-updates/universe amd64 python3-pip all 20.0.2-5ubuntu1.1 [230 kB]
Err:8 http://mirrors.linode.com/ubuntu focal-updates/main amd64 linux-libc-dev amd64 5.4.0-66.74
  404  Not Found [IP: 2600:3c00:1::68c8:17a2 80]
Fetched 45.6 MB in 0s (113 MB/s)
E: Failed to fetch http://mirrors.linode.com/ubuntu/pool/main/l/linux/linux-libc-dev_5.4.0-66.74_amd64.deb  404  Not Found [IP: 2600:3c00:1::68c8:17a2 80]
E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?

ah, did sudo apt-get update then tried again and it worked
not bothering with virtualenv
pip3 install flask
python3 -m flask --version

Python 3.8.5
Flask 1.1.2
Werkzeug 1.0.1

cd /srv/www/wikiflux.net/app
nano app.py paste/save from here
sudo ufw allow 5000
python3 app.py → says Running on http://127.0.0.1:5000/
but if I try to hit it from home, get no response
but if I ssh into box, do curl localhost:5000, I get the HTML response, so that's something....

Mar27

sticking with uwsgi
currently I launch with start_uwsgi_app.sh in my server's /app/ directory, containing:

#!/bin/bash -e
uwsgi --socket :9004 --file /srv/www/wikiflux.net/app/wikiweb.py --callable app --processes 2 >/srv/www/wikiflux.net/logs/uwsgi_app.log 2>&1 &

make test variant: start_uwsgi_app_test.sh

#!/bin/bash -e
uwsgi --socket :9004 --file /srv/www/wikiflux.net/app/app.py --callable app --processes 2 >/srv/www/wikiflux.net/logs/uwsgi_app_test.log 2>&1 &

trying to scp a copy of my working/live nginx.conf file up - getting denied. Finally manage to lock myself out.
don't see myself listed in iptables
sudo fail2ban-client stop
sudo ufw disable
now I can scp
derp that was the wrong conf - was /etc/nginx/nginx.conf but the real one was at /opt/nginx/conf/nginx.conf - now copied over
I think my plan is to create a minimal nginx.conf from my existing file and the flask-intro example... oh wait there's also the default nginx.conf on the new linode....

Mar28

So I think my plan is to take the new-linode default file, and merge in the minimal bits necessary to talk to uwsgi+flask.
nginx -s reload → nginx: [emerg] "location" directive is not allowed here in /etc/nginx/nginx.conf:47... I guess it needs to be within a server block
added server block, server_name .flux.garden, 45.33.14.189;.... reload → no complaint, but hit ip, get 502 Bad Gateway
check uwsgi_app_test.log, find start_uwsgi_app_test.sh: line 2: uwsgi: command not found well derp
command-line uwsgi → Command 'uwsgi' not found, but can be installed with:.... hmm which one?
this looks like a good reference, but still gives some choices... I think I'll take the pip approach, esp since I don't want to auto-daemonize it, since I'll have separate instances for separate domains....
sudo apt-get install python-dev hrm I see lots of python-2 stuff floating by...
so add sudo apt-get install python3-dev
sudo pip3 install uwsgi → Successfully installed uwsgi-2.0.19.1
nginx reload, hit URL, same 502
derp rerun bash start_uwsgi_app_test.sh - now hit URL and get success! http://45.33.14.189/
hrm what's next? get it responding to https://flux.garden/ and get SSL working
so change DNS for just this domain to point to new linode IP
hit http://flux.garden and get generic nginx home page, http://webseitz.flux.garden same, but hit IP and get the flask page.
not seeing the nginx hits in access.log or uwsgi_app_test.log
there are entries in /var/log/nginx/access.log
double-check the server_name docs, the wildcard is right, oops need list to be space-separated not comma-separated. Fix, reload → success!
using CertBot/LetsEncrypt doc
add-apt-repository ppa:certbot/certbot → The PPA has been DEPRECATED. To get up to date instructions on how to get certbot for your systems, please see https://certbot.eff.org/docs/install.html. More info: https://launchpad.net/~certbot/+archive/ubuntu/certbot
apt-get install certbot
apt-get install python3-certbot-nginx
certbot --nginx → Saving debug log to /var/log/letsencrypt/letsencrypt.log
hmm if I want SSL for all the hostnames in a domain, do I need wildcards? Some CAs (such as Let’s Encrypt) require that domain validation for wildcard domains must be done through modifications to DNS records which means that the dns-01 challenge type must be used. I guess that means yes. Check linode plugin info which leads to here
snap should already be installed
snap install core; sudo snap refresh core
apt-get remove certbot
snap install --classic certbot
`ln -s /snap/bin/certbot /usr/bin/certbot*
snap set certbot trust-plugin-with-root=ok
snap install certbot-dns-linode
looks like I need to generate a "Personal Access Token", but unclear on the details.... just "Domains", just "Read Only"?

Mar29

I guess I'll give it read-only for all types
store it in /.secrets/certbot/linode.ini
chmod 600 linode.ini
run

certbot certonly \
  --dns-linode \
  --dns-linode-credentials /.secrets/certbot/linode.ini \
  -d *.flux.garden

→ Encountered exception during recovery: ValueError: invalid literal for int() with base 10: '[<blank>|3|4]'
I guess I was being too literal. I'll change the file to just dns_linode_version = 3
try again → Unexpected error determining zone identifier for flux.garden: Linode api error: [{'ERRORMESSAGE': 'Authentication failed', 'ERRORCODE': 4}]
changed the domain arg to -d "*.flux.garden" → same fail

Mar30

support has responded: it looks like you'll want to be sure that the API token has read/write permissions for Domains. The reason for this is the plugin adds and then removes a TXT record from your DNS in order to process the DNS challenge created by Let's Encrypt. Creating a token with just that permission should do the trick.
revoke the old token, create new one, edit the linode.ini, run same command, get exactly same outcome.
derp, I realize this whole system is about the plugin mucking with my DNS nameserver. But I still have my registrar, Hover, as my nameserver! So need to follow these instructions.
change the Hover nameservers to point to ns1.linode.com etc
create flux.garden domain at linode, let it "insert default records"
re-run the certbot command, get exact same result. Maybe need to give it some time....
restating more verbosely, calling

certbot certonly \
  --dns-linode \
  --dns-linode-credentials /.secrets/certbot/linode.ini \
  -d "*.flux.garden"

results in

Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator dns-linode, Installer None
Requesting a certificate for *.flux.garden
Performing the following challenges:
dns-01 challenge for flux.garden
Cleaning up challenges
Unexpected error determining zone identifier for flux.garden: Linode api error: [{'ERRORMESSAGE': 'Authentication failed', 'ERRORCODE': 4}]

Mar31

no support responses
try https://letsdebug.net/ get StatusNotOperational - The current status as reported by the Let's Encrypt status page is Service Disruption well that's certainly clear.
- hit the Verbose button, get back 4 blue blocks - what order do they matter?
- First one is: An internal error occurred while checking the domain... Failed to query certwatch database to check rate limits
- 2nd is: Challenge update failures for webseitz.flux.garden in order https://acme-staging-v02.api.letsencrypt.org/acme/order/5751349/22595578 acme: error code 400 "urn:ietf:params:acme:error:dns": DNS problem: NXDOMAIN looking up TXT for _acme-challenge.webseitz.flux.garden - check that a DNS record exists for this domain
check https://check-your-website.server-daten.de/?q=flux.garden items that seem important
- IP lookup fails for *.flux.garden for A, AAAA, CNAME
- connect failure for https://flux.garden/, which my browser agrees with - hrm is that an nginx.conf issue, or uwsgi, or what? And is that even relevant to the certbot issue?
going to get that SSL working without the cert, so edit block of nginx.conf

	server {
		listen   80;
		listen 443 ssl;
		server_name .flux.garden 45.33.14.189;
		if ($scheme = http) {
        return 301 https://$server_name$request_uri;
    }

nope not buying that:

nginx -s reload
nginx: [emerg] no "ssl_certificate" is defined for the "listen ... ssl" directive in /etc/nginx/nginx.conf:37

ok let's set up for the cert/key even if we don't have it
mkdir -p /root/certs/flux.garden/
edit nginx.conf as though I had something in there, put the ssl_certificate/key bits in the same server block, pointing to files which don't actually exist.
restart → nginx: [emerg] "ssl_certificate" directive is not allowed here in /etc/nginx/nginx.conf:43 though I see it in a server block here
moved the cert lines up a few lines within the block so they come before server_name (though I see example that put them after)
restart → fail, ok at least that makes sense

nginx: [emerg] cannot load certificate "/root/certs/flux.garden/flux.garden.crt": BIO_new_file() failed (SSL: error:02001002:system library:fopen:No such file or directory:fopen('/root/certs/flux.garden/flux.garden.crt','r') error:2006D080:BIO routines:BIO_new_file:no such file)

but I feel like I'm in a chicken/egg situation here.... maybe all the HTTP stuff is irrelevant to the certbot issues, and once I settle that, then I turn on SSL support using that cert for the server....
ah, support responded...
ah, here's someone with the same error-msg I had: This looks to be due to a known issue that the Linode DNS plugin is not available because it was introduced after the PPA was created. Which then suggests this.

Meta: would this have been any easier with AWS?

Apr24 locked out

Apr25

ok let's try acme.sh
curl https://get.acme.sh | sh -s email=bill@simplest-thing.com →

[Sun Apr 25 20:18:04 UTC 2021] Extracting master.tar.gz
[Sun Apr 25 20:18:04 UTC 2021] It is recommended to install socat first.
[Sun Apr 25 20:18:04 UTC 2021] We use socat for standalone server if you use standalone mode.
[Sun Apr 25 20:18:04 UTC 2021] If you don't use standalone mode, just ignore this warning.
[Sun Apr 25 20:18:04 UTC 2021] Installing to /root/.acme.sh
[Sun Apr 25 20:18:04 UTC 2021] Installed to /root/.acme.sh/acme.sh
[Sun Apr 25 20:18:04 UTC 2021] Installing alias to '/root/.bashrc'
[Sun Apr 25 20:18:04 UTC 2021] OK, Close and reopen your terminal to start using acme.sh
[Sun Apr 25 20:18:04 UTC 2021] Installing cron job
no crontab for root
no crontab for root
[Sun Apr 25 20:18:04 UTC 2021] Good, bash is found, so change the shebang to use bash as preferred.
[Sun Apr 25 20:18:04 UTC 2021] OK
[Sun Apr 25 20:18:04 UTC 2021] Install success!

do crontab -e confirm the line is there
do acme.sh --issue --nginx -d flux.garden -d '*.flux.garden' --dns dns_cf →

[Sun Apr 25 20:31:40 UTC 2021] Using CA: https://acme-v02.api.letsencrypt.org/directory
[Sun Apr 25 20:31:40 UTC 2021] Create account key ok.
[Sun Apr 25 20:31:40 UTC 2021] Registering account: https://acme-v02.api.letsencrypt.org/directory
[Sun Apr 25 20:31:41 UTC 2021] Registered
[Sun Apr 25 20:31:41 UTC 2021] ACCOUNT_THUMBPRINT='u-bAel3mP4xzjlL9y92Nd8ELL0W0XMFXJgGICUMjQTU'
[Sun Apr 25 20:31:41 UTC 2021] Creating domain key
[Sun Apr 25 20:31:41 UTC 2021] The domain key is here: /root/.acme.sh/flux.garden/flux.garden.key
[Sun Apr 25 20:31:41 UTC 2021] Multi domain='DNS:flux.garden,DNS:*.flux.garden'
[Sun Apr 25 20:31:41 UTC 2021] Getting domain auth token for each domain
[Sun Apr 25 20:31:42 UTC 2021] Getting webroot for domain='flux.garden'
[Sun Apr 25 20:31:42 UTC 2021] Getting webroot for domain='*.flux.garden'
[Sun Apr 25 20:31:42 UTC 2021] Adding txt value: S38j3yF4qE5LKBgnKcYGwFg2icYIWrEQIBpyoS8afQ0 for domain:  _acme-challenge.flux.garden
[Sun Apr 25 20:31:42 UTC 2021] You didn't specify a Cloudflare api key and email yet.
[Sun Apr 25 20:31:42 UTC 2021] You can get yours from here https://dash.cloudflare.com/profile.
[Sun Apr 25 20:31:42 UTC 2021] Error add txt for domain:_acme-challenge.flux.garden
[Sun Apr 25 20:31:42 UTC 2021] Please add '--debug' or '--log' to check more details.
[Sun Apr 25 20:31:42 UTC 2021] See: https://github.com/acmesh-official/acme.sh/wiki/How-to-debug-acme.sh

do I really have to do that Cloudflare stuff? I guess I will... create account with bill@simplest-thing.com then go to https://dash.cloudflare.com/profile/api-tokens, get the "Origin CA Key"
find /root/.acme.sh/account.conf, it includes lines ACCOUNT_EMAIL='bill@simplest-thing.com' and UPGRADE_HASH='{not sharing} - do I replace that CloudFlare API key for the HASH?
no, it seems like I should do export CF_Token="{cloudflare key}" so do that
repeat acme.sh --issue --nginx -d flux.garden -d '*.flux.garden' --dns dns_cf →

[Sun Apr 25 21:48:59 UTC 2021] Using CA: https://acme-v02.api.letsencrypt.org/directory
[Sun Apr 25 21:48:59 UTC 2021] Multi domain='DNS:flux.garden,DNS:*.flux.garden'
[Sun Apr 25 21:48:59 UTC 2021] Getting domain auth token for each domain
[Sun Apr 25 21:49:00 UTC 2021] Getting webroot for domain='flux.garden'
[Sun Apr 25 21:49:00 UTC 2021] Getting webroot for domain='*.flux.garden'
[Sun Apr 25 21:49:00 UTC 2021] Adding txt value: YyR47c2XIe1-v1ime4xXzFPfBjlgb7s9MUZlprH-Exo for domain:  _acme-challenge.flux.garden
[Sun Apr 25 21:49:00 UTC 2021] invalid domain
[Sun Apr 25 21:49:00 UTC 2021] Error add txt for domain:_acme-challenge.flux.garden
[Sun Apr 25 21:49:00 UTC 2021] Please add '--debug' or '--log' to check more details.
[Sun Apr 25 21:49:00 UTC 2021] See: https://github.com/acmesh-official/acme.sh/wiki/How-to-debug-acme.sh

well that's a different outcome but still a fail

On to (2021-04-25) Move To DigitalOcean

Edited: 2022-06-23 15:20:35.408803 | Tweet this! | Search Twitter for discussion

Bill Seitz