[howto] Fix certbot-auto renew failure (404)

Here's how to fix this type of error:

-------------------------------------------------------------------------------

Processing /etc/letsencrypt/renewal/my-domain-example.com.conf

-------------------------------------------------------------------------------

Renewing an existing certificate

Performing the following challenges:

http-01 challenge for my-domain-example.com

http-01 challenge for www.my-domain-example.com

Waiting for verification...

Cleaning up challenges

Attempting to renew cert from /etc/letsencrypt/renewal/my-domain-example.com.conf produced an unexpected error: Failed authorization procedure. my-domain-example.com (http-01): urn:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from http://my-domain-example.com/.well-known/acme-challenge/0s8Ci_55qtap46cwbXbL8uhGRRnf_M7_tkcH1DjObjU: "

404 Not Found

---

", www.my-domain-example.com (http-01): urn:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from http://www.my-domain-example.com/.well-known/acme-challenge/3ZwAH4Q5mgHx3hCKty7OYsDHgh9ZiwlgrI-JnSMA6XQ: "

404 Not Found

---

". Skipping.

All renewal attempts failed. The following certs could not be renewed:

  /etc/letsencrypt/live/my-domain-example.com/fullchain.pem (failure)

1 renew failure(s), 0 parse failure(s)

IMPORTANT NOTES:

 - The following errors were reported by the server:

   Domain: my-domain-example.com

   Type:   unauthorized

   Detail: Invalid response from

   http://my-domain-example.com/.well-known/acme-challenge/:

   "

   

   

   

404 Not Found

   

---

"

   Domain: www.my-domain-example.com

   Type:   unauthorized

   Detail: Invalid response from

   http://www.my-domain-example.com/.well-known/acme-challenge/:

   "

   

   

   

404 Not Found

   

---

"

   To fix these errors, please make sure that your domain name was

   entered correctly and the DNS A record(s) for that domain

   contain(s) the right IP address.

The error is showing that the client is getting a 404 error webpage as a response to its challenge. The reason this seemed to be happening was that I had HSTS on, which prevented the challenge from going through properly.

For me the the answer was to disable SSL for that website. Then run the certbot-auto renew. This allowed the script to renew the certs. Afterwards simply re-enable SSL.

The simplest way for me to disable SSL was to remove the relevant entries in my config file.

/etc/nginx/sites-available/