Here's how to fix this type of error:
-------------------------------------------------------------------------------
Processing /etc/letsencrypt/renewal/my-domain-example.com.conf
-------------------------------------------------------------------------------
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for my-domain-example.com
http-01 challenge for www.my-domain-example.com
Waiting for verification...
Cleaning up challenges
Attempting to renew cert from /etc/letsencrypt/renewal/my-domain-example.com.conf produced an unexpected error: Failed authorization procedure. my-domain-example.com (http-01): urn:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from http://my-domain-example.com/.well-known/acme-challenge/0s8Ci_55qtap46cwbXbL8uhGRRnf_M7_tkcH1DjObjU: "
404 Not Found
404 Not Found
", www.my-domain-example.com (http-01): urn:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from http://www.my-domain-example.com/.well-known/acme-challenge/3ZwAH4Q5mgHx3hCKty7OYsDHgh9ZiwlgrI-JnSMA6XQ: "
404 Not Found
404 Not Found
". Skipping.
All renewal attempts failed. The following certs could not be renewed:
/etc/letsencrypt/live/my-domain-example.com/fullchain.pem (failure)
1 renew failure(s), 0 parse failure(s)
IMPORTANT NOTES:
- The following errors were reported by the server:
Domain: my-domain-example.com
Type: unauthorized
Detail: Invalid response from
http://my-domain-example.com/.well-known/acme-challenge/:
"
404 Not Found
404 Not Found
"
Domain: www.my-domain-example.com
Type: unauthorized
Detail: Invalid response from
http://www.my-domain-example.com/.well-known/acme-challenge/:
"
404 Not Found
404 Not Found
"
To fix these errors, please make sure that your domain name was
entered correctly and the DNS A record(s) for that domain
contain(s) the right IP address.
The error is showing that the client is getting a 404 error
webpage as a response to its challenge. The reason this seemed to be happening was that I had HSTS on, which prevented the challenge from going through properly.
For me the the answer was to disable SSL for that website. Then run the certbot-auto renew. This allowed the script to renew the certs. Afterwards simply re-enable SSL.
The simplest way for me to disable SSL was to remove the relevant entries in my config file.
/etc/nginx/sites-available/