Letsencrypt: pitfall with renewing

Recently I had to work on a Ubuntu 18.04 system where a colleague installed letsencrypt and created the initial certificates. There was no problem so far and everything seemed to be ok until the certs had to be renewed. They simply did not and we had to find the reason because the initial certificate neared expiration. Of course we could have just do it manually but you surely agree that this is something that should work automatically.

I started the search and the colleague who initially installed letsencrypt started as well. It turned out, that the reason for the renewal not working was that my colleague used the standalone authentication when initializing the certificates and now when Apache is running, certbot was unable to acquire port 80. So we installed the additional package python-certbot-apache and added the option --apache to the crontab entry in /etc/cron.d/cron. After a manual update we thought we would be done with this.

But two months later it turned out that we were not: Again, the certificate was not renewed. Again, from the logfile (/var/log/letsencrypt/letsencrypt.log) it turned out that the renewal was tried by using the standalone authenticator!

Now I started a deeper search and had a closer look at the line in the crontab:

0 */12 * * * root test -x /usr/bin/certbot -a \! -d /run/systemd/system && perl -e 'sleep int(rand(43200))' && certbot --apache -q renew

And I noticed, that the renewal command was never executed because of the test command in from of it!

So I had to not e that on systems using systemd this crontab entry never executes because a respective entry in systemd exists!

The service entry can be found in /lib/systemd/system/certbot.service, the scheduling for it in /lib/systemd/system/certbot.timer.

This entry also contained no hint to using apache and I was already going to create an override entry for the cerbot service when I noticed that there is a configuration file for every certificate that shall be renewed by certbot.

It looked like this:

# renew_before_expiry = 30 days
version = 0.27.0
archive_dir = /etc/letsencrypt/archive/xxx.ch
cert = /etc/letsencrypt/live/xxx.ch/cert.pem
privkey = /etc/letsencrypt/live/xxx.ch/privkey.pem
chain = /etc/letsencrypt/live/xxx.ch/chain.pem
fullchain = /etc/letsencrypt/live/xxx.ch/fullchain.pem

# Options used in the renewal process
account = b61bb893aa21cbd202aaaaaaaaaaaaaa
server = https://acme-v02.api.letsencrypt.org/directory

It is not recommended by letsencrypt to modify this file manually but in this case I had to because the configuration of the renewal changed after the first installation. So I added these two lines to the section "renewalparams":

authenticator = apache
renew_hook = systemctl reload apache2

Now, with these parameters switched on the renewal finally worked as expected!