Caddy reload --force won't recache certificates

[wisammechano]

I am updating the certificates using the cert_obtained hook but when I reload caddy with --force, it won't update the cached certificates.

I am using docker and currently I have to restart the container after certificates are obtained.

I have configured a CA that caddy need to use to generate certificates however as discussed here caddy doesn’t generate the fullchain for the host. Using the suggested approach to listen to cert_obtained, I can successfully generate the fullchain certificate for each host. However from inside the script, I can’t let caddy reload these certs.

As you can see in the logs below, there is no indication of re-caching certificates and certificate hash is the same (when generated and when requested with openssl s_client -showcerts -connect wnaji.dev:443 (wnaji.dev points to the ip address of the machine in /etc/hosts locally)

I have tried visiting the website as well, it only shows the caddy-root.crt and the caddy generated cert.

I logged in to the container to check /data/caddy/certificates/local/wnaji.dev/wnaji.dev.crt and it is already merged with the fullchain so the script is working ok but caddy is not revalidating the certificates cache.

More details: https://caddy.community/t/force-reload-certificates/27545

Log output
I ommitted some of the repeated messages as you can see I have configured lots of hosts and all issued certificates.

{"level":"info","ts":1735515098.5544586,"msg":"using config from file","file":"/etc/caddy/Caddyfile"}
{"level":"info","ts":1735515098.561638,"msg":"adapted config to JSON","adapter":"caddyfile"}
{"level":"info","ts":1735515098.5658405,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc0016e8280"}
{"level":"info","ts":1735515098.5715077,"logger":"http.auto_https","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
{"level":"info","ts":1735515098.5715418,"logger":"http.auto_https","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
{"level":"warn","ts":1735515098.5715597,"logger":"http.auto_https","msg":"server is listening only on the HTTP port, so no automatic HTTPS will be applied to this server","server_name":"srv1","http_port":80}
{"level":"debug","ts":1735515098.5716536,"logger":"http.auto_https","msg":"adjusted config","tls":{"automation":{"policies":[{"subjects":["wnaji.local"]},{}]}},"http":{"servers":{"srv0":{"listen":[":443"],"routes":[{"handle":[{"handler":"subroute","routes":[{"handle":[{"REAL_HOST":"{http.request.host}","handler":"vars"},{"REAL_IP":"{http.request.remote.host}","handler":"vars"}]},{"handle":[{"REAL_HOST":"{http.request.header.X-Forwarded-Host}","handler":"vars"}],"match":[{"header":{"X-Forwarded-Host":["*"]}}]},{"handle":[{"REAL_IP":"{http.request.header.Cf-Connecting-Ip}","handler":"vars"}],"match":[{"header":{"Cf-Connecting-Ip":["*"]}}]},{"handle":[{"handler":"headers","response":{"set":{"X-Served-By":["Grafana"]}}},{"handler":"authentication","providers":{"authorizer":{"gatekeeper_name":"admins_policy","route_matcher":"*"}}}]},{"handle":[{"handler":"reverse_proxy","upstreams":[{"dial":"grafana:3000"}]}],"match":[{"header":{"Connection":["*Upgrade*"],"Upgrade":["websocket"]}}]},{"handle":[{"handler":"reverse_proxy","headers":{"request":{"set":{"Host":["{http.vars.REAL_HOST}"],"X-Real-Ip":["{http.vars.REAL_IP}"]}}},"upstreams":[{"dial":"grafana:3000"}]}]}]}],"terminal":true},{"handle":[{"handler":"subroute","routes":[{"handle":[{"handler":"subroute","routes":[{"handle":[{"handler":"subroute","routes":[{"handle":[{"handler":"authenticator","portal_name":"admin_portal","route_matcher":"*"}]}]}],"match":[{"path":["*"]}]}]}]}]}],"terminal":true},{"handle":[{"handler":"subroute","routes":[{"handle":[{"REAL_HOST":"{http.request.host}","handler":"vars"},{"REAL_IP":"{http.request.remote.host}","handler":"vars"}]},{"handle":[{"REAL_HOST":"{http.request.header.X-Forwarded-Host}","handler":"vars"}],"match":[{"header":{"X-Forwarded-Host":["*"]}}]},{"handle":[{"REAL_IP":"{http.request.header.Cf-Connecting-Ip}","handler":"vars"}],"match":[{"header":{"Cf-Connecting-Ip":["*"]}}]},{"handle":[{"handler":"headers","response":{"set":{"X-Served-By":["Loki"]}}},{"handler":"authentication","providers":{"authorizer":{"gatekeeper_name":"admins_policy","route_matcher":"*"}}}]},{"handle":[{"handler":"reverse_proxy","upstreams":[{"dial":"loki:3100"}]}],"match":[{"header":{"Connection":["*Upgrade*"],"Upgrade":["websocket"]}}]},{"handle":[{"handler":"reverse_proxy","headers":{"request":{"set":{"Host":["{http.vars.REAL_HOST}"],"X-Real-Ip":["{http.vars.REAL_IP}"]}}},"upstreams":[{"dial":"loki:3100"}]}]}]}],"terminal":true},{"handle":[{"handler":"subroute","routes":[{"handle":[{"REAL_HOST":"{http.request.host}","handler":"vars"},{"REAL_IP":"{http.request.remote.host}","handler":"vars"}]},{"handle":[{"REAL_HOST":"{http.request.header.X-Forwarded-Host}","handler":"vars"}],"match":[{"header":{"X-Forwarded-Host":["*"]}}]},{"handle":[{"REAL_IP":"{http.request.header.Cf-Connecting-Ip}","handler":"vars"}],"match":[{"header":{"Cf-Connecting-Ip":["*"]}}]},{"handle":[{"handler":"headers","response":{"set":{"X-Served-By":["promtail"]}}},{"handler":"authentication","providers":{"authorizer":{"gatekeeper_name":"admins_policy","route_matcher":"*"}}}]},{"handle":[{"handler":"reverse_proxy","upstreams":[{"dial":"promtail:9080"}]}],"match":[{"header":{"Connection":["*Upgrade*"],"Upgrade":["websocket"]}}]},{"handle":[{"handler":"reverse_proxy","headers":{"request":{"set":{"Host":["{http.vars.REAL_HOST}"],"X-Real-Ip":["{http.vars.REAL_IP}"]}}},"upstreams":[{"dial":"promtail:9080"}]}]}]}],"terminal":true},{"handle":[{"handler":"subroute","routes":[{"handle":[{"REAL_HOST":"{http.request.host}","handler":"vars"},{"REAL_IP":"{http.request.remote.host}","handler":"vars"}]},{"handle":[{"REAL_HOST":"{http.request.header.X-Forwarded-Host}","handler":"vars"}],"match":[{"header":{"X-Forwarded-Host":["*"]}}]},{"handle":[{"REAL_IP":"{http.request.header.Cf-Connecting-Ip}","handler":"vars"}],"match":[{"header":{"Cf-Connecting-Ip":["*"]}}]},{"handle":[{"handler":"headers","response":{"set":{"X-Served-By":["Alertmanager"]}}},{"handler":"authentication","providers":{"authorizer":{"gatekeeper_name":"admins_policy","route_matcher":"*"}}}]},{"handle":[{"handler":"reverse_proxy","upstreams":[{"dial":"alertmanager:9093"}]}],"match":[{"header":{"Connection":["*Upgrade*"],"Upgrade":["websocket"]}}]},{"handle":[{"handler":"reverse_proxy","headers":{"request":{"set":{"Host":["{http.vars.REAL_HOST}"],"X-Real-Ip":["{http.vars.REAL_IP}"]}}},"upstreams":[{"dial":"alertmanager:9093"}]}]}]}],"terminal":true},{"handle":[{"handler":"subroute","routes":[{"handle":[{"REAL_HOST":"{http.request.host}","handler":"vars"},{"REAL_IP":"{http.request.remote.host}","handler":"vars"}]},{"handle":[{"REAL_HOST":"{http.request.header.X-Forwarded-Host}","handler":"vars"}],"match":[{"header":{"X-Forwarded-Host":["*"]}}]},{"handle":[{"REAL_IP":"{http.request.header.Cf-Connecting-Ip}","handler":"vars"}],"match":[{"header":{"Cf-Connecting-Ip":["*"]}}]},{"handle":[{"handler":"headers","response":{"set":{"X-Served-By":["Prometheus"]}}},{"handler":"authentication","providers":{"authorizer":{"gatekeeper_name":"admins_policy","route_matcher":"*"}}}]},{"handle":[{"handler":"reverse_proxy","upstreams":[{"dial":"prometheus:9090"}]}],"match":[{"header":{"Connection":["*Upgrade*"],"Upgrade":["websocket"]}}]},{"handle":[{"handler":"reverse_proxy","headers":{"request":{"set":{"Host":["{http.vars.REAL_HOST}"],"X-Real-Ip":["{http.vars.REAL_IP}"]}}},"upstreams":[{"dial":"prometheus:9090"}]}]}]}],"terminal":true},{"handle":[{"handler":"subroute","routes":[{"handle":[{"REAL_HOST":"{http.request.host}","handler":"vars"},{"REAL_IP":"{http.request.remote.host}","handler":"vars"}]},{"handle":[{"REAL_HOST":"{http.request.header.X-Forwarded-Host}","handler":"vars"}],"match":[{"header":{"X-Forwarded-Host":["*"]}}]},{"handle":[{"REAL_IP":"{http.request.header.Cf-Connecting-Ip}","handler":"vars"}],"match":[{"header":{"Cf-Connecting-Ip":["*"]}}]},{"handle":[{"encodings":{"gzip":{}},"handler":"encode","prefer":["gzip"]}]},{"group":"group9","handle":[{"handler":"subroute","routes":[{"handle":[{"handler":"vars","root":"/usr/share/caddy"},{"handler":"headers","response":{"set":{"X-Served-By":["Caddy"]}}},{"handler":"file_server","hide":["/etc/caddy/sites-enabled/www.caddyfile","/etc/caddy/snippets/snippets_main.caddyfile"]}]}]},{"handler":"subroute","routes":[{"handle":[{"body":"Not found","handler":"static_response","status_code":404}]}]}]}]}],"terminal":true}],"tls_connection_policies":[{}],"automatic_https":{"prefer_wildcard":true},"trusted_proxies":{"ranges":["192.168.0.0/16","172.16.0.0/12","10.0.0.0/8","127.0.0.1/8","fd00::/8","::1"],"source":"static"},"client_ip_headers":["Cf-Connecting-Ip","X-Real-IP"],"logs":{"logger_names":{"admin.wnaji.dev":["log0"],"wnaji.dev":[""],"wnaji.local":[""]},"skip_hosts":["am.wnaji.dev","grafana.wnaji.dev","loki.wnaji.dev","pm.wnaji.dev","pmt.wnaji.dev"]}},"srv1":{"listen":[":80"],"routes":[{"handle":[{"handler":"subroute","routes":[{"handle":[{"abort":true,"handler":"static_response"}],"match":[{"not":[{"client_ip":{"ranges":["192.168.0.0/16","172.16.0.0/12","10.0.0.0/8","127.0.0.1/8","fd00::/8","::1"]}}]}]},{"handle":[{"handler":"metrics"}]}]}],"terminal":true},{},{}],"automatic_https":{"disable":true,"prefer_wildcard":true},"trusted_proxies":{"ranges":["192.168.0.0/16","172.16.0.0/12","10.0.0.0/8","127.0.0.1/8","fd00::/8","::1"],"source":"static"},"client_ip_headers":["Cf-Connecting-Ip","X-Real-IP"]}},"metrics":{}}}
{"level":"warn","ts":1735515098.6594896,"logger":"pki.ca.local","msg":"installing root certificate (you might be prompted for password)","path":"storage:pki/authorities/local/root.crt"}
{"level":"info","ts":1735515098.6597884,"msg":"warning: \"certutil\" is not available, install \"certutil\" with \"apt install libnss3-tools\" or \"yum install nss-tools\" and try again"}
{"level":"info","ts":1735515098.6598015,"msg":"define JAVA_HOME environment variable to use the Java trust"}
{"level":"info","ts":1735515098.6889987,"msg":"certificate installed properly in linux trusts"}
{"level":"debug","ts":1735515098.6891181,"logger":"security","msg":"started app instance","app":"security"}
{"level":"info","ts":1735515098.6891527,"logger":"http","msg":"enabling HTTP/3 listener","addr":":443"}
{"level":"info","ts":1735515098.6901045,"msg":"failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 7168 kiB, got: 416 kiB). See https://github.com/quic-go/quic-go/wiki/UDP-Buffer-Sizes for details."}
{"level":"debug","ts":1735515098.690417,"logger":"http","msg":"starting server loop","address":"[::]:443","tls":true,"http3":true}
{"level":"info","ts":1735515098.6904392,"logger":"http.log","msg":"server running","name":"srv0","protocols":["h1","h2","h3"]}
{"level":"warn","ts":1735515098.6904752,"logger":"http","msg":"HTTP/3 skipped because it requires TLS","network":"tcp","addr":":80"}
{"level":"debug","ts":1735515098.6905017,"logger":"http","msg":"starting server loop","address":"[::]:80","tls":false,"http3":false}
{"level":"warn","ts":1735515098.690506,"logger":"http","msg":"HTTP/2 skipped because it requires TLS","network":"tcp","addr":":80"}
{"level":"info","ts":1735515098.6905122,"logger":"http.log","msg":"server running","name":"srv1","protocols":["h1","h2","h3"]}
{"level":"info","ts":1735515098.690524,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["pm.wnaji.dev","wnaji.dev","wnaji.local","grafana.wnaji.dev","admin.wnaji.dev","loki.wnaji.dev","pmt.wnaji.dev","am.wnaji.dev"]}
{"level":"info","ts":1735515098.691035,"msg":"autosaved config (load with --resume flag)","file":"/config/caddy/autosave.json"}
{"level":"info","ts":1735515098.6910675,"msg":"serving initial configuration"}
{"level":"info","ts":1735515098.694189,"logger":"tls","msg":"cleaning storage unit","storage":"FileStorage:/data/caddy"}
{"level":"info","ts":1735515098.6964662,"logger":"tls","msg":"finished cleaning storage units"}
{"level":"info","ts":1735515098.7086887,"logger":"tls.obtain","msg":"acquiring lock","identifier":"admin.wnaji.dev"}
{"level":"info","ts":1735515098.7096665,"logger":"tls.obtain","msg":"lock acquired","identifier":"admin.wnaji.dev"}
{"level":"info","ts":1735515098.712084,"logger":"tls.obtain","msg":"obtaining certificate","identifier":"admin.wnaji.dev"}
{"level":"debug","ts":1735515098.7124054,"logger":"events","msg":"event","name":"cert_obtaining","id":"7c76954c-0a74-4adb-8e07-e71e0d8d36e2","origin":"tls","data":{"identifier":"admin.wnaji.dev"}}
{"level":"debug","ts":1735515098.713924,"logger":"tls","msg":"created CSR","identifiers":["admin.wnaji.dev"],"san_dns_names":["admin.wnaji.dev"],"san_emails":[],"common_name":"","extra_extensions":0}
{"level":"debug","ts":1735515098.7142751,"logger":"tls.obtain","msg":"trying issuer 1/1","issuer":"local"}
{"level":"info","ts":1735515098.7145271,"logger":"tls.obtain","msg":"acquiring lock","identifier":"wnaji.dev"}
{"level":"info","ts":1735515098.7152116,"logger":"tls.obtain","msg":"lock acquired","identifier":"wnaji.dev"}
{"level":"info","ts":1735515098.7152731,"logger":"tls.obtain","msg":"obtaining certificate","identifier":"wnaji.dev"}
{"level":"debug","ts":1735515098.7152932,"logger":"events","msg":"event","name":"cert_obtaining","id":"9ae843bf-feca-45c4-a157-df28ebfd64a9","origin":"tls","data":{"identifier":"wnaji.dev"}}
{"level":"debug","ts":1735515098.7153718,"logger":"tls","msg":"created CSR","identifiers":["wnaji.dev"],"san_dns_names":["wnaji.dev"],"san_emails":[],"common_name":"","extra_extensions":0}
{"level":"debug","ts":1735515098.7154784,"logger":"pki.ca.local","msg":"using intermediate signer","serial":"682604251874200514516414899705486030998329637113","not_before":"2024-12-29 02:34:13 +0000 UTC","not_after":"2026-12-29 02:34:13 +0000 UTC"}
{"level":"debug","ts":1735515098.7155623,"logger":"tls.obtain","msg":"trying issuer 1/1","issuer":"local"}
{"level":"debug","ts":1735515098.7221806,"logger":"pki.ca.local","msg":"using intermediate signer","serial":"682604251874200514516414899705486030998329637113","not_before":"2024-12-29 02:34:13 +0000 UTC","not_after":"2026-12-29 02:34:13 +0000 UTC"}
{"level":"info","ts":1735515098.7434256,"logger":"tls.obtain","msg":"certificate obtained successfully","identifier":"admin.wnaji.dev","issuer":"local"}
{"level":"debug","ts":1735515098.7436337,"logger":"events","msg":"event","name":"cert_obtained","id":"f90d4b74-9175-4c31-bd66-1115527a8dc2","origin":"tls","data":{"certificate_path":"certificates/local/admin.wnaji.dev/admin.wnaji.dev.crt","csr_pem":"###","identifier":"admin.wnaji.dev","issuer":"local","metadata_path":"certificates/local/admin.wnaji.dev/admin.wnaji.dev.json","private_key_path":"certificates/local/admin.wnaji.dev/admin.wnaji.dev.key","renewal":false,"storage_path":"certificates/local/admin.wnaji.dev"}}
{"level":"debug","ts":1735515098.7437687,"logger":"events","msg":"invoking subscribed handler","name":"cert_obtained","id":"f90d4b74-9175-4c31-bd66-1115527a8dc2","origin":"tls","data":{"certificate_path":"certificates/local/admin.wnaji.dev/admin.wnaji.dev.crt","csr_pem":"###","identifier":"admin.wnaji.dev","issuer":"local","metadata_path":"certificates/local/admin.wnaji.dev/admin.wnaji.dev.json","private_key_path":"certificates/local/admin.wnaji.dev/admin.wnaji.dev.key","renewal":false,"storage_path":"certificates/local/admin.wnaji.dev"},"subscribed_to":"cert_obtained","handler":{"command":"sh","args":["-c","/scripts/cert_obtained.sh {event.data.certificate_path}"],"timeout":30000000000}}
{"level":"info","ts":1735515098.7439406,"logger":"tls.obtain","msg":"releasing lock","identifier":"admin.wnaji.dev"}
{"level":"warn","ts":1735515098.7501154,"logger":"tls","msg":"stapling OCSP","error":"no OCSP stapling for [admin.wnaji.dev]: no OCSP server specified in certificate","identifiers":["admin.wnaji.dev"]}
{"level":"debug","ts":1735515098.7537384,"logger":"tls.cache","msg":"added certificate to cache","subjects":["admin.wnaji.dev"],"expiration":1735558299,"managed":true,"issuer_key":"local","hash":"1d647578f39d34e9cc882d90885d9ea22fd75a40746a1706c2d5c834dbf7e263","cache_size":4,"cache_capacity":10000}
{"level":"debug","ts":1735515098.753839,"logger":"events","msg":"event","name":"cached_managed_cert","id":"ef088575-51e7-4df8-9069-3734c3afc81b","origin":"tls","data":{"sans":["admin.wnaji.dev"]}}
{"level":"info","ts":1735515098.7492619,"logger":"tls.obtain","msg":"certificate obtained successfully","identifier":"wnaji.dev","issuer":"local"}
{"level":"debug","ts":1735515098.755704,"logger":"events","msg":"event","name":"cert_obtained","id":"ee815a16-1ec1-40b1-bce9-4ab3f9bfd451","origin":"tls","data":{"certificate_path":"certificates/local/wnaji.dev/wnaji.dev.crt","csr_pem":"###","identifier":"wnaji.dev","issuer":"local","metadata_path":"certificates/local/wnaji.dev/wnaji.dev.json","private_key_path":"certificates/local/wnaji.dev/wnaji.dev.key","renewal":false,"storage_path":"certificates/local/wnaji.dev"}}
{"level":"debug","ts":1735515098.7559724,"logger":"events","msg":"invoking subscribed handler","name":"cert_obtained","id":"ee815a16-1ec1-40b1-bce9-4ab3f9bfd451","origin":"tls","data":{"certificate_path":"certificates/local/wnaji.dev/wnaji.dev.crt","csr_pem":"###","identifier":"wnaji.dev","issuer":"local","metadata_path":"certificates/local/wnaji.dev/wnaji.dev.json","private_key_path":"certificates/local/wnaji.dev/wnaji.dev.key","renewal":false,"storage_path":"certificates/local/wnaji.dev"},"subscribed_to":"cert_obtained","handler":{"command":"sh","args":["-c","/scripts/cert_obtained.sh {event.data.certificate_path}"],"timeout":30000000000}}
{"level":"info","ts":1735515098.756312,"logger":"tls.obtain","msg":"releasing lock","identifier":"wnaji.dev"}
{"level":"warn","ts":1735515098.7569404,"logger":"tls","msg":"stapling OCSP","error":"no OCSP stapling for [wnaji.dev]: no OCSP server specified in certificate","identifiers":["wnaji.dev"]}
{"level":"debug","ts":1735515098.7571466,"logger":"tls.cache","msg":"added certificate to cache","subjects":["wnaji.dev"],"expiration":1735558299,"managed":true,"issuer_key":"local","hash":"f2f69f2a69dd9848eb2b0a53c53f98c14a4f7d78d8644f043bddacbba60c03c5","cache_size":5,"cache_capacity":10000}
{"level":"debug","ts":1735515098.7573564,"logger":"events","msg":"event","name":"cached_managed_cert","id":"068b68b0-63a5-4774-9407-4dd9f68ea4e7","origin":"tls","data":{"sans":["wnaji.dev"]}}
// Restart invoked by script after 5 seconds sleep
{"level":"info","ts":1735515103.862306,"msg":"using config from file","file":"/etc/caddy/Caddyfile"}
{"level":"info","ts":1735515103.8697217,"msg":"adapted config to JSON","adapter":"caddyfile"}
{"level":"warn","ts":1735515103.869749,"msg":"Caddyfile input is not formatted; run 'caddy fmt --overwrite' to fix inconsistencies","adapter":"caddyfile","file":"/etc/caddy/Caddyfile","line":1}
{"level":"info","ts":1735515103.8715878,"logger":"admin.api","msg":"received request","method":"POST","host":"localhost:2019","uri":"/load","remote_ip":"127.0.0.1","remote_port":"56564","headers":{"Accept-Encoding":["gzip"],"Cache-Control":["must-revalidate"],"Content-Length":["10752"],"Content-Type":["application/json"],"Origin":["http://localhost:2019"],"User-Agent":["Go-http-client/1.1"]}}
{"level":"info","ts":1735515103.8743622,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//[::1]:2019","//127.0.0.1:2019","//localhost:2019"]}
{"level":"info","ts":1735515103.8755226,"logger":"http.auto_https","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
{"level":"info","ts":1735515103.8756557,"logger":"http.auto_https","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
{"level":"warn","ts":1735515103.8756828,"logger":"http.auto_https","msg":"server is listening only on the HTTP port, so no automatic HTTPS will be applied to this server","server_name":"srv1","http_port":80}
{"level":"debug","ts":1735515103.8757353,"logger":"http.auto_https","msg":"adjusted config","tls":{"automation":{"policies":[{"subjects":["wnaji.local"]},{}]}},"http":{"servers":{"srv0":{"listen":[":443"],"routes":[{"handle":[{"handler":"subroute","routes":[{"handle":[{"REAL_HOST":"{http.request.host}","handler":"vars"},{"REAL_IP":"{http.request.remote.host}","handler":"vars"}]},{"handle":[{"REAL_HOST":"{http.request.header.X-Forwarded-Host}","handler":"vars"}],"match":[{"header":{"X-Forwarded-Host":["*"]}}]},{"handle":[{"REAL_IP":"{http.request.header.Cf-Connecting-Ip}","handler":"vars"}],"match":[{"header":{"Cf-Connecting-Ip":["*"]}}]},{"handle":[{"handler":"headers","response":{"set":{"X-Served-By":["Grafana"]}}},{"handler":"authentication","providers":{"authorizer":{"gatekeeper_name":"admins_policy","route_matcher":"*"}}}]},{"handle":[{"handler":"reverse_proxy","upstreams":[{"dial":"grafana:3000"}]}],"match":[{"header":{"Connection":["*Upgrade*"],"Upgrade":["websocket"]}}]},{"handle":[{"handler":"reverse_proxy","headers":{"request":{"set":{"Host":["{http.vars.REAL_HOST}"],"X-Real-Ip":["{http.vars.REAL_IP}"]}}},"upstreams":[{"dial":"grafana:3000"}]}]}]}],"terminal":true},{"handle":[{"handler":"subroute","routes":[{"handle":[{"handler":"subroute","routes":[{"handle":[{"handler":"subroute","routes":[{"handle":[{"handler":"authenticator","portal_name":"admin_portal","route_matcher":"*"}]}]}],"match":[{"path":["*"]}]}]}]}]}],"terminal":true},{"handle":[{"handler":"subroute","routes":[{"handle":[{"REAL_HOST":"{http.request.host}","handler":"vars"},{"REAL_IP":"{http.request.remote.host}","handler":"vars"}]},{"handle":[{"REAL_HOST":"{http.request.header.X-Forwarded-Host}","handler":"vars"}],"match":[{"header":{"X-Forwarded-Host":["*"]}}]},{"handle":[{"REAL_IP":"{http.request.header.Cf-Connecting-Ip}","handler":"vars"}],"match":[{"header":{"Cf-Connecting-Ip":["*"]}}]},{"handle":[{"handler":"headers","response":{"set":{"X-Served-By":["Loki"]}}},{"handler":"authentication","providers":{"authorizer":{"gatekeeper_name":"admins_policy","route_matcher":"*"}}}]},{"handle":[{"handler":"reverse_proxy","upstreams":[{"dial":"loki:3100"}]}],"match":[{"header":{"Connection":["*Upgrade*"],"Upgrade":["websocket"]}}]},{"handle":[{"handler":"reverse_proxy","headers":{"request":{"set":{"Host":["{http.vars.REAL_HOST}"],"X-Real-Ip":["{http.vars.REAL_IP}"]}}},"upstreams":[{"dial":"loki:3100"}]}]}]}],"terminal":true},{"handle":[{"handler":"subroute","routes":[{"handle":[{"REAL_HOST":"{http.request.host}","handler":"vars"},{"REAL_IP":"{http.request.remote.host}","handler":"vars"}]},{"handle":[{"REAL_HOST":"{http.request.header.X-Forwarded-Host}","handler":"vars"}],"match":[{"header":{"X-Forwarded-Host":["*"]}}]},{"handle":[{"REAL_IP":"{http.request.header.Cf-Connecting-Ip}","handler":"vars"}],"match":[{"header":{"Cf-Connecting-Ip":["*"]}}]},{"handle":[{"handler":"headers","response":{"set":{"X-Served-By":["promtail"]}}},{"handler":"authentication","providers":{"authorizer":{"gatekeeper_name":"admins_policy","route_matcher":"*"}}}]},{"handle":[{"handler":"reverse_proxy","upstreams":[{"dial":"promtail:9080"}]}],"match":[{"header":{"Connection":["*Upgrade*"],"Upgrade":["websocket"]}}]},{"handle":[{"handler":"reverse_proxy","headers":{"request":{"set":{"Host":["{http.vars.REAL_HOST}"],"X-Real-Ip":["{http.vars.REAL_IP}"]}}},"upstreams":[{"dial":"promtail:9080"}]}]}]}],"terminal":true},{"handle":[{"handler":"subroute","routes":[{"handle":[{"REAL_HOST":"{http.request.host}","handler":"vars"},{"REAL_IP":"{http.request.remote.host}","handler":"vars"}]},{"handle":[{"REAL_HOST":"{http.request.header.X-Forwarded-Host}","handler":"vars"}],"match":[{"header":{"X-Forwarded-Host":["*"]}}]},{"handle":[{"REAL_IP":"{http.request.header.Cf-Connecting-Ip}","handler":"vars"}],"match":[{"header":{"Cf-Connecting-Ip":["*"]}}]},{"handle":[{"handler":"headers","response":{"set":{"X-Served-By":["Alertmanager"]}}},{"handler":"authentication","providers":{"authorizer":{"gatekeeper_name":"admins_policy","route_matcher":"*"}}}]},{"handle":[{"handler":"reverse_proxy","upstreams":[{"dial":"alertmanager:9093"}]}],"match":[{"header":{"Connection":["*Upgrade*"],"Upgrade":["websocket"]}}]},{"handle":[{"handler":"reverse_proxy","headers":{"request":{"set":{"Host":["{http.vars.REAL_HOST}"],"X-Real-Ip":["{http.vars.REAL_IP}"]}}},"upstreams":[{"dial":"alertmanager:9093"}]}]}]}],"terminal":true},{"handle":[{"handler":"subroute","routes":[{"handle":[{"REAL_HOST":"{http.request.host}","handler":"vars"},{"REAL_IP":"{http.request.remote.host}","handler":"vars"}]},{"handle":[{"REAL_HOST":"{http.request.header.X-Forwarded-Host}","handler":"vars"}],"match":[{"header":{"X-Forwarded-Host":["*"]}}]},{"handle":[{"REAL_IP":"{http.request.header.Cf-Connecting-Ip}","handler":"vars"}],"match":[{"header":{"Cf-Connecting-Ip":["*"]}}]},{"handle":[{"handler":"headers","response":{"set":{"X-Served-By":["Prometheus"]}}},{"handler":"authentication","providers":{"authorizer":{"gatekeeper_name":"admins_policy","route_matcher":"*"}}}]},{"handle":[{"handler":"reverse_proxy","upstreams":[{"dial":"prometheus:9090"}]}],"match":[{"header":{"Connection":["*Upgrade*"],"Upgrade":["websocket"]}}]},{"handle":[{"handler":"reverse_proxy","headers":{"request":{"set":{"Host":["{http.vars.REAL_HOST}"],"X-Real-Ip":["{http.vars.REAL_IP}"]}}},"upstreams":[{"dial":"prometheus:9090"}]}]}]}],"terminal":true},{"handle":[{"handler":"subroute","routes":[{"handle":[{"REAL_HOST":"{http.request.host}","handler":"vars"},{"REAL_IP":"{http.request.remote.host}","handler":"vars"}]},{"handle":[{"REAL_HOST":"{http.request.header.X-Forwarded-Host}","handler":"vars"}],"match":[{"header":{"X-Forwarded-Host":["*"]}}]},{"handle":[{"REAL_IP":"{http.request.header.Cf-Connecting-Ip}","handler":"vars"}],"match":[{"header":{"Cf-Connecting-Ip":["*"]}}]},{"handle":[{"encodings":{"gzip":{}},"handler":"encode","prefer":["gzip"]}]},{"group":"group9","handle":[{"handler":"subroute","routes":[{"handle":[{"handler":"vars","root":"/usr/share/caddy"},{"handler":"headers","response":{"set":{"X-Served-By":["Caddy"]}}},{"handler":"file_server","hide":["/etc/caddy/sites-enabled/www.caddyfile","/etc/caddy/snippets/snippets_main.caddyfile"]}]}]},{"handler":"subroute","routes":[{"handle":[{"body":"Not found","handler":"static_response","status_code":404}]}]}]}]}],"terminal":true}],"tls_connection_policies":[{}],"automatic_https":{"prefer_wildcard":true},"trusted_proxies":{"ranges":["192.168.0.0/16","172.16.0.0/12","10.0.0.0/8","127.0.0.1/8","fd00::/8","::1"],"source":"static"},"client_ip_headers":["Cf-Connecting-Ip","X-Real-IP"],"logs":{"logger_names":{"admin.wnaji.dev":["log0"],"wnaji.dev":[""],"wnaji.local":[""]},"skip_hosts":["am.wnaji.dev","grafana.wnaji.dev","loki.wnaji.dev","pm.wnaji.dev","pmt.wnaji.dev"]}},"srv1":{"listen":[":80"],"routes":[{"handle":[{"handler":"subroute","routes":[{"handle":[{"abort":true,"handler":"static_response"}],"match":[{"not":[{"client_ip":{"ranges":["192.168.0.0/16","172.16.0.0/12","10.0.0.0/8","127.0.0.1/8","fd00::/8","::1"]}}]}]},{"handle":[{"handler":"metrics"}]}]}],"terminal":true},{},{}],"automatic_https":{"disable":true,"prefer_wildcard":true},"trusted_proxies":{"ranges":["192.168.0.0/16","172.16.0.0/12","10.0.0.0/8","127.0.0.1/8","fd00::/8","::1"],"source":"static"},"client_ip_headers":["Cf-Connecting-Ip","X-Real-IP"]}},"metrics":{}}}
{"level":"warn","ts":1735515103.9559255,"logger":"pki.ca.local","msg":"installing root certificate (you might be prompted for password)","path":"storage:pki/authorities/local/root.crt"}
{"level":"info","ts":1735515103.9563527,"msg":"warning: \"certutil\" is not available, install \"certutil\" with \"apt install libnss3-tools\" or \"yum install nss-tools\" and try again"}
{"level":"info","ts":1735515103.9563766,"msg":"define JAVA_HOME environment variable to use the Java trust"}
{"level":"info","ts":1735515104.0122855,"msg":"certificate installed properly in linux trusts"}
{"level":"debug","ts":1735515104.0124445,"logger":"security","msg":"started app instance","app":"security"}
{"level":"info","ts":1735515104.0124853,"logger":"http","msg":"enabling HTTP/3 listener","addr":":443"}
{"level":"debug","ts":1735515104.013814,"logger":"http","msg":"starting server loop","address":"[::]:443","tls":true,"http3":true}
{"level":"info","ts":1735515104.0138438,"logger":"http.log","msg":"server running","name":"srv0","protocols":["h1","h2","h3"]}
{"level":"warn","ts":1735515104.0139952,"logger":"http","msg":"HTTP/3 skipped because it requires TLS","network":"tcp","addr":":80"}
{"level":"debug","ts":1735515104.0140834,"logger":"http","msg":"starting server loop","address":"[::]:80","tls":false,"http3":false}
{"level":"warn","ts":1735515104.01412,"logger":"http","msg":"HTTP/2 skipped because it requires TLS","network":"tcp","addr":":80"}
{"level":"info","ts":1735515104.0141284,"logger":"http.log","msg":"server running","name":"srv1","protocols":["h1","h2","h3"]}
{"level":"info","ts":1735515104.0141377,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["pm.wnaji.dev","wnaji.dev","wnaji.local","grafana.wnaji.dev","admin.wnaji.dev","loki.wnaji.dev","pmt.wnaji.dev","am.wnaji.dev"]}
{"level":"debug","ts":1735515104.0142326,"logger":"security","msg":"stopped app instance","app":"security"}
{"level":"info","ts":1735515104.0142446,"logger":"http","msg":"servers shutting down with eternal grace period"}
{"level":"info","ts":1735515104.0146883,"msg":"autosaved config (load with --resume flag)","file":"/config/caddy/autosave.json"}
{"level":"info","ts":1735515104.0149457,"logger":"admin.api","msg":"load complete"}
{"level":"error","ts":1735515104.014911,"logger":"events.handlers.exec","msg":"background command failed","error":"signal: killed"}
{"level":"info","ts":1735515104.0193648,"logger":"admin","msg":"stopped previous server","address":"localhost:2019"}
{"level":"debug","ts":1735515112.0454206,"logger":"events","msg":"event","name":"tls_get_certificate","id":"c8803fdd-e9b5-4585-8689-12c43d0ec380","origin":"tls","data":{"client_hello":{"CipherSuites":[4866,4867,4865,49196,49200,159,52393,52392,52394,49195,49199,158,49188,49192,107,49187,49191,103,49162,49172,57,49161,49171,51,157,156,61,60,53,47],"ServerName":"wnaji.local","SupportedCurves":[29,23,30,25,24,256,257,258,259,260],"SupportedPoints":"AAEC","SignatureSchemes":[1027,1283,1539,2055,2056,2074,2075,2076,2057,2058,2059,2052,2053,2054,1025,1281,1537,771,769,770,1026,1282,1538],"SupportedProtos":null,"SupportedVersions":[772,771],"RemoteAddr":{"IP":"185.106.*.*","Port":62822,"Zone":""},"LocalAddr":{"IP":"172.18.0.2","Port":443,"Zone":""}}}}
{"level":"debug","ts":1735515112.045781,"logger":"tls.handshake","msg":"choosing certificate","identifier":"wnaji.local","num_choices":1}
{"level":"debug","ts":1735515112.04579,"logger":"tls.handshake","msg":"default certificate selection results","identifier":"wnaji.local","subjects":["wnaji.local"],"managed":true,"issuer_key":"local","hash":"0b9d078f048b54c9611b362caa90408412003a3e08e5fac128ea7caa586db879"}
{"level":"debug","ts":1735515112.0458148,"logger":"tls.handshake","msg":"matched certificate in cache","remote_ip":"185.106.*.*","remote_port":"62822","subjects":["wnaji.local"],"managed":true,"expiration":1735558299,"hash":"0b9d078f048b54c9611b362caa90408412003a3e08e5fac128ea7caa586db879"}
{"level":"info","ts":1735515122.5332406,"msg":"shutting down apps, then terminating","signal":"SIGTERM"}
{"level":"warn","ts":1735515122.5332956,"msg":"exiting; byeee!! 👋","signal":"SIGTERM"}
{"level":"debug","ts":1735515122.5333157,"logger":"security","msg":"stopped app instance","app":"security"}
{"level":"info","ts":1735515122.5333645,"logger":"http","msg":"servers shutting down with eternal grace period"}
{"level":"info","ts":1735515122.5339742,"logger":"admin","msg":"stopped previous server","address":"localhost:2019"}
{"level":"info","ts":1735515122.5339904,"msg":"shutdown complete","signal":"SIGTERM","exit_code":0}

Caddy Version
v2.9.0-beta.3 h1:tlqfbJMRNY6vnWwaQrnWrgS+wkDXr9GIFUD/P+HY9vA=

Cert Obtained Script

#!/bin/sh

CERT_PATH="/data/caddy/$1"
FULLCHAIN_PATH="/data/myca/fullchain.crt"
LAST_CHANGED_FP="/tmp/last_changed"

# Ensure required files exist
if [ ! -f "$CERT_PATH" ]; then
  echo "Error: Certificate file $CERT_PATH does not exist."
  exit 1
fi

if [ ! -f "$FULLCHAIN_PATH" ]; then
  echo "Error: Fullchain file $FULLCHAIN_PATH does not exist."
  exit 1
fi

# Backup current cert
cp "$CERT_PATH" "$CERT_PATH.bak" || {
  echo "Error: Failed to back up $CERT_PATH."
  exit 1
}

# Make fullchain
cat "$CERT_PATH.bak" "$FULLCHAIN_PATH" > "$CERT_PATH" || {
  echo "Error: Failed to create fullchain at $CERT_PATH."
  exit 1
}

# Record last changed file
echo "$CERT_PATH" > "$LAST_CHANGED_FP"

# Wait for 5 seconds to debounce reloads
sleep 5

# Check if this is still the last changed file
LAST_CHANGED=$(cat "$LAST_CHANGED_FP")
if [ "$LAST_CHANGED" = "$CERT_PATH" ]; then
  # Reload Caddy
  caddy reload --config "$CADDY_CONFIG" --force || {
    echo "Error: Failed to reload Caddy."
    exit 1
  }
  # Clean up
  rm "$LAST_CHANGED_FP"
fi

[mholt]

That might be because our cert cache aggressively reuses certificates across reloads, to make them more efficient. I marked this as a bug because it might be unexpected behavior, but to be fair, --force is only documented as reloading the Caddy config, not necessarily the certificates in external files.

"Fixing" this "bug" will involve a potentially significant performance hit when --force is used, if lots of certificates are being loaded.

[wisammechano]

@mholt , for some more reference, commit 2772ede that introduced --force for this specific use case upon the issue #4005. I don't know if it only refers to manually generated certificates perhaps?

I even tried manually using POST /load with the Cache-Control: must-revalidate header (which is equivalent to --force), but it didn't work.

[JasonLovesDoggo]

That might be because our cert cache aggressively reuses certificates across reloads, to make them more efficient. I marked this as a bug because it might be unexpected behavior, but to be fair, --force is only documented as reloading the Caddy config, not necessarily the certificates in external files.

"Fixing" this "bug" will involve a potentially significant performance hit when --force is used, if lots of certificates are being loaded.

Could a new --refresh-certs flag be added? that should mitigate the performance issue.