Oh good to know!

Do you know if Caddy can self update or if is there some other easy method? Manually doing it to get the cloudflare plugin is a pain.

No, you have to build Caddy with plugins. We provide xcaddy to make it easy. Sign up for notifications on github for releases, and just write yourself a tiny bash script to build the binary with xcaddy, and restart the service. You could potentially do a thing where you hook into apt to trigger your script after Caddy's deb package version changes, idk. But it's up to you to handle.