I also self host some services: keycloak, forgejo, home assistant, immich, grafana/prometheus, jellyfin, pureftp (to store my surveillance cameras videos), and a vibecoded financial app, and soon hermes agent on a 2023 laptop.

I chose to run most of them on Kubernetes since I am used to work with it. I use:

- ha-proxy running on the host and istio running inside the cluster for the sni based ingress routing - cert-manager with a self-signed CA I had to import the CA cert on my devices - cloudnative pg for postgresql dbs (keycloak and immich)

For the backup it is mainly some cron jobs that save to the cloud.

It is overkill I admit but kubernetes makes some integrations easier. Even though it was also a way for me to maintain some Kubernetes skills for my work. The tricky part is the network connectivity, for instance home assistant runs inside the cluster on its own network and it makes things more complex like homekit bridge integration (different network than my devices). But with the help of AI the iptables configuration makes wonders.

The kubernetes cluster is k3s running on NixOS micro vms, fully managed declaratively thanks to NixOS.

The services that run outside the cluster (ha-proxy, jellyfin) are all managed declaratively.

At some point I had thought of using proxmox with lxc but I am very happy with NixOS.

▲

attilaolah 8 hours ago | parent | next [-]

I also used to run everything on K8s (hybrid Talos / Alpine+k0s on nodes too small for Talos). It is a fun exercise, but with many small nodes eventually I realised I was burning a quarter of my fleet's CPU on just monitoring and routing services.

Eventually I switched to Systemd only, Clan-managed NixOS instances on bare metal.

The killer feature for me is socket activation. It even works with Jellyfin, wrapped in systemd-socket-proxyd. I have Jellyfin listen on a UDS, and Systemd will activate it when my TV connects. The downside is a few more seconds of waiting when I open Jellyfin on the TV. The upside is, after 20 minutes of idling, it is fully powered down.

This allows packing even more services on smaller machines, provided you don't e.g. play Minecraft and watch Jellyfin at the same time.

▲

gf000 an hour ago | parent | next [-]

How does 20 minutes of idling work? Is that jellyfin shutting itself down?

	▲	attilaolah an hour ago \| parent \| next [-]
		No it doesn't support that, but systemd-socket-proxyd does. So my Systemd service exec line looks like this: ExecStart = /path/to/systemd/lib/systemd/systemd-socket-proxyd --exit-idle-time=20min /path/to/jellyfin.socket; https://github.com/attilaolah/dotfiles/blob/144d1711755d7a9e...
	▲	attilaolah an hour ago \| parent \| prev [-]
		To be clear, it will shut down Jellyfin after 20 minutes of inactivity on the socket. It may as well be that Jellyfin is doing some background work like re-indexing videos, so it is not perfect, but it is the next best thing until Jellyfin supports first-class socket activation.

▲

bengalister 6 hours ago | parent | prev | next [-]

Yes i will probably end up using systemd services when I will hit resource limits. So far everything works just fine on my LG16 32 GB i7 laptop and consumes around 15w.

▲

apexalpha 8 hours ago | parent | prev | next [-]

Just looked up Clan since I'm a big fan of Nix but currently run my cluster with k3s, which indeed has some overhead.

It seems however that Clan is one of those classic projects where the website and the docs assume that everyone already knows what it is and how it works and goes straight to the benefits of doing it this (whatever 'this' is) way.

After reading both the site, docs and guides I am now scouring Reddit and YouTube for an actual explanation of whatever the hell it is and how it works.

	▲	attilaolah an hour ago \| parent [-]
		Yeah Clan does have some rough edges. And the website is indeed like that, especially the bits about disko / partitioning, etc.

▲

dv_dt 5 hours ago | parent | prev [-]

We have come full circle back to inetd, only systemd is a little maze of twisty little passages in comparison

▲

chanux 5 hours ago | parent | prev | next [-]

Do you mind sharing your use case for Keycloak? Is it for the various personal apps?

▲

mirthflat83 6 hours ago | parent | prev | next [-]

How's your vibecoded financial app setup? Are you using Plaid?

▲

pydry 8 hours ago | parent | prev [-]

isn't it a nightmare running home assistant with kubernetes? i had a bad enough time just using docker compose.

	▲	bengalister 6 hours ago \| parent [-]
		So far it is ok. I even have a usb zigbee dongle receiver in usb passthrough and the container in privileged mode. I am moving to the Apple ecosystem and I managed to have homekit bridge connected to Apple home. I had to use an avahi relay on the NixOS host for the discovery protocol to work. The home assistant is not the same network as my devices. The homekit bridge allows to configure the advertised IP. I use tailscale (outside the k8s cluster) for connectivity. My phone companion app is not connected to my home assistant server outside my home but could through tailscale. So definitely there is added complexity, especially on the iptables configuration. Without the AI help, I don’t think I would have managed to do it regardless of the time saving.