I have a couple of FreeBSD VMs on Vultr. For quite some time sshd regularly dies on one of them and I am not sure why.

A couple of theories I’ve had is that maybe

a) my VM was compromised and there is a persistent rootkit installed that kills sshd, or

b) file corruption after previous unclean shutdown has left some file needed by sshd corrupted and it leads to this behaviour, or

c) maybe it’s running out of memory sometimes

Each time I want to ssh into the machine I usually have to first connect with the VNC from the vultr dashboard to start sshd up again.

It’s running the latest FreeBSD, as every now and then I log in and do an upgrade on it some time after a new version has been released.

A persistent rootkit may have been installed if it was compromised between when some vulnerability became known and when I later upgraded next time.

If a file was corrupted in an unclean shutdown in the past maybe it’s a file that has not been changed between FreeBSD versions so even though upgrades replace some files maybe it’s the same corrupted file all along.

Ideally I’d just reinstall the machine, but that’s always more of a hassle than it should be so I continue running the VM in this broken state where sshd keeps dying every now and then.

▲ cyberpunk 5 hours ago | parent | next [-]

Well, you can at least run "freebsd-update IDS" on this system to verify the base system, "pkg check -s -a" would check the integrity of files installed via pkg also.

It will only take a few min..

▲ QuantumNomad_ 5 hours ago | parent [-]

Neat! Haven’t tried either of those before.

While I’m at it I also took a quick look now at output of `top` and it’s sitting at 27 MB free RAM lol. So from that, out of memory is very likely the reason I keep having sshd die on me.

▲ ilikecode 3 hours ago | parent | next [-]

Try adding this to your /etc/rc.conf:

    sshd_oomprotect=YES

Then run

   service sshd restart

▲ silisili 5 hours ago | parent | prev [-]

Not familiar with BSD but does it not syslog oom kills like linux?

	▲	QuantumNomad_ 4 hours ago \| parent [-]
		After rebooting the VM now, sshd died even when there was hundreds of megabytes free RAM available. So it seems I spoke too soon when I said it seemed to be for that reason. Previously I haven't seen much detailed reason for why it dies in system messages. But this time it said something very specific: > sshd[2036]: fatal: pack_hostkeys: serialize hostkey private: string is too large Which kind of sounds like one of the sshd hostkey files might be corrupt? And maybe it only triggers after a while becuase it happens when scanners try to connect to it and during ssh negotiation sshd ends up selecting a different hostkey type than the one it uses when I connect to the machine myself? I'm going to regenerate all of the three hostkey files on the server, and after that also disable the two that I can do without anyway.

▲ toyg 2 hours ago | parent | prev [-]

I have run OpenBSD on Vultr for probably a decade at this point, and never seen that behaviour.