Running a number of production services on-premise on a big machine using native zones, a few using LX zones (the built-in Linux compatibilty layer), and a single bhyve zone. Actually, years ago this machine was the very first server we set up when our company was just getting started and for the first few years it ran pretty much everything. Zones were ideal for that, also to allow us to pack more services on less hardware, while having decent separation and everything snapshotting/backupping using ZFS. Nowadays we have a bunch more servers, with varying *nix operating systems (SmartOS, Debian, FreeBSD), as well as macOS and even Windows for some specific CI functions. (:

The global zone works great as a hypervisor if you prefer working over SSH in a real shell, and being able to run a lot of services natively just makes things like memory allocation to VM's and having a birds eye view of performance easier. Being able to CoW cp/mv files between zones because it's actually the same filesystem makes certain operations much easier than with actual VM's. Bhyve works well for the things that need an actual Linux kernel or other OS, at the cost of losing some of the zone benefits mentioned earlier.

Highlighting a few things we today run on SmartOS, grouped by their technology stacks: C (haproxy, nginx, PostgreSQL, MariaDB), PHP (various web apps), Java (Keycloak), Elixir/Phoenix (Plausible, fork of Firezone), Rust (rathole, some internal glue services), Go (Grafana, Consul, Prometheus). Most of those are readily available in the package manager, and a few offer native Solaris binaries which run fine on illumos. Others we do local builds in a utility zone before copying the binary package to the where it actually runs.

On LX zones we also run a number of services without problems, usually because they have Debian packaging available but are not in pkgsrc (for example Consul/Nomad, Fabio, some internal things that was already Linux-specific and we haven't bothered to port yet).

And at home a LX zone also runs Jellyfin just fine. (:

I used it for smaller scale (low 10s of physical servers) back in the day also. But my problems with that started when i needed a lot more, the sysadmin/devops/whatever story doesn't scale.

Yes, ansible exists but it's actually quite hard to run ansible on a few hundred machines -- you need lots of RAM just to run the playbook and your first hundred or so separate deployments, you do need to reach for something like Kubernetes.

As for LX, why emulate linux when it's .... right there? The linux kernel is not a lot of overhead vs having to justify emulating the linux ABI on an OS the industry has largely abandoned.