I can only speak for myself, but I considered a few options, including "simple k8s" like [Skate](https://skateco.github.io/), and ultimately decided to build on uncloud.

It was as much personal "taste" than anything, and I would describe the choice as similar to preferring JSON over XML.

For whatever reason, kubernetes just irritates me. I find it unpleasant to use. And I don't think I'm unique in that regard.

100%. I’m really not sure why K8S has become the complexity boogeyman. I’ve seen CDK apps or docker compose files that are way more difficult to understand than the equivalent K8S manifests.

That’s awesome if k3s works for you, nothing wrong with this. You’re simply not the target user then.

k3s makes it easy to deploy, not to debug any problems with it. It's still essentially adding few hundred thousand lines of code into your infrastructure, and if it is a small app you need to deploy, also wasting a bit of ram

If you already know k8s, this is probably true. If you don't it's hard to know what bits you need, and need to learn about, to get something simple set up.

Perhaps it feels so easy given your familiarity with it.

I have struggled to get things like this stood up and hit many footguns along the way

Indeed, it seems a knee jerk response without justification. k3s is pretty damn minimal.

Totally valid concern. That was a shortcut to iterate quickly in early development. It’s time to do it properly now. Appreciate the feedback. This is exactly the kind of thing I need to hear before more people try it.

+1 on this

I wanted to try it out but was put off by this[0]. It’s just straight up curl | bash as root from raw.githubusercontent.com.

If this is the install process for a server (and not just for the CLI) I don’t want to think about security in general for the product.

Sorry, I really wanted to like this, but pass.

[0] https://github.com/psviderski/uncloud/blob/ebd4622592bcecedb...

There is a `--no-install` flag on both `uc machine init` and `uc machine add` that skips that `curl | bash` install step.

You need to prepare the machine some other way first then, but it's just installing docker and the uncloud service.

I use the `--no-install` option with my own cluster, as I have my own pre-provisioning process that includes some additional setup beyond the docker/uncloud elements.

Curious, what would be an ideal (secure) approach for you to install this (or similar) tool?

There isn't a cli function for adding a connection (independently of adding a new machine/node) yet, but they are in a simple config file (`~/.config/uncloud/config.yaml`) that you can copy or easily create manually for now. It looks like this:

    current_context: default
    contexts:
      default:
        connections:
          - ssh: admin@192.168.0.10
            ssh_key_file: ~/.ssh/uncloud
          - ssh: admin@192.168.0.11
            ssh_key_file: ~/.ssh/uncloud
          - ssh: administrator@93.x.x.x
            ssh_key_file: ~/.ssh/uncloud
          - ssh: sysadmin@65.x.x.x
            ssh_key_file: ~/.ssh/uncloud

I took some inspiration from Kamal, e.g. the imperative model but kamal is more a deployment tool.

In addition to deployments, uncloud handles clustering - connects machines and containers together. Service containers can discover other services via internal DNS and communicate directly over the secure overlay network without opening any ports on the hosts.

As far as I know kamal doesn’t provide an easy way for services to communicate across machines.

Services can also be scaled to multiple replicas across machines.

I think there are two different cases here. Not sure which one you’re talking about.

1. External requests, e.g. from the internet via the reverse proxy (Caddy) running in the cluster.

The rollout works on the container, not the server level. Each container registers itself in Caddy so it knows which containers to forward and distribute requests to.

When doing a rollout, a new version of container is started first, registers in caddy, then the old one is removed. This is repeated for each service container. This way, at any time there are running containers that serve requests.

It doesn’t say any server that requests shouldn’t go there. It just updates upstreams in the caddy config to send requests to the containers that are up and healthy.

2. Service to service requests within the cluster. In this case, a service DNS name is resolved to a list of IP addresses (running containers). And the client decides which one to send a request to or whether to distribute requests among them.

When the service is updated, the client needs to resolve the name again to get the up-to-date list of IPs. Many http clients handle this automatically so using http://service-name as an endpoint typically just works. But zero downtime should still be handled by the client in this case.

Thank you! That's actually the trade off.

There is no automatic rescheduling in uncloud by design. At least for now. We will see how far we can get without it.

If you want your service to tolerate a host going down, you should deploy multiple replicas for that service on multiple machines in advance. 'uc scale' command can be used to run more replicas for an already deployed service.

Longer term, I'm thinking we can have a concept of primary/standby replicas for services that can only have one running replica, e.g. databases. Something similar to how Fly.io does this: https://fly.io/docs/apps/app-availability/#standby-machines-...

Regarding deploying on the less filled machine first is doable but not supported right now. By default, it picks the first machine randomly and tries to distributes replicas evenly among all available machines. You can also manually specify what target machine(s) each service should run on in your Compose file.

I want to avoid recreating the complexity with placement constraints, (anti-)affinity, etc. that makes K8s hard to reason about. There is a huge class of apps that need more or less static infra, manual placement, and a certain level of redundancy. That's what I'm targeting with Uncloud.

For the public cluster with multiple ingress (caddy) nodes you'd need a load balancer in front of them to properly handle routing and outage of any of them. You'd use the IP of the load balancer on the DNS side.

Note that a DNS A record with multiple IPs doesn't provide failover, only round robin. But you can use the Cloudflare DNS proxy feature as a poor man's LB. Just add 2+ proxied A records (orange cloud) pointing to different machines. If one goes down with a 52x error, Cloudflare automatically fails over to the healthy one.

Not OP, but you could do "simple" dns load balancing between both endpoints.

podman-compose works fine. It’s a very simple format.

This is exactly how it works now. The Compose file is the declarative specification of your services you want to run.

When you run 'uc deploy' command:

- it reads the spec from your compose.yaml

- inspects the current state of the services in the cluster

- computes the diff and deployment plan to reconcile it

- executes the plan after the confirmation

Please see the docs and demo: https://uncloud.run/docs/guides/deployments/deploy-app

The main difference with Docker Swarm is that the reconciliation process is run on your local/CI machine as part of the 'uc deploy' CLI command execution, not on the control plane nodes in the cluster.

And it's not running in the loop automatically. If the command fails, you get an instant feedback with the errors you can address or rerun the command again.

It should be pretty straightforward to wrap the CLI logic in a Terraform or Pulumi provider. The design principals are very similar and it's written in Go.

Do you have an example of what you mean? I'm not entirely clear on your question.

There is an open issue that confirms enabling ipv6 for containers works: https://github.com/psviderski/uncloud/issues/126 But this hasn’t been enabled by default.

What specifically do you mean by ipv6 support?

It's not a k8s replacement. It's for the small dev team with no k8s experience. For people that might not use Docker Swarm because they see it's a pretty dead project. For people who think "everyone uses k8s", so we should, too.

I need to run on-prem, so managed k8s is not an option. Experts tells me I should have 2 FTE to run k8s, which I don't have. k8s has so many components, how should I debug that in case of issues without k8s experience? k8s APIs change continuously, how should I manage that without k8s experience?

It's not a k8s replacement. But I do see a sweet spot for such a solution. We still run Docker Swarm on 5 servers, no hyperscalers, no API changes expected ;-)

Those are all sub-par cloud technologies which perform very badly and do not scale at all.

Some people would rather build their own solutions to do these things with fine-grain control and the ability to handle workloads more complex that a shopping cart website.

If you are a small operation and trying to self host k3s or k8s or any number of out of the box installations that are probably at least as complex as docker compose swarms, for any non trivial production case, presents similar problems in monitoring and availability as ones you’d get with off the shelf cloud provider managed services, except the managed solutions come without the pain in the ass. Except you don’t have a control plane.

I have managed custom server clusters in a self hosted situation. the problems are hard, but if you’re small, why would you reach for such a solution in the first place? you’d be better off paying for a managed service. What situation forces so many people to reach to self hosted kubernetes?

k3s uses sqlite, so not etcd.

Why would you want to do that though?

On cloud, in my experience, you are mostly paying for compute with managed kubernetes instances. The overhead and price is almost never kubernetes itself, but the compute and storage you are provisioning, which, thanks to the control plane, you have complete control over. what am i missing?

I wouldn’t dare try to with a small shop try to self host a production kubernetes solution unless i was under duress. But I just dont see what the control plane has to do with it. It’s the feature that makes kubernetes worth it.

Tinkerbell / MetalKube, ClusterAPI, Rook, Cilium?

A control plane makes controlling machines easier, that's the point of a control plane.

The experience you are describing has overwhelmingly not been my own, nor anyone in my space I know.

I can only speak most recently for EKS, but the cost is spent almost entirely on compute. I’m a one man shop managing 10,000 containers. I basically only spend on the compute itself, which is not all that much, and certainly far, far less than hiring a sys admin. Self hosted anything would be a huge PITA for me and likely end up costing more.

Yes, you can avoid kubernetes and being a “slave” to cloud providers, but I personally believe you’re making infrastructure tradeoffs in a bad way, and likely spending as much in the long run anyway.

maybe my disconnect here is that I mostly deal with full production scale applications, not hobby projects I am hosting on my own network (nothing wrong with that, and I would agree k8s is overkill for something like that).

Eventually though, at scale, I strongly believe you will need or want a control plane of some type for your container fleets, and that typically ends up looking or acting like k8s.

Isn't Nomad pretty much dead now?

It supports docker swarm, but I've never used it like that. As I may need multi node in the future, I was asking the question to see if it would make it 'easy' to orchestrate multiple containers. The simplicity of Dokku is hard to beat, however.

edit: Well, it would appear that the very maintainer of Dokku himself replied to the parent comment. My information is clearly outdated and I'd only look at this comment[0] to get the proper info.

[0] https://news.ycombinator.com/item?id=46144275#46145919

Dokku is multi node. It supports docker-local (single node) and k3s (multi-node) as schedulers, with most features implemented as expected when deploying to k3s.

In another reply, I commended him on doing it if the reasoning was “because he felt like the world needed it” or if it was to scratch an itch. But if he wants to make this a business - that’s a different beast with different challenges that he needs to address.

I can’t find it now, but I could swear I saw somewhere he said he’s working on this full time and living off of savings. If I’m wrong he will correct me I am sure. If that’s the case, I assume he wants to make this a business.

He has to think about those objection. Would he better off by first creating a known Kubernetes environment and then building an easier wrapper around that so someone could manage the K8s cluster in case they don’t want a dependency only on his product? I don’t have those answers

The enterprise equivalent of COBOL today is Java and to a much lesser extent C#. Those were both championed by large corporations - Sun and Microsoft.

The move to VMs at first and then to the cloud were also marketed by existing companies with huge budgets where people who made decisions had the “No one ever got fired for choosing $LargeWellknownCompany that is in the upper right corner of Gartner’s Magic Square”.

I love Docker. I think everyone going to EKS before they need to is dumb. There are dozens of services out there that let you give it a Docker container and just run it.

And I think that spending energy avoiding “cloud lock-in” is dumb. Choose your infrastructure and build. Migrations are going to be a pain at any decent scale anyway and you are optimizing for the wrong thing if you are worried about lock in.

As an individual especially in today’s market, it’s foolish (not referring to you - any developer or adjacent) not to always be thinking of what keeps you the most employable if the rug gets pulled from under you.

As a decision maker who is held accountable for architecture and when things go wrong they look at or when the next person has to come along to maintain it, they are going to look at me like I am crazy if I choose a non industry standard solution just because I was too lazy to choose the industry standard.

Again I don’t mean that you are being “lazy”. That’s how people think.

But if I were hiring someone - and I’m often interviewing people for cloudy/devOps type roles. Why would I hire someone with experience with a Docker orchestration framework I never heard of over someone who knew K8s?

And the final question you should ask yourself is why are you really doing this?

Is it to scratch an itch out of passion and it’s something that you feel the world should have? If so in all sincerity, I wish you luck on your endeavor. You might get lucky like Bun just did. I had effusive praise for them doing something out of passion instead of as VC bait.

Are you doing it for financial gain? If so, you have to come up with a strategy to overcome resistance from people like Ive outlined.