Remix.run Logo
gobdovan 14 hours ago

Nice list! I'd say the SQLite with WAL is the biggest money saver mentioned.

One note: you can absolutely use Python or Node just as well as Go. There's Hetzner that offers 4GB RAM, 10TB network (then 1$/TB egress), 2CPUs machines for 5$.

Two disclaimers for VPS:

If you're using a dedicated server instead of a cloud server, just don't forget to backup DB to a Storage box often (3$ /mo for 1TB, use rsync). It's a good practice either way, but cloud instances seem more reliable to hardware faults. Also avoid their object store.

You are responsible for security. I saw good devs skipping basic SSH hardening and get infected by bots in <1hr. My go-to move when I spin up servers is a two-stage Terraform setup: first, I set up SSH with only my IP allowed, set up Tailscale and then shutdown the public SSH IP entrypoint completely.

Take care and have fun!

jon-wood 10 hours ago | parent | next [-]

Personally for backups I’d avoid using a product provided by the same company as the VM I’m backing up. You should be defending against the individual VM suffering corruption of some kind, needing to roll back to a previous version because of an error you made, and finally your VM provider taking a dislike to you (rationally or otherwise) and shutting down your account.

If you’re backing up to a third party losing your account isn’t a disaster, bring up a VM somewhere else, restore from backups, redirect DNS and you’re up and running again. If the backups are on a disk you can’t access anymore then a minor issue has just escalated to an existential threat to your company.

Personally I use Backblaze B2 for my offsite backups because they’re ridiculously cheap, but other options exist and Restic will write to all of them near identically.

joelthelion 9 hours ago | parent | prev | next [-]

> You are responsible for security. I saw good devs skipping basic SSH hardening and get infected by bots in <1hr. My go-to move when I spin up servers is a two-stage Terraform setup: first, I set up SSH with only my IP allowed, set up Tailscale and then shutdown the public SSH IP entrypoint completely.

Note that you don't need all of that to keep your SSH server secure. Just having a good password (ideally on a non-root account) is more than enough.

chillfox 7 hours ago | parent | next [-]

Disable password auth and go with key based, it's easier and more secure.

gobdovan 9 hours ago | parent | prev [-]

I'd call it unnecessary exposure. Under both modern threat models and classic cybernetic models (check out law of requisite variety) removing as much surface attack area as possible is optimal. Especially disabling passwords in SSH is infosec 1o1 these days. No need to worry about brute force attacks, credential stuffing, or simple human error, which was the cause of all attacks I've seen directly.

It's easier to add a small config to Terraform to make your config at least key-based.

t_mahmood 13 hours ago | parent | prev | next [-]

About security, wall of shame story,

Once I had Postgresql db with default password on a new vps, and forgetting to disable password based login, on a server with no domain. And it got hacked in a day, and was being used as bot server. And that was 10 years ago.

Recently deployed server, and was getting ssh login attempts within an hour, and it didn't had a domain. Fortunately, I've learned my lesson, and turned of password based login as soon as the server was up and running.

And similar attempts bogged down my desktop to halt.

Having an machine open to the world is now very scary. Thanks God for service like tailscale exists.

dwedge 11 hours ago | parent [-]

Nothing would happen, ssh is designed to be open to the world. Using tailscale or a vpn to hide your IP is fine, but using tailscale ssh maybe not.

t_mahmood 6 hours ago | parent [-]

Well continuous attempts definitely bogged down my desktop pretty bad. Also, getting OOM on a 64gb machine multiple times a day is quiet annoying.

And one simple mistake, and we're screwed

dwedge 2 hours ago | parent | next [-]

Yeah that sounds quite annoying, but has nothing to do with ssh log noise. Maybe investigate what's causing the OOM. I have multiple 1GB vps with ssh open to the world and they never OOM, and they're obviously not just running ssh. It sounds like you've been compromised.

ericpauley 4 hours ago | parent | prev [-]

If sshd is OOMing on 64GB something else is going on…

dwedge 12 hours ago | parent | prev | next [-]

I need more info about devs getting infected over ssh in less than an hour. Unless they had a comically weak root password or left VNC I don't believe it at all

gobdovan 9 hours ago | parent [-]

Yes, <1h was a weak root password. All attacks I've seen directly were always user error. The point is effectively removing attack surfaces rather than enhancing security in needlessly exposed internet-facing protocols.

dwedge 7 hours ago | parent [-]

It must have been comically weak, like "root", "password" or something like that

selcuka 14 hours ago | parent | prev | next [-]

> Nice list! I'd say the SQLite with WAL is the biggest money saver mentioned.

Funny you said that. I migrated an old, Django web site to a slightly more modern architecture (docker compose with uvicorn instead of bare metal uWSGI) the other day, and while doing that I noticed that it doesn't need PostgreSQL at all. The old server had it already installed, so it was the lazy choice.

I just dumped all data and loaded it into an SQLite database with WAL and it's much easier to maintain and back up now.

gobdovan 13 hours ago | parent [-]

Yep, it literally is a one-file backup. And runtime it's so much faster for apps where write serialisation is acceptable.

bilinguliar 4 hours ago | parent | prev | next [-]

Sqlite + Litestream for backups.

InfraScaler 12 hours ago | parent | prev | next [-]

Does WAL really offer multiple concurrent writers? I know little about DBs and I've done a couple of Google searches and people say it allows concurrent reads while a write is happening, but no concurrent writers?

Not everybody says so... So, can anyone explain what's the right way to think about WAL?

gobdovan 11 hours ago | parent | next [-]

No, it does not allow concurrent writes (with some exceptions if you get into it [0]). You should generally use it only if write serialisation is acceptable. Reads and writes are concurrent except for the commit stage of writes, which SQLite tries to keep short but is workload- and storage-dependent.

Now this is more controversial take and you should always benchmark on your own traffic projections, but:

consider that if you don't have a ton of indexes, the raw throughput of SQLite is so good that on many access patterns you'd already have to shard a Postgres instance anyway to surpass where SQLite single-write limitation would become the bottleneck.

[0] https://www.sqlite.org/src/doc/begin-concurrent/doc/begin_co...

pixelesque 12 hours ago | parent | prev [-]

No it doesn't - it allows a single writer and concurrent READs at the same time.

InfraScaler 12 hours ago | parent [-]

Thanks! even I run a sqlite in "production" (is it production if you have no visitors?) and WAL mode is enabled, but I had to work around concurrent writes, so I was really confused. I may have misunderstood the comments.

yomismoaqui 11 hours ago | parent [-]

Writes are super fast in SQLite even if they are not concurrent.

If you were seeing errors due to concurrent writes you must adjust BUSY_TIMEOUT

InfraScaler 7 hours ago | parent [-]

Thanks I'll have a look. For now I just had a sane retry strategy. Not that I have any traffic, mind you :-)))

egwor 11 hours ago | parent | prev | next [-]

First step is to get ssh setup correctly, and second step is to enable a firewall to block incoming connections on everything except the key ports (ssh but on a different port/web/ssl). This immediately eliminates a swathe of issues!

bornfreddy 9 hours ago | parent [-]

Also use fail2ban. If nothing else to decrease the amount of junk in logs.

asymmetric 12 hours ago | parent | prev | next [-]

> Also avoid their object store.

Curious as to why you say this. I’m using litestream to backup to Hetzner object storage, and it’s been working well so far.

I guess itt’s probably more expensive than just a storage box?

Not sure but I also don’t have to set up cron jobs and the like.

gobdovan 11 hours ago | parent [-]

Historical reliability and compatibility. They claimed they were S3 compatible, but they were requiring deprecated S3 SDKs, plus S3 advanced features are unimplemented (but at least they document it [0]). There was constant timeouts for object creation and updates, very slow speeds and overall instability. Even now, if you check out r/hetzner on reddit, you'll see it's a reliability nightmare (but take it with a grain of salt, nobody reports lack of problems). Not as relevant for DB backups, but billing is dumb, even if you upload a 1KB file, they charge you for 64KB.

At least with Storage Box you know it's just a dumb storage box. And you can SSH, SFTP, Samba and rsync to it reliably.

[0] https://docs.hetzner.com/storage/object-storage/supported-ac...

nurgalive 10 hours ago | parent | prev [-]

When creating a VPS on Hetzner, it lets you by default to configure the key auth only.

jimnotgym 8 hours ago | parent [-]

From memory this is the case on DO as well