| ▲ | frollogaston 10 hours ago |
| This advice is good, but every startup I've worked with has run into lower hanging fruit than this even. Less scaling problems and more just organizational. Usually what fixes that is: 1. Don't use an ORM. 2. Use serial PKs, not meaningful fields (article mentions this). 3. Use jsonb if needed, but sparingly. 4. Make your source of truth append-only, meaning you only insert, never update or delete. You can have secondary denormalized tables that are mutated, but that's only for performance/convenience and shouldn't be your sot. 5. Use connection pools, but be mindful of how many connections you're using. You probably don't need PgBouncer unless you've messed something up. 6. In code, avoid explicit transactions unless there's a clear reason you need them. Usually only need those for denormalized parts. Just take a conn from the pool, do something, commit, return conn to pool. If you're going to keep an xact open, never do long-running stuff in the middle like RPCs. Too often I see people leave xacts open without much thought. Edit: Also don't use SERIALIZABLE xacts almost ever. 7. Something is probably wrong if you're using explicit locking like SELECT FOR UPDATE. 8. Don't reinvent a type system by having a single table where each row can mean many different things depending on a "type int" enum col. Seems oddly specific, but for some reason someone always tries this. 9. Related to above, don't reinvent a graph DB, typically with "node"/"edge" tables that FK into themselves or in a cycle. 99% of the time what you're trying to do is easily solvable with regular normalized tables. |
|
| ▲ | ozim 9 hours ago | parent | next [-] |
| Don't use an ORM. Highly debatable. When your highest cost is developers salaries. Don't reinvent a type system by having a single table where each row can mean many different things depending on a "type int" enum col. Easy to say, harder to not do when you have business requirements on table, customer pressure and budget already gone on discussing with DBA who maybe is right but you are burning money right here and right now. The same with point no. 9 |
| |
| ▲ | sandeepkd 9 hours ago | parent | next [-] | | I would highly recommend using ORM's, with the caveat to know exactly when not to use them. Startups do not fall in that bucket. 1. Most of these advices are unfortunately impractical and incomplete for startups. A good Data Model is highly dependent on understanding the business requirements, data flows. Means unless you are repeating yourself in the same domain its really hard to come up with a good schema in first iteration. 2. Startups are in the mode of discovering the schema for most part 3. Deleting columns is harder than adding additional columns, no one takes that risk so everyone ends up with schema bloat 4. Once you go a little bigger you will realize that the integer based UUID are not that great of a choice. Those are separate tables which maintain those index counters and not part of your DDLs. They have their own set of issues with data merging, backups and recovery For the OP, I looked at the repo (https://github.com/hatchet-dev/hatchet/blob/main/sql/schema/...) , 1. seems like the schema has database functions - Thats a potential scaling issue, plus you are asking vertical only scalable component to do something which could have taken care by horizontally scalable component 2. TEXT datatype for pretty much every attribute - this is a footgun, you cant use them for indexes properly, in the absence of length checks they can be abused from client side | | |
| ▲ | frollogaston 8 hours ago | parent [-] | | I've never actually known business requirements ahead of time, when I worked for a startup and for a large company. Stuff happens. You build your schema iteratively and yes, accept some temporary bloat when cols get added thoughtlessly. The UUID backup/recovery thing isn't an issue if you're doing append-only. Joins on UUIDs are far slower, enough that even at small scale it can cause issues when you have many joins. Anyway I won't argue too hard against UUIDs cause they work too, just anything is better than using meaningful fields as the PKs. |
| |
| ▲ | waisbrot 9 hours ago | parent | prev | next [-] | | In my experience, ORMs save a little bit of writing SQL and then cost an unbounded quantity of time in debugging mysterious problems because knowing why a query is slow now requires understanding the DB, your own code, and also the ORM. | | |
| ▲ | frollogaston 8 hours ago | parent [-] | | And also more loc than SQL usually. It's not even a deferred cost, it's at least slightly worse upfront |
| |
| ▲ | xmprt 9 hours ago | parent | prev | next [-] | | ORMs are just tech debt. Even if your highest cost is developer salaries, you're just pushing that cost down the line. | | |
| ▲ | Stitch4223 4 hours ago | parent | next [-] | | For active record patterns I disagree: it’s nice to work with data in a way that feels natural. Chugging simple stuff around in a recognizable way is what you’d end up writing anyway in many applications. Writing your ORM-lite is waste. When it comes to advanced queries learning the ORM equivalent to the SQL it should write… ORM’s can be outright terrible, and I completely agree with you. Every ORM has their own names and way of doing things, so this knowledge is hard to port to other ORMs. It’s requires you knowing the right SQL first, then knowing how to write that in the ORM dialect. Reaching for the more advanced ORM trickery means grasping hard to grasp subjects twice with the risk of misunderstanding twice, and as a bonus worse ORM documentation than plain active record features. Oh, and others need to understand what you’ve written as well. | |
| ▲ | kentm 9 hours ago | parent | prev [-] | | I'd also argue whether ORMs actually save that much time in practice. In Java, for example, the main time sink is the JDBC plumbing and its easy to use something like JDBI that handles that plumbing without abstracting away the underlying SQL. The application->database layer is pretty impactful and it pays to pay attention to it, because poor access patterns will cause a lot of trouble in the future, and its made worse by not having a very accurate understanding of whats going on in that layer. I think a lot of developers don't have a good sense on where their time sinks actually are. Boilerplate is not pleasant to write but also not the timeline-destroyer people tend to think it is. And its often not enough of a time-sink to warrant introducing "magic" that will have very large negative future impacts. | | |
| ▲ | frollogaston 8 hours ago | parent [-] | | Yeah, they'll often conflate the ORM with the nice stuff you want like connection pools. And the thing about time estimates is real. Another thing is they'll optimize too hard for having fewer tables. |
|
| |
| ▲ | frollogaston 9 hours ago | parent | prev | next [-] | | It was kinda debatable until people started Claude coding everything. Even before, I would've said every SWE should just know SQL, it's not much buy-in to understand the foundation of like your entire backend. Also I'm not a DBA if that's what you meant. | | |
| ▲ | InsideOutSanta 9 hours ago | parent | next [-] | | Leaning SQL is arguably less dev work over the long run than learning an ORM and then learning how it works so you can fix performance issues. | | |
| ▲ | frollogaston 8 hours ago | parent [-] | | Yep, there have been extended periods of time where my entire job title might as well have been "ORM remover" because they backed themselves into a corner |
| |
| ▲ | ozim 7 hours ago | parent | prev [-] | | I have never seen a dev and we never would hire a dev that doesn’t know SQL and yet we still use ORM for each and every app we develop. |
| |
| ▲ | swasheck 8 hours ago | parent | prev [-] | | you’re going to pay the cost regardless. one approach absorbs the cost up front, and the other defers it, with interest |
|
|
| ▲ | poncho_romero 7 hours ago | parent | prev | next [-] |
| In the PHP back end I work on, I find the ORM immensely helpful because we need to instantiate objects to do things like permissions checks. The alternative seems like much more work. What am I missing with regards to ORMs being a bad idea? |
| |
| ▲ | ozim 7 hours ago | parent [-] | | It is just „SQL people” crying out not knowing how to work with ORM. They always claim that devs who use ORM don’t know SQL. But I never seen or hired a dev that doesn’t know SQL and yet for each project we use ORM. I also never seen anyone claiming that you doesn’t have to know SQL and ORM is enough from the opposite side. If you really run into spot where your ORM breaks you can always drop to SQL. If you build project „SQL first” you robbed yourself from ORM upsides and you are bound to badly implement your own one in the long run. | | |
| ▲ | frollogaston 6 hours ago | parent [-] | | The issue is if you use the ORM, you still need to understand the DB it's on top of. There isn't much point in using such a leaky abstraction when it's easy enough to just use SQL. |
|
|
|
| ▲ | sklarsa 8 hours ago | parent | prev | next [-] |
| There's nothing wrong with an ORM if you want to move quickly and get something up-and-running, which is the case for pretty much every startup who needs an article like this. As long as you're aware of the typical footguns (N+1 queries) and understand your ORM's lazy-loading scheme, IMO it's worth the tradeoff of developing yet another way to manage and parametrize your queries. I'd rather spend the time building out my product than prematurely optimizing and overthinking my DB schemas at the earliest phases of a project. |
| |
| ▲ | preg_match 7 hours ago | parent [-] | | The main problem is ORMs are more work, because you need to understand the ORM. That’s its own beast, each ORM is different, and they change between releases. But SQL is something you already should understand, so you get to reuse the knowledge. Also you can make SQL much more tolerable if you use migrations. That’s really where ORMs shine, but you don’t need the rest. Also query builders, although those also have a risk of abuse similar to ORMs. 99% of SQL should be static. That sounds like an absurdly high percent but it’s true if you use all the SQL features. |
|
|
| ▲ | edoceo 10 hours ago | parent | prev | next [-] |
| What's wrong with select for update? I've found it useful in a few places. It is that fixed when there is an append only source of truth? |
| |
| ▲ | frollogaston 10 hours ago | parent [-] | | Right, that whole class of problems mostly goes away when you're not mutating your sot. I've actually never needed to use SELECT FOR UPDATE that I can think of. It does have its legitimate uses, but there are footguns that general SWEs not super familiar with DBs won't know about, like how it still doesn't prevent all types of race conditions unless you're in SERIALIZABLE mode. | | |
| ▲ | jvidalv 9 hours ago | parent [-] | | For games is a must have. | | |
| ▲ | frollogaston 7 hours ago | parent [-] | | If there's a high frequency of changes from a single user all being stored, probably need more specific approaches like that |
|
|
|
|
| ▲ | manphone 10 hours ago | parent | prev | next [-] |
| #8 is https://en.wikipedia.org/wiki/Entity%E2%80%93attribute%E2%80... and it’s one upside is that it’s very flexible and it’s downside is literally everything else. If you can be 100% certain that all of the destination value types are the same then it can have compressibility benefits. |
| |
| ▲ | sgarland 8 hours ago | parent [-] | | I have no idea why you’ve been downvoted, as you accurately described EAV. I’m a DBRE who cares deeply about schema design, FWIW. | | |
|
|
| ▲ | foo42 7 hours ago | parent | prev | next [-] |
| On point 4, is the general flow that an "update" would read the current latest, check whatever conditions, then rather than updating, insert a new record (all in a transaction), or is your append only sot more an ordered "log" of update requests with the result of any conditional operations requiring a replay (from at least a check point). Or something else entirely. I'm generally an immutable first, type developer but ive not had to do much schema design for a while and looking back to previous attempts we never hit scale to stress test my approaches. |
| |
| ▲ | frollogaston 6 hours ago | parent [-] | | It's the first thing, read-then-insert, not blind insert. Also you don't always need to do the read and the insert in the same transaction. |
|
|
| ▲ | dwb 9 hours ago | parent | prev | next [-] |
| I can certainly see the appeal of number four, but on more than a couple of systems I've worked on, it would have blown up the amount of data stored in a fair proportion of tables for very questionable benefit. It's a good technique to call out, but is it really right to dictate it for everything? And what is people's opinion of the reverse: source-of-truth in traditionally-modelled mutable relational tables, with a change log written out with triggers? |
| |
| ▲ | frollogaston 8 hours ago | parent [-] | | I've done it the other way before, sometimes a team choice rather than my own, and it's ended badly any time the changelog table is actually needed. Startups or just chaotic projects will change requirements and then need data that's only in changelog tables. The changelog tables in the meantime were only barely maintained as an afterthought, lacking FKs of course, and only useful for manual debug if even that. |
|
|
| ▲ | renegade-otter 10 hours ago | parent | prev | next [-] |
| Most of the time you will save yourself a lot of grief by having a transaction decorator for each endpoint, as each HTTP call should be atomic. And then have another read-only decorator for RO transaction. |
| |
| ▲ | frollogaston 10 hours ago | parent | next [-] | | That's an easy way to accidentally leave a xact open way too long. You might have enough connections in to support this normally, but when things go slightly wrong, they go very wrong. | | |
| ▲ | 10000truths 6 hours ago | parent | next [-] | | Every RDBMS out there has an option to configure a DB-enforced transaction timeout. But that configuration is tied to a connection, not to a transaction. So using that configuration means giving up connection pooling. Which a lot of people don't want to do because it impacts latency and DB resource usage. | | |
| ▲ | frollogaston 6 hours ago | parent [-] | | Every xact within a given connection will use the same connection-wide config, but the timeout is counting how long a single transaction takes, right? I don't see why you'd need to give up pooling for this unless you need different settings per xact. | | |
| ▲ | 10000truths 6 hours ago | parent [-] | | Pooling means you're sending multiple queries (from different HTTP requests) over the same DB connection. A well-designed DB wire protocol will allow for pipelining those queries: [send Query 1] -> [send Query 2] -> [send Query 3] -> [receive Result 1] -> [receive Result 2] -> [receive Result 3] But in Postgres and MySQL, pipelined queries are not executed in parallel. They're just queued up for a single thread (per connection) to execute sequentially. Thus, if Query 1 is a transaction that takes too long, then it ends up blocking the execution of Query 2 and Query 3. |
|
| |
| ▲ | mrkaye97 10 hours ago | parent | prev [-] | | +1 to this - I've griped pretty often that FastAPI's documentation implicitly recommends this (https://fastapi.tiangolo.com/tutorial/sql-databases/#create-...) by suggesting using dependency injection to manage database connections, only to start seeing connection pool exhausted errors as soon as the number of concurrent requests exceeds the number of allowed connections. | | |
| |
| ▲ | mikeocool 10 hours ago | parent | prev [-] | | If your end point does something like: * read from the database * make a request to an API (or really any kind of long running non-database thing) * write to the database You're going to end up with a transaction that is open way longer than it needs to be, particularly if you're upstream API is misbehaving, which will potentially end up causing a lot more grief. |
|
|
| ▲ | atom_arranger 10 hours ago | parent | prev | next [-] |
| Can you elaborate on 4 a bit, are you saying to always use event sourcing, or something like it? |
| |
| ▲ | frollogaston 10 hours ago | parent [-] | | Yes, it's that. You do probably end up wanting to store some "latest" denorm tables at some point, but it takes surprisingly long to reach that point, and isn't hard when you get there. There are disadvantages to this, but it's a safe default. The alternative is possibly losing important data, finding out later you want historical records of things that are stored in kludgy separate tables, getting into more advanced locking situations, and having more complex DB migrations. Which I've had to pull teams out of many times. | | |
| ▲ | 0x696C6961 9 hours ago | parent [-] | | Using event sourcing instead of basic crud should go on a startup suicide guide ... | | |
| ▲ | atom_arranger 7 hours ago | parent [-] | | I don’t have a lot of experience related to this so I’m just noting some things. Some people in this thread don’t seem to think it’s that hard or overcomplicated. When reading Designing Data-Intensive Applications my main takeaway was that event sourcing can make it easier to solve a lot of issues like performance, scaling, consistency, auditability, etc. It would be interesting to look into what a low overhead way of implementing CRUD with event sourcing in Postgres would look like, then decide if it’s too complex. | | |
| ▲ | 0x696C6961 6 hours ago | parent | next [-] | | Don't trust anyone who tells you event sourcing is simple to implement. | |
| ▲ | frollogaston 7 hours ago | parent | prev [-] | | Try making Hackernews lite. Users can post, comment on posts, vote/unvote posts, and delete their own posts and comments. CRUD tables might be post, comment, maybe vote. Event tables might be create_post, delete_post, create_comment, delete_comment, vote, unvote. |
|
|
|
|
|
| ▲ | christophilus 4 hours ago | parent | prev | next [-] |
| Can’t do RLS without a transaction, though, right? |
|
| ▲ | waisbrot 9 hours ago | parent | prev | next [-] |
| This is an excellent distillation. Yes, in practice you may find that one or two of these don't apply to your own special start-up. But probably they all actually do. |
| |
|
| ▲ | kentm 10 hours ago | parent | prev | next [-] |
| For #4 I always tell people to assume 99% append only, but do consider the need for updates/deletes in edge cases. If any of the data is PII and you will be subject to GDPR (you probably want to be at some point) then you will need a way to hard delete it. A large portion of append-only datasets I’ve worked with have run into some edge case that required an update. Also if you’re doing an append only log plus mutable view then please use triggers or materlialized views. I ran into a few cases where the approach was to just update both tables and that lead to divergences between the two. |
| |
| ▲ | frollogaston 10 hours ago | parent [-] | | Yeah, deletes need to be on the table (no pun intended) for PII redaction, and also emergencies where you manually do it. Both those situations are much safer when the DB is normally append-only. | | |
| ▲ | kentm 10 hours ago | parent [-] | | Agreed. Usually an updated-on timestamp is sufficient to cover your bases without over-complicating the table. And your PII redaction is probably going to be shredding or nulling the fields instead of hard record-level deletes. |
|
|
|
| ▲ | j45 10 hours ago | parent | prev [-] |
| Nice summary - While it's not my first choice, or habit, in many use cases, using an ORM is still OK looking back in the start especially when the schema is being fleshed out prior to understanding what needs optimizing. It's trivial to start optimizing a query after taking it away from ORM. The new angle I think is the ability for LLMs to use ORMs given the documentation, etc. The only other thing I'd say is the benefits of running something that works with Postgres, such as Supabase, Hasura, etc. It really can be the best of multiple worlds, especially in the beginning in terms of having flexibility in one place. |