Remix.run Logo
sgt 20 hours ago

You can absolutely scale Django.

The framework itself is not the limiting factor. The main constraint of performance usually comes from Python itself (really slow). And possibly I/O.

There are well established ways to work around that. In practice, lots of heavy lifting happens in the DB, can you can offload workloads to separate processes as well (whether those are Python, Go, Rust, Java etc).

You need to identify the hotspots, and blindly trusting a framework to "do the job for you" (or for that matter, trusting an LLM to write the code for you without understanding the underlying queries) is not a good idea.

I'm not saying you are doing that, but how often do you use the query planner? Whenever I've heard someone saying Django can't scale, it's not Django's fault.

> When you start to run into scaling problems, your solution is within that framework and that becomes a limiting factor from my experience.

Using Django doesn't mean that everything needs to run inside of it. I am working on an API that needs async perf, and I run separate FastAPI containers will still using Django to maintain the data model + migrations.

Occasionally I will drop down to raw SQL, or materialized views (if you are not using them with Django, you are missing out). And the obvious for any Django dev; select_related, prefetch_related, annotate, etc etc.

otherme123 19 hours ago | parent | next [-]

> And the obvious for any Django dev; select_related, prefetch_related, annotate

And sometimes not so obvious, I have been bitten by forgetting one select_related while inadvertedly joining 5 tables but using only 4 select_related: the tests work OK, but the real data has a number of records that cause a N+1. A request that used to take 100ms now issues "30 seconds timeout" from time to time.

Once we added the missing select_related we went back to sub-second request, but it was very easy to start blaming Django itself because the number of records to join was getting high.

The cases that we usually walk out of the Django path is for serializations and representations, trying to avoid the creation of intermediate objects when we only need the "values()" return.

rtpg 2 hours ago | parent | next [-]

I really want django-seal to be upstreamed, because accidental N+1's are really nasty and django-seal helps a lot with finding those

bb88 9 hours ago | parent | prev [-]

You may already know this, this is meant for others hitting this issue frankly.

In Django, you can count the number of queries in a unit test. You don't need 1M objects in the unit test, but maybe 30 in your case.

If the unit code uses more than X queries, then you should assume you have an N+1 bug. Like if you have 3 prefetch related and 2 select related's on 30 objects, but you end up with more than 30 queries, then you have an N+1 someplace.

Even better that unit test will protect you from hitting that error in the future in that chunk of code accessing that table.

tclancy 20 hours ago | parent | prev [-]

Yeah, I don’t get the issues here. I’ve led projects that served millions of requests a day, had dozens of apps and while there are always going to be pain points and bottlenecks, nothing about the framework itself is a hinderance to refactoring. If anything, Django plus good tests made me much braver about what I would try.