Do the db guys at your company help you optimize queries and table set up at all? Ours basically don’t at all. Their job is to maintain the db apparently and us devs are left to handle this and it seems wrong. I’ve been partitioning tables and creating indexes the past few weeks trying to speed up a view and running explain analyze and throwing the results in Gemini and my queries are still slow af. I had one sql class in college, it’s not my thing. Seems like if dbas would spend a few minutes with me asking about the data and what we are trying to do they could get this guys results relatively easily. Am I wrong?

We didn’t use the DBA’s for this but my last few teams, we got good at DB’s, performance etc. DBA’s were too general and they kept the lights on, but for real performance you should get one or two people who know what they’re doing for your applications. Or learn. I took on juniors who are now fantastic.

For the first decade I wanted nothing to do with DB’s aside from places to store data. One day I saw a few things that made a massive difference and then went wild on learning how to speed things up. It’s fantastic and because few devs know this stuff well, it becomes a superpower. You wouldn’t believe what you can squeeze out of modern SQL DB’s and hardware, without touching any kind of optimised solutions. Which I love too but that’s a different post.

Maybe ask the DBA’s a few questions and see if that triggers any interest for you. Look at query plans and how many rows are processed for a query. How many columns. What is being locked. Can you remove locks when you’re just running a query and how much does that speed things up. There are queries for all sorts of metrics, eg which indexes are huge but never used. The DB can often suggest indexes, but don’t just use add the suggestions. Use them as a starting point to reason about your own. Try get down to low millisecond queries for really frequent stuff, because it’ll make them fast and means less time locking the DB, less RAM, less temp table storage.

All my other skills have aged. Fundamental database knowledge lasts.

What are “db guys” in this context?

There are two distinct job functions.

1) dba - maintain OS, db software/hardware - DBs are complex beasts you want experts setting up hardware or deploying into the clown in a smarter way (virtual machines in the clown will cost you a fortune and your sanity) 2) database developers- specialists in writing sql.

The two functions work together but are distinct.

Nowadays due to private equity and Stanford MBAs we have junior engineers doing all plus “dev ops”.

It’s an absolute circus.

IMO, this is why we end up with ask these crazy DB startups - routing around the damage.

RDMS in modern hardware are insanely fast and powerful.

If you make it your thing and keep on being good at your other thing, you’re gonna be 90% more valuable than most of your coworkers.

I totally lose respect for sr engineers who can’t write sql to find even simple answers to questions.

It’s never bad to have another arrow in your quiver

95% it’s a missing/improper index but the insane workarounds “engineers” put in place actively work against the database.

Does your hotpath query have a covering index? Is it seek-able. Are you predicates SARG-able.

Seriously biggest bang for your buck is to understand SARG-ability and understand indexes in your database really well.

Performance engineering is a modestly specialized subdomain - I think that applies to databases just as it does to code.

Nope. If they didn't actively work against us, we would thank the lucky stars.

I always see these fancy DB engines and data lake blog posts and I am curious… why?

At every place I’ve worked at this is a solved problem: Hive+Spark, just keep everything sharded across a ton of machines.

It’s cheaper to pay for a Hive cluster that does dumb queries than paying these expensive DB licenses, data engineers building arbitrary indices, etc… just throw compute at the problem, who cares. 1TB of RAM /flash is so cheap these days.

Even working on the worlds “biggest platforms” a daily partition of user data is like 2TB.

You’re telling me a F500 can’t buy a 5 machine/40TB cluster for like $40k and basically be set?