| ▲ | frollogaston 5 hours ago | |||||||||||||
Everyone knows SQL already. The harder parts that pay off are schema design, knowing how to interact with your DB in code, and knowing all the ins and outs of whatever DBMS you're using. | ||||||||||||||
| ▲ | bob1029 4 hours ago | parent [-] | |||||||||||||
I would emphasize the importance of batching and set operations. This is where I think many developers lose track of the rabbit, because you don't have much control over either of these things via ORMs. You have to get your hands dirty with raw command text. The value of this stuff is difficult to overstate. Batching allows for you to rapidly load the RDBMS. The first few times you test, it will probably go so fast you won't believe it loaded anything at all. Set operations allow for you to bring this newly loaded data to visibility in production tables nearly instantly. Your OLAP & OLTP workloads should be dominating the compute. ETL ops (loading/set ops) should be a ghost in terms of cpu time and memory. None of this is vendor specific knowledge. Every major engine has a reasonable way to bulk load and perform quick merging of records. | ||||||||||||||
| ||||||||||||||