| ▲ | bob1029 4 hours ago | |
I would emphasize the importance of batching and set operations. This is where I think many developers lose track of the rabbit, because you don't have much control over either of these things via ORMs. You have to get your hands dirty with raw command text. The value of this stuff is difficult to overstate. Batching allows for you to rapidly load the RDBMS. The first few times you test, it will probably go so fast you won't believe it loaded anything at all. Set operations allow for you to bring this newly loaded data to visibility in production tables nearly instantly. Your OLAP & OLTP workloads should be dominating the compute. ETL ops (loading/set ops) should be a ghost in terms of cpu time and memory. None of this is vendor specific knowledge. Every major engine has a reasonable way to bulk load and perform quick merging of records. | ||
| ▲ | evdubs 4 hours ago | parent | next [-] | |
> I would emphasize the importance of batching and set operations. Please, preach your gospel more loudly and frequently. It always feels like people complain about RDBMSs being slow because they run insert queries one at a time. | ||
| ▲ | frollogaston 4 hours ago | parent | prev [-] | |
Well yeah they should've banned ORMs in the Geneva Convention. Quickest way to irreversibly ruin your schema design and backend code. | ||