The amount of memory required to cache all possible diffs (defined as an ordered pair of commits) would likely be in the exobytes. At current ram prices that's easily a trillion dollars of ram to run that cache lol. Git focuses on making diff calculations efficient largely because the space of possible diffs is very expensive to enumerate.

The following from Claude: """ A diff between two randomly chosen commits usually spans years of history, so it's not a few KB — the tree itself is ~1.5 GB of text, and a multi-year span rewrites a large slice of it. Call it 100–200 MB per pair on average: 8.5×10¹¹ pairs × ~2×10⁸ bytes ≈ 10²⁰ bytes, or ~150 exabytes """

I'm not sure if the website allows for diffs against arbitrary tree states, but if it does than the diff space is completely unbounded, and the ram demand is theoretically infinite.

cache all of them? no... but you could cache some of them, likely the newer are accessed more frequently.