I guess I was also confused about the different recovery modes as pointed out by aspenmayer below.

I used Shift+Option+Command+R (or hold Option and choose WiFi instead of disk) which is internet recovery using the macOS version that came pre-installed (or closest)

Whereas Command+R is local recovery which might be any macOS version that last changed the local recovery environment.