Now that AWS charges for public IPv4 addresses, is it still free if you need to access IPv4-only hosts?

Yeah not free if you definitely need IPv4. AWS has been adding a lot more IPv6 support to their services so hopefully the trend continues in AWS and the broader industry. You can probably get pretty far though if your app doesn't have hard requirements to communicate with IPv4 only hots.