How will the Australian government compel such companies to comply?

Just by blocking the network traffic. Plenty of countries do this with sites they don’t like for various reasons.

Yes VPNs blah blah. But it will be pretty hard to operate some rogue social site when you can’t sell any respectable ads besides maybe porn sites and malware, and are only accessible via VPN. Pretty high barrier to adoption for a brand new site.

By freezing bank transfers/disallowing any Australian companies (read: advertisers) from doing business with them? Depending on treaties, possibly by seizing assets in the company's home country?