They could have gone down the path of being a service with a monthly subscription. Instead of making the customer become the product.

Imagine an alternate universe where, since you were paying them, they kept you safe and secure online, and kept the bad actors away.

The only offering that possibly might have been compelling enough to charge for was Messenger if it existed in a vacuum, but there were already numerous services offering much the same for free (e.g. MSN, ICQ, AIM), and when others realized that is what the people actually wanted, many more immediately threw their hat in the ring (e.g. iMessage). There would have been no practical hope of it making it as a paid service.

assuming they were able to acquire customers and dominate the world with that business model, would that have prevented them from doing algorithmic feeds and promoting clickbait and poisoning politics and the rest?

sure, people would have been able to cancel their monthly facebook subscriptions if they didn't like that stuff. but we can effectively do that now just by not using it.