My concern is that a lot of people getting added to those lists for being "offensive people" or "trolls" are actually just people who civilly express viewpoints that the list maintainer disagrees with.

That probably happens, but most cases are combatting the trend of right wingers creating throw away accounts to say the most offensive things they can for the reactions and getting blocked

I'd invite you to check out the options this labeller provides as an example. You can also see who is on the list, so anyone can audit the list. Transparency is built in

https://bsky.app/profile/skywatch.blue | https://bsky.app/profile/skywatch.blue/lists/3lbckxhgu3r2v (list of trolls)

The thing to notice here is how much control you get as an individual. The moderation source code backing this(https://github.com/bluesky-social/ozone) has features for the subject of a label to contest as well.

What is most powerful about the labeller ecosystem is that it is an ecosystem where competition can occur. There are an ever expanding set of choices, people can make their own choices, and they can stack them together

https://bsky.social/about/blog/03-12-2024-stackable-moderati...

The number 1 thing users like about Bluesky is that they are put back into the drivers seat of their social media experience