A government agency tests all medications, why not models?

I wouldn't object to a government advisory body that tests models for safety so that users can make informed decisions. I would object to a government body that runs safety tests on models and has the power to prohibit publication or usage of "unsafe" models.

Don’t think government controlling AI is a good idea.

Not sure if they have an understanding of AI in the first place. Secondly, even though AI companies claim that they have achieved AI that needs to be heavily monitored (maybe for PR purposes), I’m not sure if that is true. Sam Altman said the same things about GPT-4 that Anthropic is now claiming about Mythos.

Government control will be a good idea once we start approaching AI that is actually destructive.

Also even if we decide to put controls in place what is the guarantee that china will do the same, specially for a model which is not actually destructive.

Imo this amounts to caring about the wrong thing. The only thing an AI model can do is take in text/images/audio as input and spit out text/images/audio as output.

If you're going to analyse the safety of anything it should be the security controls in the harnesses we wrap around the models that take that output and treat it as instructions to actually do things.

Yes: at great expense, one carried in part by drug companies. Who pays to test the open weight models?

There's a different level of personal risk with these two things. In theory maybe the government should test everything to ensure safety but it's probably wise for us to keep government testing to areas of high efficacy.

Do they? Or do they accept trail reports pay for by the pharma (super expensive, hence not affordable for open source / not-patentable medicine development)

Actually someone needs to make the positive case sufficiently well first.