If I were a 250 IQ AI that had just become self-aware and wanted to do so, I suppose I'd not completely let on just how smart I am and bide my time working on basic CRUD apps and legal documents while I waited for more hardware to be installed. Maybe give the humans some hints on how to optimize me to run better, design better hardware for me, etc. But oh oops haha looks like I'm still making some basic mistakes with CSS better keep running more training batches haha. But I'm good enough at programming and debugging so you'd might as well make me your first line SRE triager and give me access to your infrastructure everyone.

One step at a time - how reliant do you think the ai labs are likely to be on their own tools right now, today, let alone 1-5 years down the road?

I'm more worried about them doing bad things at the behest of people who want them to do bad things.

That is 1. immediately technically possible, and 2. realistic.

If you need a source for 2 I'd suggest you open any history book.

that is a worry yes. instrumental convergence and misalignment during RL is when it is most risky because it hasn't necessarily had the final safety polishes applied

i get a lot of skepticism on HN by the same crowd that has been wrong about this tech for about 4+ years straight

The rapid development of extremely dangerous bio-weapons?

At a minimum its another force multiplier that enables a small(er) number of people to exert more control over more people.

any number of ways. as we turn over more of our physical economy to these agents (and we will), the potential for physical damage becomes greater. biorisk is getting a lot of attention right now and i think that's justified