I’m also curious if an AI could process the screen feed quickly enough to compete in first-person shooter games. Seems like it would be difficult without extremely high end hardware for the foreseeable future?

▲

ModernMech 4 days ago | parent | next [-]

I had students build this kind of thing in 2020 by screenshotting the game and processing it with a standard OpenCV pipeline. No GenAI needed.

▲

throwaway894345 3 days ago | parent [-]

Thank you for educating me. How does OpenCV work from the perspective of recognizing things in an image? Is there some kind of underlying model there that learns what a target looks like or not?

▲

ModernMech 3 days ago | parent [-]

The way they did it, they were writing an aimbot. So the pipeline was:

- take a screenshot

- run massive skeletal detection on it to get the skeletons of any humanoids

- of those skeletons, pick a target closest to the player

- for that target, get coordinates of head node

- run a PID to control the cursor to where the head node is located

- move the camera one frame, repeat the whole process. If you can fit that pipeline to 16ms it can run in real time.

▲

throwaway894345 2 days ago | parent [-]

Wow, that's fascinating. Were they able to fit the whole thing inside the 16ms frame?

	▲	ModernMech 2 days ago \| parent [-]
		oh yeah, with little problem, especially with a GPU it's not hard at all.

▲

Scion9066 4 days ago | parent | prev | next [-]

There's already models specifically for things like identifying players in Counter-Strike 2, including which team they're on.

Someone has even rigged up a system like that to a TENS system to stimulate the nerves in their arm and hand to move the mouse in the correct direction and fire when the crosshair is over the enemy.

We are definitely already there.

	▲	fragmede 3 days ago \| parent [-]
		They documented it on YouTube for us to see: https://youtu.be/9alJwQG-Wbk holy shit that's amazing.

▲

Thaxll 4 days ago | parent | prev [-]

It already exists.