By corrupting memory and causing the CPU to jump from the intended instruction to a different one controlled by the attacker, at which point their instructions control what your process does next. Getting shell does not mean getting it to feed a spicy string to system(), it means tricking something in the process into overwriting memory with bytes that, if read as instructions, tell the CPU to make a system call to spawn a different process running arbitrary code.

You could do that to an internet-connected toaster with a JPEG. The system() function has nothing to do with it.