Another definition: A modern GPU a general purpose computer that can make parallelized and efficient computations. It's optimized to run limited number of operations but on large number of data points.

This happens to be useful both for graphics (same "program" running on on huge number of pixels/vertices) and neural networks (same neural operations on huge number of inputs/activations)