That's very true, lenses on a camera work very similarly to a telescope. A larger objective (opening at the end of the lens) combined with a large aperture (lower f number) means that a lens is able to gather a lot more light at a given focal length. Certainly some of my commentary is related to the fact that my primary lenses are f/0.95, f/1.2, and f/1.8. I only shoot "fast" primes on a camera body.

That said, a /lot/ of low light performance is simply having a much larger sensor with a wider pixel pitch that is able to gather more light in the given time allotted. You cannot beat physical size in some ways for digital photography and light gathering is one of them, as it is primarily about surface area.