I thought most asteroids were basically just gravel piles loosely held together by internal gravity?

If we're talking about objects in the asteroid belt, the internal consistency varies wildly. Most smaller objects are indeed rubble piles: boulders, pebbles, and sand-like grains, down to dust. As you'd expect, they're very weak structurally. Notably, the OSIRIS-REx probe was nearly swallowed by its loose material during the sample collection on Bennu.

Some of the other large, iron-rich asteroids like Ceres, Vesta, Pallas, and Interamnia are more like protoplanets than rubble piles.

Besides the concern for structural integrity/stability, they also have reasonable amounts of water ice, volatiles, metals, ad other resources needed to supply an outpost.