Yes, one challenge with defining Subtractive primaries is that they are dependent on the white point of the "white" light source (e.g. D65 vs D50). While this seems inconvenient, it's worth noting that the apparent color of greys for Additive primaries is also dependent on surrounding illumination.

So primaries are useful for generating roughly orthogonal changes in perceived color, but they don't tell you how they will be perceived in absolute terms without knowing surrounding illumination. In the simplest case, asking if something is bright (even without color) is impossible without knowing the surroundings.