The courts, all the way to the top, have consistently interpreted the Constitution as a document that circumscribes the behavior of the government, not as a document that grants privileges to "the people" or a subset of that (e.g. citizens only).