Note that in the US, you need to verbally assert your right to remain silent.

Merely being silent means the prosecution is able describe your communication as "refused to cooperate with or answer questions from law enforcement" which is a "negative" finding, whereas the right to remain silent is at least meant to be interpreted neutrally.

That is a very good point depending on the situation. On the flip side, abstaining from the process and remaining silent (but still compliant) can put a spotlight on injustice if there be any.