Defendants don't invoke that because in most states and federally, they build the case against you slowly over a long period of time before arrest, then stall as long as possible for discovery, then when they finally fulfill discovery they overwhelm you with a bunch of useless stuff so that it takes forever to get the useful information. Invoking right to speedy trial means the prosecution gets a very strong advantage to the defense.