> Plus the low speed lines are less deadly since the total energy is proportional to velocity squared, and v is low.

You're forgetting about the probability of a crash.

The vast majority of train crashes is due to an impact with a vehicle on a railway crossing.

However, high-speed rail is grade separated, so it doesn't have railway crossings, which means the main cause of crashes is fundamentally impossible.

In other words: Regular rail has a high rate of crashes (with a small number of fatalities each) due to car/truck drivers screwing up. High-speed rail has a low rate of crashes (with a large-ish number of fatalities each) due to catastrophic failure of track & train equipment.