Small fusion reactors don't really solve any of the key challenges. You need reaction mass to accelerate, you run out of reaction mass way too quickly even with a magical energy source on board to throw it out the back of the ship really fast.