There are unlikely to be many six-fingered hands in the training data. So there's little reason for the model to develop the ability to recognize one when it encounters it. Maybe the result improves if you break the task down into two steps of listing the bounding boxes of all fingers in the image and then counting the number of bounding boxes.