Yep, there might be two factors here. From what you describe in your experiences, these seem like real artifacts that occur due to physical changes happening on and around the eyeball? (Photopsia seems to be what you describe).

In my experience when hallucinating (from sleep deprivation), it is the brain failing to correctly interpret patterns seen through the eyes.

For e.g., I would think I see a cardboard box on the ground ahead of me, but then when I get closer I realise it is just dirt of different shades that was perceived as a 3d box. Similar experience when hallucinating people. In my mind I imagine my friend with a certain colour shirt, then during a period of sleep deprivation, I think I see him on the trail ahead of me. But it was just a rock that was a similar colour.

Do hallucinogenics typically affect the brain and also the actual optics? Maybe it is some combination that is causing the perception of "little people".