This sounds exactly like UX usability testing. Sit down with someone and watch them work through whatever process you’ve created.

It’s normal to compensate them for their time.

Normally though you don’t modify it after each participant. But for something very niche like following a README (as opposed to an e-commerce flow targeted to the general population) it might be fine, if less rigorous.

Yes, this is right! It's closest to usability testing in UX design.

As I already mentioned in another comment, I recommend this post for people who want to go a bit deeper: https://www.nngroup.com/articles/usability-testing-101/