Well yeah, you're using them as reference books. You wouldn't necessarily approach a textbook the same way, since the point there is to guide you through a series of lessons that gradually build on each other. Similarly for narrative works. Jumping into the middle of a nonlinear narrative entirely misses the intentional choices behind the structure, for example.

You can read how you want, of course. The consequence is sometimes simply that you close yourself off from other aspects of the medium. There aren't many aspects bigger than narrative structure, but that's your choice to make.